Bulk Operations

The Bulk Operations API allows administrators to perform large-scale operations such as bulk user imports, batch updates, and asynchronous job tracking. This API is designed for efficiency and handles both synchronous (small batches) and asynchronous (large batches) processing.

Overview

Key Features:

• Bulk user import/update (up to 10,000 users per request)

• Synchronous processing for small batches (< 100 users)

• Asynchronous processing with job tracking for large batches (>= 100 users)

• Job status monitoring with S3 download links

• Error tracking and validation

• 30-day job retention with automatic cleanup

Permissions: All bulk operations require admin or super admin privileges.

---

User Import

Bulk Import Users

Import or update multiple users in a single request.

POST /v1/bulk/users

Request Body:

Field	Type	Required	Description
users	array	Yes	Array of user objects (max 10,000)
options	object	No	Import options

User Object Fields:

Field	Type	Required	Description
userId	string	Conditional	User ID (required if email not provided)
email	string	Conditional	Email address (required if userId not provided)
customFields	object	No	Custom profile fields
metadata	object	No	Additional metadata

Import Options:

Field	Type	Default	Description
upsert	boolean	true	Update existing users or create new ones
skipDuplicates	boolean	false	Skip duplicate users instead of updating
sendWelcomeEmail	boolean	false	Send welcome email to new users

Example Request (Small Batch - Synchronous):

curl -X POST "https://api.monterosa.cloud/loyalty/v1/bulk/users" \
  -H "Authorization: Bearer ADMIN_TOKEN" \
  -H "X-Monterosa-Org-ID: YOUR_ORG_UUID" \
  -H "Content-Type: application/json" \
  -d '{
    "users": [
      {
        "userId": "user_001",
        "email": "[email protected]",
        "customFields": {
          "firstName": "John",
          "lastName": "Doe",
          "tier": "bronze"
        }
      },
      {
        "userId": "user_002",
        "email": "[email protected]",
        "customFields": {
          "firstName": "Jane",
          "lastName": "Smith",
          "tier": "silver"
        }
      }
    ],
    "options": {
      "upsert": true,
      "skipDuplicates": false,
      "sendWelcomeEmail": false
    }
  }'

Example Response (Synchronous - < 100 users):

{
  "success": true,
  "data": {
    "status": "completed",
    "totalRecords": 2,
    "processed": 2,
    "succeeded": 2,
    "failed": 0,
    "errors": []
  }
}

Example Request (Large Batch - Asynchronous):

curl -X POST "https://api.monterosa.cloud/loyalty/v1/bulk/users" \
  -H "Authorization: Bearer ADMIN_TOKEN" \
  -H "X-Monterosa-Org-ID: YOUR_ORG_UUID" \
  -H "Content-Type: application/json" \
  -d '{
    "users": [
      // ... 500 users
    ],
    "options": {
      "upsert": true
    }
  }'

Example Response (Asynchronous - >= 100 users):

{
  "success": true,
  "data": {
    "jobId": "bulk_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "status": "processing",
    "totalRecords": 500,
    "processed": 0,
    "succeeded": 0,
    "failed": 0,
    "statusUrl": "/bulk/YOUR_ORG_UUID/jobs/bulk_a1b2c3d4-e5f6-7890-abcd-ef1234567890"
  }
}

---

Processing Modes

Synchronous Processing (< 100 users)

For small batches, the API processes users immediately and returns results synchronously:

• Processes in batches of 25 (DynamoDB BatchWrite limit)

• Returns complete results in the response

• No job tracking required

• Ideal for: Real-time imports, small data migrations

Asynchronous Processing (>= 100 users)

For large batches, the API creates a background job:

• Uploads user data to S3

• Creates a job record for tracking

• Returns job ID immediately (202 Accepted)

• Background Lambda processes the job

• Poll job status endpoint for progress

• Ideal for: Large imports, scheduled migrations

---

Job Tracking

Get Job Status

Check the status and progress of a bulk operation job.

GET /v1/bulk/jobs/{jobId}

Path Parameters:

Parameter	Type	Required	Description
jobId	string	Yes	Job ID from bulk operation response

Example Request:

curl -X GET "https://api.monterosa.cloud/loyalty/v1/bulk/jobs/bulk_a1b2c3d4-e5f6-7890-abcd-ef1234567890" \
  -H "Authorization: Bearer ADMIN_TOKEN" \
  -H "X-Monterosa-Org-ID: YOUR_ORG_UUID"

Example Response (In Progress):

{
  "success": true,
  "data": {
    "jobId": "bulk_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "type": "user_import",
    "status": "processing",
    "totalRecords": 500,
    "processed": 250,
    "succeeded": 245,
    "failed": 5,
    "errors": [
      {
        "row": 15,
        "userId": "user_015",
        "error": "Invalid email format"
      },
      {
        "row": 28,
        "error": "userId or email is required"
      }
    ],
    "startedAt": "2025-10-11T10:00:00.000Z",
    "completedAt": null,
    "createdBy": "[email protected]",
    "downloadUrl": null
  }
}

Example Response (Completed):

{
  "success": true,
  "data": {
    "jobId": "bulk_a1b2c3d4-e5f6-7890-abcd-ef1234567890",
    "type": "user_import",
    "status": "completed",
    "totalRecords": 500,
    "processed": 500,
    "succeeded": 495,
    "failed": 5,
    "errors": [
      {
        "row": 15,
        "userId": "user_015",
        "error": "Invalid email format"
      },
      {
        "row": 28,
        "error": "userId or email is required"
      },
      {
        "row": 103,
        "userId": "user_103",
        "error": "Email already exists"
      },
      {
        "row": 247,
        "error": "customFields must be an object"
      },
      {
        "row": 389,
        "userId": "user_389",
        "error": "Invalid email format"
      }
    ],
    "startedAt": "2025-10-11T10:00:00.000Z",
    "completedAt": "2025-10-11T10:05:23.000Z",
    "createdBy": "[email protected]",
    "downloadUrl": "https://loyalty-api-uploads.s3.amazonaws.com/bulk-imports/results.json?X-Amz-Expires=3600&..."
  }
}

Job Statuses:

• pending - Job created, waiting to start

• processing - Job is being processed

• completed - Job finished successfully

• failed - Job failed due to system error

---

Validation

User Validation Rules

Each user object is validated before processing:

Required Fields:

• At least one of: userId or email

Email Validation:

• Must match regex: ^[^\s@]+@[^\s@]+\.[^\s@]+$

Custom Fields Validation:

• Must be an object (if provided)

• Cannot be a string, array, or primitive

Example Validation Errors:

{
  "errors": [
    {
      "row": 0,
      "error": "userId or email is required"
    },
    {
      "row": 5,
      "userId": "user_005",
      "error": "Invalid email format"
    },
    {
      "row": 12,
      "userId": "user_012",
      "error": "customFields must be an object"
    }
  ]
}

---

Use Cases

1. Initial Data Migration

Import existing user base from another system:

// Prepare user data
const users = existingUsers.map(user => ({
  userId: user.id,
  email: user.email,
  customFields: {
    firstName: user.first_name,
    lastName: user.last_name,
    tier: user.membership_level,
    legacyId: user.legacy_id
  }
}));

// Split into chunks of 5000 users
const chunks = chunkArray(users, 5000);

// Import each chunk
for (const chunk of chunks) {
  const response = await fetch('/v1/bulk/users', {
    method: 'POST',
    headers: {
      'Authorization': `Bearer ${adminToken}`,
      'X-Monterosa-Org-ID': orgId,
      'Content-Type': 'application/json'
    },
    body: JSON.stringify({
      users: chunk,
      options: {
        upsert: true,
        skipDuplicates: false
      }
    })
  });

  const result = await response.json();

  if (result.data.jobId) {
    // Track async job
    await trackJob(result.data.jobId);
  }
}

2. Batch User Updates

Update custom fields for multiple users:

curl -X POST "https://api.monterosa.cloud/loyalty/v1/bulk/users" \
  -H "Authorization: Bearer ADMIN_TOKEN" \
  -H "X-Monterosa-Org-ID: YOUR_ORG_UUID" \
  -H "Content-Type: application/json" \
  -d '{
    "users": [
      {
        "userId": "user_001",
        "customFields": {
          "tier": "gold",
          "vip": true
        }
      },
      {
        "userId": "user_002",
        "customFields": {
          "tier": "platinum",
          "vip": true
        }
      }
    ],
    "options": {
      "upsert": true
    }
  }'

3. Job Status Polling

Monitor job progress:

async function trackJob(jobId) {
  let status = 'processing';

  while (status === 'processing' || status === 'pending') {
    const response = await fetch(`/v1/bulk/jobs/${jobId}`, {
      headers: {
        'Authorization': `Bearer ${adminToken}`,
        'X-Monterosa-Org-ID': orgId
      }
    });

    const result = await response.json();
    status = result.data.status;

    console.log(`Job ${jobId}: ${result.data.processed}/${result.data.totalRecords} processed`);

    if (status === 'processing' || status === 'pending') {
      // Wait 5 seconds before checking again
      await new Promise(resolve => setTimeout(resolve, 5000));
    }
  }

  if (status === 'completed') {
    console.log(`Job completed successfully`);
    console.log(`Succeeded: ${result.data.succeeded}`);
    console.log(`Failed: ${result.data.failed}`);

    if (result.data.downloadUrl) {
      console.log(`Download results: ${result.data.downloadUrl}`);
    }
  }
}

4. CSV Import

Import users from CSV file:

import Papa from 'papaparse';

// Parse CSV file
const parseCSV = (file) => {
  return new Promise((resolve) => {
    Papa.parse(file, {
      header: true,
      complete: (results) => {
        const users = results.data.map(row => ({
          userId: row.user_id,
          email: row.email,
          customFields: {
            firstName: row.first_name,
            lastName: row.last_name,
            tier: row.tier || 'bronze'
          }
        }));
        resolve(users);
      }
    });
  });
};

// Import users
const file = document.getElementById('csvFile').files[0];
const users = await parseCSV(file);

const response = await fetch('/v1/bulk/users', {
  method: 'POST',
  headers: {
    'Authorization': `Bearer ${adminToken}`,
    'X-Monterosa-Org-ID': orgId,
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({ users })
});

---

Error Handling

Request Validation Errors

Empty Users Array:

{
  "success": false,
  "error": {
    "message": "users array cannot be empty"
  }
}

Too Many Users:

{
  "success": false,
  "error": {
    "message": "Maximum 10,000 users per request"
  }
}

Invalid JSON:

{
  "success": false,
  "error": {
    "message": "Invalid JSON in request body"
  }
}

Job Errors

Job Not Found:

{
  "success": false,
  "error": {
    "message": "Job not found"
  }
}

System Error:

{
  "success": false,
  "error": {
    "message": "Failed to process bulk users"
  }
}

---

Limits and Quotas

Operation	Limit
Users per request	10,000
Synchronous threshold	100 users
Batch write size (DynamoDB)	25 users
Job retention	30 days (TTL)
Download URL expiry	1 hour
Max concurrent jobs	Unlimited (queued)

---

Best Practices

1. Batch Size: Use 1,000-5,000 users per request for optimal performance

2. Error Handling: Always check for validation errors in the response

3. Job Tracking: Poll job status every 5-10 seconds for large batches

4. Idempotency: Include unique userId values to prevent duplicates

5. Data Validation: Validate data before import to minimize errors

6. Progress Monitoring: Track processed count relative to totalRecords

7. Download Results: Download job results within 1 hour (signed URL expiry)

8. Retry Logic: Implement retry logic for failed requests with exponential backoff

---

Performance Considerations

Synchronous Operations (< 100 users)

• Processes in ~2-5 seconds

• Blocks until complete

• Use for: Real-time imports, small updates

Asynchronous Operations (>= 100 users)

• Returns immediately with job ID

• Processing time: ~1-2 seconds per 100 users

• Use for: Large imports, scheduled tasks

Example Processing Times:

• 100 users: ~1-2 seconds (async)

• 500 users: ~5-10 seconds (async)

• 1,000 users: ~10-20 seconds (async)

• 10,000 users: ~1-2 minutes (async)

---

Data Export

When a job completes, a download URL is provided for:

• Complete results with success/failure status

• Detailed error messages

• Row-level error tracking

Download URL Format:

https://loyalty-api-uploads.s3.amazonaws.com/bulk-imports/{orgId}/{jobId}/results.json?X-Amz-Expires=3600&...

Note: Download URLs expire after 1 hour for security.

---

Related APIs

• User Profiles API - Individual user management

• Audience API - Internal bulk operations

• Transactions API - Bulk transaction operations

---

Support

For questions about the Bulk Operations API:

• Documentation: https://products.monterosa.co/mic/reference

• Support: [email protected]