Batches API

API for managing pipeline batch executions.

Overview

A Batch represents a single execution instance of a pipeline. When a batch is created, the pipeline runs in the Prefect workflow engine, and you can monitor the progress of each step.

Endpoints

POST /api/v1/batches/{pipeline_id}

Executes a pipeline batch.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID

Query Parameters

Parameter	Type	Required	Description
pipeline_version	string	No	Execute specific version (default: latest)

Response

200 OK

{
  "item": "batch-xyz789"
}

Field	Type	Description
item	string	Created batch ID

---

GET /api/v1/batches/{pipeline_id}

Retrieves the latest batch status of a pipeline.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID

Response

200 OK

{
  "id": "batch-xyz789",
  "name": "sales-pipeline",
  "deployment": "run-pipeline/sales-pipeline",
  "state": {
    "state": "running",
    "steps": {
      "extract": {
        "state": "success",
        "offsets": {"partition_0": 1000}
      },
      "transform": {
        "state": "running",
        "offsets": {"partition_0": 500}
      },
      "load": {
        "state": "pending",
        "offsets": {}
      }
    }
  }
}

Batch State Values

State	Description
stop	Stopped
running	Running
pending	Pending
scheduled	Scheduled

Step State Values

State	Description
none	Not started
pending	Pending
running	Running
success	Success
failed	Failed

---

GET /api/v1/batches/{pipeline_id}/{batch_id}

Retrieves the status of a specific batch.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID
batch_id	string	Yes	Batch ID

Response

Same format as GET /api/v1/batches/{pipeline_id}.

---

GET /api/v1/batches/{pipeline_id}/

Retrieves all batch lists for a pipeline.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID

Query Parameters

Parameter	Type	Default	Description
limit	integer	100	Maximum number of items to retrieve
token	string	-	Pagination token

Response

200 OK

{
  "items": {
    "batch-xyz789": "{...batch json...}",
    "batch-abc123": "{...batch json...}"
  },
  "token": "next-page-token"
}

---

PUT /api/v1/batches/{pipeline_id}/{batch_id}/stop

Stops a running batch.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID
batch_id	string	Yes	Batch ID

Response

200 OK

{
  "message": "Batch stopped successfully"
}

---

DELETE /api/v1/batches/{pipeline_id}/{batch_id}

Deletes a batch. Running batches are stopped first.

Path Parameters

Parameter	Type	Required	Description
pipeline_id	string	Yes	Pipeline ID
batch_id	string	Yes	Batch ID

Response

200 OK

{
  "message": "Batch deleted successfully"
}

Usage Examples

cURL

# Execute batch
curl -X POST https://api.dhub.io/api/v1/batches/pipeline-abc123 \
  -H "Authorization: Bearer <access_token>"

# Get latest batch status
curl https://api.dhub.io/api/v1/batches/pipeline-abc123 \
  -H "Authorization: Bearer <access_token>"

# Stop batch
curl -X PUT https://api.dhub.io/api/v1/batches/pipeline-abc123/batch-xyz789/stop \
  -H "Authorization: Bearer <access_token>"

# Delete batch
curl -X DELETE https://api.dhub.io/api/v1/batches/pipeline-abc123/batch-xyz789 \
  -H "Authorization: Bearer <access_token>"

Batch Status Polling

async function pollBatchStatus(pipelineId, batchId, interval = 5000) {
  while (true) {
    const response = await fetch(`/api/v1/batches/${pipelineId}/${batchId}`, {
      headers: { 'Authorization': `Bearer ${accessToken}` }
    });
    const batch = await response.json();
    
    console.log(`Batch state: ${batch.state.state}`);
    
    // Check step states
    for (const [stepName, stepState] of Object.entries(batch.state.steps)) {
      console.log(`  ${stepName}: ${stepState.state}`);
    }
    
    // Check if completed
    const allDone = Object.values(batch.state.steps)
      .every(s => s.state === 'success' || s.state === 'failed');
    
    if (allDone || batch.state.state === 'stop') {
      console.log('Batch completed');
      break;
    }
    
    await new Promise(resolve => setTimeout(resolve, interval));
  }
}