Traces API

The Traces API retrieves pipeline execution history. It searches data collected through distributed tracing to check pipeline execution status, elapsed time, and individual step results.

Trace Search Engine

The Traces API uses an internal trace search engine as its backend. In environments where the search engine is not configured, a 503 Service Unavailable response is returned.

Endpoints

Method	Path	Description
GET	/traces/pipelines/{pipeline_id}	Get latest pipeline trace
GET	/traces/pipelines/{pipeline_id}/	List pipeline trace history
GET	/traces/pipelines/{pipeline_id}/{batch_id}	Get trace for a specific batch
GET	/traces/steps/{batch_id}	Get per-step traces for a batch

---

GET /traces/pipelines/{pipeline_id}

Retrieves the most recent execution trace for the pipeline.

Path Parameters

Parameter	Type	Description
pipeline_id	string	Pipeline ID

Response

200 OK

{
  "id": "batch-20240615-001",
  "name": "daily-etl",
  "deployment": "daily-etl/run-20240615",
  "state": {
    "state": "completed",
    "steps": {
      "extract": {
        "state": "completed",
        "offsets": {"records": 1000},
        "comment": null,
        "stacktrace": null,
        "start_time": 1718448000000,
        "end_time": 1718448060000
      },
      "transform": {
        "state": "completed",
        "offsets": {"records": 950},
        "comment": null,
        "stacktrace": null,
        "start_time": 1718448060000,
        "end_time": 1718448120000
      }
    }
  },
  "start_time": 1718448000000,
  "end_time": 1718448120000
}

404 Not Found — When no trace exists for the pipeline

---

GET /traces/pipelines/{pipeline_id}/

Retrieves the full execution history for a pipeline. Sorted by start time in descending order.

Query Parameters

Parameter	Type	Default	Description
limit	integer	100	Maximum number of items to retrieve (max 1000)

Response

200 OK

{
  "items": {
    "batch-20240615-001": {
      "id": "batch-20240615-001",
      "name": "daily-etl",
      "deployment": "daily-etl/run-20240615",
      "state": { ... },
      "start_time": 1718448000000,
      "end_time": 1718448120000
    },
    "batch-20240614-001": {
      "id": "batch-20240614-001",
      "name": "daily-etl",
      "deployment": "daily-etl/run-20240614",
      "state": { ... },
      "start_time": 1718361600000,
      "end_time": 1718361720000
    }
  },
  "token": null
}

---

GET /traces/pipelines/{pipeline_id}/{batch_id}

Retrieves the pipeline execution trace for a specific batch ID.

Path Parameters

Parameter	Type	Description
pipeline_id	string	Pipeline ID
batch_id	string	Batch ID

Response

The response format is the same as the latest trace retrieval.

---

GET /traces/steps/{batch_id}

Retrieves individual step execution traces for a specific batch. Used for checking step-level details.

Path Parameters

Parameter	Type	Description
batch_id	string	Batch ID

Query Parameters

Parameter	Type	Required	Description
step_name	string	No	Filter for a specific step only

Response

200 OK

[
  {
    "span_id": "abc123",
    "span_name": "pipeline.step",
    "span_attributes": {
      "pipeline.step.name": "extract",
      "pipeline.batch.id": "batch-20240615-001"
    },
    "span_start_timestamp_nanos": 1718448000000000000,
    "span_end_timestamp_nanos": 1718448060000000000,
    "span_status": "OK"
  }
]

---

Usage Examples

# Get latest pipeline execution result
curl https://api.dhub.io/api/v1/traces/pipelines/pipeline-abc123 \
  -H "Authorization: Bearer <access_token>"

# Execution history list (last 10)
curl "https://api.dhub.io/api/v1/traces/pipelines/pipeline-abc123/?limit=10" \
  -H "Authorization: Bearer <access_token>"

# Get per-step details for a specific batch
curl https://api.dhub.io/api/v1/traces/steps/batch-20240615-001 \
  -H "Authorization: Bearer <access_token>"

Pipeline Debugging

Find steps with state set to failed in the trace and check the stacktrace field to quickly identify the cause of errors. For detailed debugging methods, refer to the Pipeline Debugging guide.