Debugging
This guide explains tools and methods for analyzing and resolving issues that occur during pipeline execution. D.Hub records execution history at the pipeline, batch, and step levels to support systematic debugging.
Execution History (Traces)
Detailed execution records (Traces) are automatically saved each time a pipeline runs.
Execution History Structure
The execution history is organized into three levels:
| Level | Description |
|---|---|
| Pipeline | The overall execution unit for the pipeline |
| Batch | An individual execution instance of the pipeline (a new batch is created for each run) |
| Step | The execution unit for each node within a batch |
Viewing Execution History
- Click a specific execution run in the Run History Bar at the top of the workflow editor.
- Or select a pipeline from the pipeline list and open the execution history tab.
Information available for each execution run:
| Item | Description |
|---|---|
| Batch ID | Unique identifier for the execution instance |
| Start/End Time | Execution start and completion time |
| Duration | Total time taken for the execution |
| Status | Execution result (SUCCESS, FAILED, etc.) |
| Trigger | Execution cause (manual run, schedule, event) |
Batch Status
Each batch has one of the following statuses:
| Status | Description | Follow-up Action |
|---|---|---|
| RUNNING | Currently in execution | Wait for completion or stop with Stop if needed |
| SUCCESS | All steps completed successfully | Check output datasets |
| FAILED | An error occurred in one or more steps | Analyze the error, fix, and re-run |
| CANCELLED | Execution cancelled by user | Re-run if needed |
Error Analysis
Checking Step-level Logs
Steps for analyzing a failed batch:
- Select the failed batch: Click the batch shown in red (failed) in the Run History.
- Identify the failed step: Look for nodes with red borders on the canvas.
- Check the Inspector panel: Click the failed node to view error details in the Inspector panel.
- Check the History tab: Review the batch execution history and detailed logs in the Code node's History tab.
Interpreting Error Messages
Error information available in the Inspector panel:
| Item | Description |
|---|---|
| Error Type | Error type (ConnectionError, ValueError, etc.) |
| Error Message | Message describing the error cause |
| Stack Trace | Detailed call stack for Python code |
| Timestamp | Time when the error occurred |
Common Error Types and Solutions
| Error Type | Cause | Solution |
|---|---|---|
| ConnectionError | Failed to connect to data source (DB, API) | Check network status, credentials, endpoint address |
| SchemaError | Input/output data schema mismatch | Verify column names and data type mapping |
| SyntaxError | Python/SQL code syntax error | Fix the code syntax and re-run |
| RuntimeError | Exception during code execution | Analyze the stack trace and fix the logic |
| TimeoutError | Execution time exceeded | Optimize queries, reduce data size, adjust timeout values |
| ResourceError | Insufficient CPU/memory resources | Increase resource limits or process data in smaller batches |
If errors keep recurring in a Code node, adding logging (print or logging) within the code to check intermediate data states can help identify the cause more quickly.
Step-level Detailed Traces
The following information is available in each step's trace:
| Item | Description |
|---|---|
| Execution Time | Duration from step start to end |
| Input Record Count | Number of records input to the step |
| Output Record Count | Number of records output from the step |
| Logs | Log messages output during step execution |
| Errors | Detailed information about errors that occurred |
Re-running Batches
You can re-run a failed batch after making corrections.
How to Re-run
- Analyze the failure cause and fix the code or settings.
- Save the changes.
- Click the Run button on the top toolbar to create a new batch.
A new batch is created when re-running. The history of the previously failed batch is preserved, so it can be used for comparative analysis.
Debugging Best Practices
- Log review order: Narrow the scope by analyzing in the order Pipeline → Batch → Step.
- Incremental testing: For complex pipelines, add nodes one at a time and verify each stage.
- Test with small data: Test with a small sample dataset before running on the full dataset.
- Compare with previous successful batches: Compare input data and settings between the failed batch and the last successful batch.