Skip to main content

PipelineData

Shared request and response schema. Request: Pipeline port reference accepted in a pipeline definition. Exactly one of dataset, entity, relation, source, view, or variable must be set. Sources and views are input-only and are rejected for output ports. Variables are limited to Python steps, cannot be named event or objects, and do not support version, row_filters, or column_masks. Response: Pipeline port reference returned in a pipeline definition. Exactly one of dataset, entity, relation, or variable must be set because source and view are input-only. Variables are limited to Python steps, cannot be named event or objects, and do not support version, row_filters, or column_masks.

dataset object

Referenced Dataset (Delta Lake table) ID

anyOf
string
entity object

Referenced entity type identifier.

anyOf
string
relation object

Referenced relation type identifier.

anyOf
string
source object

Request: Referenced source identifier. Sources can only be used as pipeline inputs. Response: Source identifier. Sources are input-only, so pipeline outputs reject this field.

anyOf
string
view object

Request: Referenced view identifier. Views can only be used as pipeline inputs. Response: View identifier. Views are input-only, so pipeline outputs reject this field.

anyOf
string
variable object

In-memory inter-step channel name (not a stored resource). A step output publishes its value to contexts[]; a downstream step input reads it back. Python steps only.

anyOf
string
version object

Dataset version (Delta only; ignored for entity/relation/source)

anyOf
integer
options object
anyOf
object
row_filters object

Predicate applied by the source when reading data, or null when no predicate applies.

anyOf
kindKind (string)

Possible values: [and, or]

Default value: and
conditions object[]

Nested conditions (leaf RowFilterCondition or group RowFilter)

  • Array [
    • anyOf
    kindKind (string)
    Constant value: condition
    Default value: condition
    columnColumn (string)required

    Column name to filter on

    operatorRowFilterOperator (string)required

    Allowed Row Filter Operator values. Values: eq, ne, gt, gte, lt, lte, in, not_in, and 4 more.

    Possible values: [eq, ne, gt, gte, lt, lte, in, not_in, between, like, is_null, is_not_null]

    value object

    Filter value. Single value for scalar operators, list for 'in'/'not_in', 2-element list for 'between'. Omit for 'is_null'/'is_not_null'.

    anyOf
  • ]
    • column_masks object

    Projection and masking rules applied by the source. The keys select columns and the values define masks; null keeps every column without masking.

    anyOf
    object
    PipelineData
    {
      "dataset": "string",
      "entity": "string",
      "relation": "string",
      "source": "string",
      "view": "string",
      "variable": "string",
      "version": 0,
      "options": {},
      "row_filters": {
        "kind": "and",
        "conditions": [
          {
            "kind": "condition",
            "column": "string",
            "operator": "eq"
          },
          null
        ]
      },
      "column_masks": {}
    }