PipelineStep

Shared request and response schema. Request: Pipeline step accepted in a definition. Exactly one of script or transform must be set. Transform configuration refers to inputs and outputs by port key, and transform steps cannot use secrets. Response: Pipeline step returned in a definition. Exactly one of script or transform is set. Transform configuration refers to inputs and outputs by port key, and transform steps do not use secrets.

nameName (string)required

Step name

alias object

anyOf

• string
• null

string

description object

Step-owned description, independent of the referenced Code's description so steps sharing one Code can be labeled distinctly.

anyOf

• string
• null

string

script object

Code resource ID to execute (code step)

anyOf

• string
• null

string

transform object

Native transform spec (transform step)

anyOf

• object
• null

oprequired

Possible values: [aggregate, join, rename_cast, select_filter, union]

aggregate

versionVersion (integer)

Constant value: 1

Default value: 1

config objectrequired

Config for the aggregate op: group-by + allowlisted aggregate functions. An empty group_by aggregates the whole input to a single row. input is a port key.

inputInput (string)required

Input port key

group_bystring[]

Grouping columns (empty = whole-input aggregate)

aggregations object[]required

Aggregations (non-empty)

Array [

functionFunction (string)required

Allowlisted aggregate function

Possible values: [sum, count, avg, min, max]

column object

Aggregated column (omit only for count)

anyOf

• string
• null

string

aliasAlias (string)required

Output column name

]

join

versionVersion (integer)

Constant value: 1

Default value: 1

config objectrequired

Two-input equi-join configuration. left_input and right_input are pipeline port keys, keys contains at least one equality pair, and columns explicitly selects the output. Other join predicates are not accepted.

left_inputLeft Input (string)required

Left input port key

right_inputRight Input (string)required

Right input port key

join_typeJoin Type (string)required

inner | left | right | full

Possible values: [inner, left, right, full]

keys object[]required

Equi-join column pairs (non-empty)

Array [

leftLeft (string)required

Join column on the left input

rightRight (string)required

Join column on the right input

]

columns object[]required

Output projection (non-empty)

Array [

inputInput (string)required

Source input port key (left_input or right_input)

nameName (string)required

Source column name on that input

alias object

Output column name (defaults to name)

anyOf

• string
• null

string

]

rename_cast

versionVersion (integer)

Constant value: 1

Default value: 1

config objectrequired

Config for the rename_cast op. Listed columns are renamed / cast; columns not listed pass through unchanged. input is a port key.

inputInput (string)required

Input port key

columns object[]required

Column rename / cast rules (non-empty)

Array [

nameName (string)required

Source column name

alias object

New column name (defaults to name)

anyOf

• string
• null

string

cast object

Allowlisted target type

anyOf

• string
• null

string

Possible values: [utf8, int64, float64, bool]

]

select_filter

versionVersion (integer)

Constant value: 1

Default value: 1

config objectrequired

Select-and-filter transform configuration accepted in a request. input is a pipeline port key, not a resource identifier; columns=null keeps every column, and where=null applies no predicate.

inputInput (string)required

Input port key (a PipelineStep.inputs key)

columns object

Column names returned for this resource.

anyOf

• object[]
• null

Array [

nameName (string)required

Source column name on the input

alias object

Output column name (defaults to name)

anyOf

• string
• null

string

]

where object

Structured row predicate

anyOf

• RowFilter
• null

kindKind (string)

Possible values: [and, or]

Default value: and

conditions object[]

Nested conditions (leaf RowFilterCondition or group RowFilter)

Array [

anyOf

• RowFilterCondition
• Option 2

kindKind (string)

Constant value: condition

Default value: condition

columnColumn (string)required

Column name to filter on

operatorRowFilterOperator (string)required

Allowed Row Filter Operator values. Values: eq, ne, gt, gte, lt, lte, in, not_in, and 4 more.

Possible values: [eq, ne, gt, gte, lt, lte, in, not_in, between, like, is_null, is_not_null]

value object

Filter value. Single value for scalar operators, list for 'in'/'not_in', 2-element list for 'between'. Omit for 'is_null'/'is_not_null'.

anyOf

• Option 1
• null

]

union

versionVersion (integer)

Constant value: 1

Default value: 1

config objectrequired

Union configuration for two or more pipeline input ports. Inputs are aligned by column name, missing values become null, and duplicate rows are retained.

inputsstring[]required

Input port keys to union (>= 2)

inputs object

Input dataset mapping

anyOf

• object
• null

object

outputs object

Output dataset mapping

anyOf

• object
• null

object

dependencies object

List of dependent step names

anyOf

• string[]
• null

Array [

string

]

options object

anyOf

• object
• null

object

secrets object

Secret values to inject into contexts['secrets']

anyOf

• object[]
• null

Array [

nameName (string)required

contexts['secrets'] key the step reads

refRef (string)required

Secret reference, e.g. secret://secret./token

]

PipelineStep

{
  "name": "string",
  "alias": "string",
  "description": "string",
  "script": "string",
  "transform": {
    "version": 1,
    "op": "select_filter",
    "config": {
      "input": "string",
      "columns": [
        {
          "name": "string",
          "alias": "string"
        }
      ],
      "where": {
        "kind": "and",
        "conditions": [
          {
            "kind": "condition",
            "column": "string",
            "operator": "eq"
          },
          null
        ]
      }
    }
  },
  "inputs": {},
  "outputs": {},
  "dependencies": [
    "string"
  ],
  "options": {},
  "secrets": [
    {
      "name": "string",
      "ref": "string"
    }
  ]
}