PipelineStep

Shared request and response schema. Request: Pipeline step accepted in a definition. Exactly one of script or transform must be set. Transform configuration refers to inputs and outputs by port key, and transform steps cannot use secrets. Response: Pipeline step returned in a definition. Exactly one of script or transform is set. Transform configuration refers to inputs and outputs by port key, and transform steps do not use secrets.

nameName (string)필수

Step name

alias object

하나 이상(anyOf)

• string
• null

string

description object

Step-owned description, independent of the referenced Code's description so steps sharing one Code can be labeled distinctly.

하나 이상(anyOf)

• string
• null

string

script object

Code resource ID to execute (code step)

하나 이상(anyOf)

• string
• null

string

transform object

Native transform spec (transform step)

하나 이상(anyOf)

• object
• null

op필수

Possible values: [aggregate, join, rename_cast, select_filter, union]

aggregate

versionVersion (integer)

상수 값: 1

기본값: 1

config object필수

Config for the aggregate op: group-by + allowlisted aggregate functions. An empty group_by aggregates the whole input to a single row. input is a port key.

inputInput (string)필수

Input port key

group_bystring[]

Grouping columns (empty = whole-input aggregate)

aggregations object[]필수

Aggregations (non-empty)

Array [

functionFunction (string)필수

Allowlisted aggregate function

Possible values: [sum, count, avg, min, max]

column object

Aggregated column (omit only for count)

하나 이상(anyOf)

• string
• null

string

aliasAlias (string)필수

Output column name

]

join

versionVersion (integer)

상수 값: 1

기본값: 1

config object필수

Two-input equi-join configuration. left_input and right_input are pipeline port keys, keys contains at least one equality pair, and columns explicitly selects the output. Other join predicates are not accepted.

left_inputLeft Input (string)필수

Left input port key

right_inputRight Input (string)필수

Right input port key

join_typeJoin Type (string)필수

inner | left | right | full

Possible values: [inner, left, right, full]

keys object[]필수

Equi-join column pairs (non-empty)

Array [

leftLeft (string)필수

Join column on the left input

rightRight (string)필수

Join column on the right input

]

columns object[]필수

Output projection (non-empty)

Array [

inputInput (string)필수

Source input port key (left_input or right_input)

nameName (string)필수

Source column name on that input

alias object

Output column name (defaults to name)

하나 이상(anyOf)

• string
• null

string

]

rename_cast

versionVersion (integer)

상수 값: 1

기본값: 1

config object필수

Config for the rename_cast op. Listed columns are renamed / cast; columns not listed pass through unchanged. input is a port key.

inputInput (string)필수

Input port key

columns object[]필수

Column rename / cast rules (non-empty)

Array [

nameName (string)필수

Source column name

alias object

New column name (defaults to name)

하나 이상(anyOf)

• string
• null

string

cast object

Allowlisted target type

하나 이상(anyOf)

• string
• null

string

Possible values: [utf8, int64, float64, bool]

]

select_filter

versionVersion (integer)

상수 값: 1

기본값: 1

config object필수

Select-and-filter transform configuration accepted in a request. input is a pipeline port key, not a resource identifier; columns=null keeps every column, and where=null applies no predicate.

inputInput (string)필수

Input port key (a PipelineStep.inputs key)

columns object

Column names returned for this resource.

하나 이상(anyOf)

• object[]
• null

Array [

nameName (string)필수

Source column name on the input

alias object

Output column name (defaults to name)

하나 이상(anyOf)

• string
• null

string

]

where object

Structured row predicate

하나 이상(anyOf)

• RowFilter
• null

kindKind (string)

Possible values: [and, or]

기본값: and

conditions object[]

Nested conditions (leaf RowFilterCondition or group RowFilter)

Array [

하나 이상(anyOf)

• RowFilterCondition
• Option 2

kindKind (string)

상수 값: condition

기본값: condition

columnColumn (string)필수

Column name to filter on

operatorRowFilterOperator (string)필수

Allowed Row Filter Operator values. Values: eq, ne, gt, gte, lt, lte, in, not_in, and 4 more.

Possible values: [eq, ne, gt, gte, lt, lte, in, not_in, between, like, is_null, is_not_null]

value object

Filter value. Single value for scalar operators, list for 'in'/'not_in', 2-element list for 'between'. Omit for 'is_null'/'is_not_null'.

하나 이상(anyOf)

• Option 1
• null

]

union

versionVersion (integer)

상수 값: 1

기본값: 1

config object필수

Union configuration for two or more pipeline input ports. Inputs are aligned by column name, missing values become null, and duplicate rows are retained.

inputsstring[]필수

Input port keys to union (>= 2)

inputs object

Input dataset mapping

하나 이상(anyOf)

• object
• null

object

outputs object

Output dataset mapping

하나 이상(anyOf)

• object
• null

object

dependencies object

List of dependent step names

하나 이상(anyOf)

• string[]
• null

Array [

string

]

options object

하나 이상(anyOf)

• object
• null

object

secrets object

Secret values to inject into contexts['secrets']

하나 이상(anyOf)

• object[]
• null

Array [

nameName (string)필수

contexts['secrets'] key the step reads

refRef (string)필수

Secret reference, e.g. secret://secret./token

]

PipelineStep

{
  "name": "string",
  "alias": "string",
  "description": "string",
  "script": "string",
  "transform": {
    "version": 1,
    "op": "select_filter",
    "config": {
      "input": "string",
      "columns": [
        {
          "name": "string",
          "alias": "string"
        }
      ],
      "where": {
        "kind": "and",
        "conditions": [
          {
            "kind": "condition",
            "column": "string",
            "operator": "eq"
          },
          null
        ]
      }
    }
  },
  "inputs": {},
  "outputs": {},
  "dependencies": [
    "string"
  ],
  "options": {},
  "secrets": [
    {
      "name": "string",
      "ref": "string"
    }
  ]
}