diff --git a/pages/data-orchestrator/quickstart.mdx b/pages/data-orchestrator/quickstart.mdx index 0d3c7a32cb..ac4b8cbdf9 100644 --- a/pages/data-orchestrator/quickstart.mdx +++ b/pages/data-orchestrator/quickstart.mdx @@ -31,7 +31,10 @@ Follow this guide to create your first workflow and take it through its full lif Data Orchestrator requires the user to upload a YAML file to use as a workflow definition. - This is a basic example of a sequential workflow using three Serverless Jobs. For more examples, see the [Action tasks](/data-orchestrator/reference-content/action-tasks/) and [Logic tasks](/data-orchestrator/reference-content/logic-tasks/) documentation. + This is a basic example of a sequential workflow using three Serverless Jobs. Remember to replace the placeholders with the appropriate values. + Replace the placeholders with the appropriate values. The `namespace` and `name` fields are required, and can be used as metadata to help organize YAML files. + + For more examples, see the [Action tasks](/data-orchestrator/reference-content/action-tasks/) and [Logic tasks](/data-orchestrator/reference-content/logic-tasks/) documentation. 1. Create a YAML file, copy and paste the following content into the file. @@ -64,11 +67,8 @@ Data Orchestrator requires the user to upload a YAML file to use as a workflow d id: "" region: "fr-par" ``` -2. Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. - - The fields `namespace` and `name` are required. They can be used as metadata to help organize your YAML files. - -3. Save the YAML file. + +2. Save the YAML file. ## Create a Data Orchestrator workflow diff --git a/pages/data-orchestrator/reference-content/action-tasks.mdx b/pages/data-orchestrator/reference-content/action-tasks.mdx index e69fa8116a..277dbc4a15 100644 --- a/pages/data-orchestrator/reference-content/action-tasks.mdx +++ b/pages/data-orchestrator/reference-content/action-tasks.mdx @@ -3,16 +3,25 @@ title: Workflow definition action tasks description: Learn how to use the action-tasks format in YAML workflow definition files for Scaleway Data Orchestrator. tags: data-orchestrator workflow create action tasks dates: + validation: 2026-08-01 posted: 2026-06-22 --- Data Orchestrator in Code view requires that you upload a YAML file to use as a workflow definition. +This page lists the action tasks specifically developed for Data Orchestrator workflow definitions. +Data Orchestrator is based on the [Serverless workflow.io](https://serverlessworkflow.io/) standard. For information on other action tasks, see the [Open Workflow DSL Reference](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md). -This page lists the different action tasks supported in Data Orchestrator workflow definitions. +In the following examples, replace the placeholders with the appropriate values. The `namespace` and `name` fields are required, and can be used as metadata to help organize YAML files. +## Action tasks - This is a basic example of a workflow calling one Serverless Job. + The following example shows a workflow that calls a Serverless Job. Copy Serverless Job IDs from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. + + + The workflow and the Serverless Job must be in the same project. + + ```yaml document: dsl: 1.0.3 @@ -20,20 +29,27 @@ This page lists the different action tasks supported in Data Orchestrator workfl name: version: 1.0.0 do: - - task: - call: "serverless_job" - metadata: - description: "The first job" - with: - id: "" - region: "fr-par" + - callServerlessJob: + call: "serverless_job" + metadata: + description: "The first job" + with: + id: "" + region: "fr-par" ``` - Replace the ID placeholder with the ID of your Serverless Job. You can copy it from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. - - This is an example of a how to get a Serverless Function. + The Serverless Function call lets you to call the default endpoint of a Serverless Functions. + + Instead of using an HTTP endpoint (with potential authentication), you provide the function identifier (`id`) and the region (`region`) to invoke the function. The call handles authentication automatically. + + The following example shows how to call a Serverless Function. Copy Serverless Function IDs from the [Serverless Functions page](https://console.scaleway.com/functions/) of the console. + + + The workflow and the Serverless Function must be in the same project. + + ```yaml document: dsl: 1.0.3 @@ -41,17 +57,119 @@ This page lists the different action tasks supported in Data Orchestrator workfl name: version: 1.0.0 do: - - callFunctionGet: + - callServerlessFunction: call: "serverless_function" with: method: GET id: "" - region: fr-par + region: "fr-par" + output: "response" + ``` + + + + The Serverless Container call lets you to call the default endpoint of a Serverless Containers. + + Instead of using an HTTP endpoint (with potential authentication), you provide the container identifier (`containerId`) - or the container name (`name`) together with the namespace identifier (`namespaceName` or `namespaceId`) - and region (`region`) to invoke the container. The call handles authentication automatically. + + The following example shows how to call a Serverless Container. Copy Serverless Container IDs from the [Serverless Containers page](https://console.scaleway.com/containers/) of the console. + + The workflow and the Serverless Container must be in the same project. + + + ```yaml + document: + dsl: 1.0.3 + namespace: + name: + version: 1.0.0 + do: + - callServerlessContainer: + call: "serverless_container" + with: + method: GET + id: "" + region: "fr-par" + output: "response" + ``` + + + + The HTTP call lets you to call any external service over HTTP. + The following example shows how to make an HTTP call. + + ```yaml + document: + dsl: 1.0.3 + namespace: + name: + version: 1.0.0 + do: + - httpCall: + call: "http" + with: + method: GET + url: "" output: "response" - - setOutput: - set: - result: "${ . }" ``` - Replace the ID placeholder with the ID of your Serverless Function. You can copy it from the [Serverless Functions page](https://console.scaleway.com/functions/) of the console. + +## Properties + + + + **Workflow definition properties** + + | Property name | Type | Required | Description | + | :----------------- | :----- | :------- | :----------------------------------------------------------------------------------------------------- | + | `id` | string | yes | The ID of the Serverless Job. | + | `region` | string | yes | The region where the Serverless Job is deployed.
Supported values: `fr-par`, `nl-ams`, `pl-waw`. | + + + + **Workflow definition properties** + + | Property name | Type | Required | Description | + | :----------------- | :--------------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | + | `id` | string | yes | The ID of the Serverless Function. | + | `region` | string | yes | The region where the Serverless Function is deployed. | + | `method` | string | yes | The HTTP request method. | + | `headers` | map | no | A name/value mapping of the HTTP headers to use, if any. | + | `body` | any | no | The HTTP request body, if any. | + | `query` | map[string, any] | no | A name/value mapping of the query parameters to use, if any. | + | `output` | string | no | The output format of the HTTP call.
Supported values are:
- `content`: output the content of the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response), possibly deserialized.
- `raw`: output the base-64 encoded [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response) content, if any.
- `response`: output the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response).
Defaults to `content`. | + + + + **Workflow definition properties** + + | Property name | Type | Required | Description | | + | :----------------- | :--------------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | + | `containerId` | string | no | The ID of the Serverless Container.
Required if `name` is not set. | | + | `name` | string | no | The name of the Serverless Container.
Required if `containerId` is not set. | | + | `namespaceId` | string | no | The ID of the Serverless Container namespace.
Required if both `containerId` and `namespaceName` are not set. | | + | `namespaceName` | string | no | The name of the Serverless Container namespace.
Required if both `containerId` and `namespaceId` are not set. | | + | `region` | string | yes | The Scaleway region where the container is deployed.
Supported values: `fr-par`, `nl-ams`, `pl-waw`. | | + | `method` | string | yes | The HTTP request method. | | + | `headers` | map | no | A name/value mapping of the HTTP headers to use, if any. | | + | `body` | any | no | The HTTP request body, if any. | | + | `query` | map[string, any] | no | A name/value mapping of the query parameters to use, if any. | | + | `output` | string | no | The output format of the HTTP call.
Supported values are:
- `content`: output the content of the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response), possibly deserialized.
- `raw`: output the base-64 encoded [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response) content, if any.
- `response`: output the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response).
Defaults to `content`. | | + + + + **Workflow definition properties** + + | Property name | Type | Required | Description | | | | + | :----------------- | :----------------- | :------- | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --- | --- | --- | + | `method` | string | yes | The HTTP request method. | | | | + | `endpoint` | string\|[endpoint] | yes | An URI or object that describes the HTTP endpoint to call. | | | | + | `headers` | map | no | A name/value mapping of the HTTP headers to use, if any. | | | | + | `body` | any | no | The HTTP request body, if any. | | | | + | `query` | map[string, any] | no | A name/value mapping of the query parameters to use, if any? | | | | + | `output` | string | no | The output format of the HTTP call.
Supported values are:
- `content`: output the content of the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response), possibly deserialized.
- `raw`: output the base-64 encoded [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response) content, if any.
- `response`: output the [HTTP response](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#http-response).
Defaults to `content`. | | | | + | `redirect` | boolean | no | Specifies whether to include or exclude status codes `300–399` from the error range.
- If set to `true`, runtimes raise an error for status codes outside the `200–399` range.
- If set to `false`, runtimes raise an error for response status codes outside the `200–299` range.
Defaults to `false`. | | | | + + + \ No newline at end of file diff --git a/pages/data-orchestrator/reference-content/logic-tasks.mdx b/pages/data-orchestrator/reference-content/logic-tasks.mdx index 3c5b3ed577..7f67ee2747 100644 --- a/pages/data-orchestrator/reference-content/logic-tasks.mdx +++ b/pages/data-orchestrator/reference-content/logic-tasks.mdx @@ -3,16 +3,21 @@ title: Workflow definition logic tasks description: Learn the logic tasks format for YAML Workflow definition files in Scaleway Data Orchestrator. tags: data-orchestrator workflow create logic tasks dates: + validation: 2026-08-01 posted: 2026-05-18 --- Data Orchestrator in Code view requires the user to upload a YAML file to use as a workflow definition. -This page explains how to use logic tasks to create more complex workflows. +This page explains how to use logic tasks to create more complex workflows. Data Orchestrator is based on the [Serverless workflow.io](https://serverlessworkflow.io/) standard. For information on other tasks, see the [Open Workflow DSL Reference](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md). + +In the following examples, replace the placeholders with the appropriate values. The `namespace` and `name` fields are required, and can be used as metadata to help organize YAML files. + +## Logic tasks - This is a basic example of a sequential workflow using three Serverless Jobs. + The following example shows a sequential workflow using three Serverless Jobs. ```yaml document: dsl: 1.0.3 @@ -42,12 +47,12 @@ This page explains how to use logic tasks to create more complex workflows. id: "" region: "fr-par" ``` - Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. - - A **Fork** lets you run multiple tasks at the same time in their respective branches. The following example runs 2 Serverless Jobs in parallel. + A **Fork** task lets you run multiple tasks at the same time in their respective branches. + + The following example runs 2 Serverless Jobs in parallel. ```yaml document: dsl: 1.0.3 @@ -74,11 +79,12 @@ This page explains how to use logic tasks to create more complex workflows. id: "" region: "fr-par" ``` - Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. - A **Switch** lets your workflows dynamically select different paths based on specified conditions or criteria. The following example runs a conditional or default Serverless Job depending on 2 preset conditions. + A **Switch** task lets your workflows dynamically select different paths based on specified conditions or criteria. + + The following example runs a conditional or default Serverless Job depending on 2 preset conditions. ```yaml document: dsl: 1.0.3 @@ -121,11 +127,12 @@ This page explains how to use logic tasks to create more complex workflows. region: "fr-par" then: exit ``` - Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. - A **Try-catch** lets you handle known error types, such as authentication, authorization, and timeout. You can retry a failed task a set number of times before falling back to an alternative task if it keeps failing. The following example tries a Serverless Job up to 4 times. If all attempts fail, a fallback task is executed. + A **Try-catch** task lets you handle known error types, such as authentication, authorization, and timeout. You can retry a failed task a set number of times before falling back to an alternative task if it keeps failing. + + The following example tries a Serverless Job up to 4 times. If all attempts fail, a fallback task is executed. ```yaml document: dsl: 1.0.3 @@ -156,7 +163,6 @@ This page explains how to use logic tasks to create more complex workflows. id: "" region: "fr-par" ``` - Replace the ID placeholders with the IDs of your Serverless Jobs. You can copy them from the [Serverless Jobs page](https://console.scaleway.com/serverless-jobs/) of the console. @@ -166,16 +172,16 @@ This page explains how to use logic tasks to create more complex workflows. **Workflow definition properties** - | Property name | Type | Required | Description | - | :----------------- | :------------------- | :------- | :---------------------------------------------------------- | - | `dsl` | string | yes | The version of the DSL used to define the workflow | - | `namespace` | string | yes | The namespace of the workflow | - | `name` | string | yes | The name of the workflow | - | `version` | string | yes | The [semantic version](https://semver.org/) of the workflow | - | `title` | string | no | The title of the workflow | - | `summary` | string | no | The Markdown summary of the workflow | - | `tags` | map [string, string] | no | A key/value mapping of the tags of the workflow, if any | - | `metadata` | map | no | Additional information about the workflow | + | Property name | Type | Required | Description | + | :----------------- | :------------------- | :------- | :----------------------------------------------------------- | + | `dsl` | string | yes | The version of the DSL used to define the workflow. | + | `namespace` | string | yes | The namespace of the workflow. | + | `name` | string | yes | The name of the workflow. | + | `version` | string | yes | The [semantic version](https://semver.org/) of the workflow. | + | `title` | string | no | The title of the workflow. | + | `summary` | string | no | The Markdown summary of the workflow. | + | `tags` | map [string, string] | no | A key/value mapping of the tags of the workflow, if any. | + | `metadata` | map | no | Additional information about the workflow. | @@ -184,7 +190,7 @@ This page explains how to use logic tasks to create more complex workflows. | Property name | Type | Required | Description | | :----------------- | :---------------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | `branches` | map [string, task] | no | The tasks to perform concurrently | + | `branches` | map [string, task] | no | The tasks to perform concurrently. | | `compete` | boolean | no | - If `false` (default), returns an array of outputs from all branches, in declaration order.
- If `true`, only the branches following the first task to complete successfully will continue running. All other competing tasks are terminated whether they succeed or not. | @@ -192,9 +198,9 @@ This page explains how to use logic tasks to create more complex workflows. **Switch properties** - | Property name | Type | Required | Description | - | :----------------- | :------------------- | :------- | :----------------------------------------------------- | - | `case` | map [string, string] | yes | A name/value map of the cases to switch on (see below) | + | Property name | Type | Required | Description | + | :----------------- | :------------------- | :------- | :------------------------------------------------------ | + | `case` | map [string, string] | yes | A name/value map of the cases to switch on (see below). | **Switch > Case properties** @@ -203,7 +209,7 @@ A switch case defines a condition to evaluate and an action to execute when that | Property name | Type | Required | Description | | :----------------- | :-------------- | :------- | :------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `when` | string | no | A runtime expression used to determine whether or not the case matches.
If not set, the case will be matched by default if no other case matches.
There can be only one default case, all others must set a condition. | - | `then` | `flowDirective` | yes | The flow directive to execute when the case matches | + | `then` | `flowDirective` | yes | The flow directive to execute when the case matches. | **Switch > Case > Flow directives** @@ -212,10 +218,10 @@ A switch case defines a condition to evaluate and an action to execute when that | Flow directive | Description | | :------------------ | :--------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | `continue` | Instruct the workflow to proceed with the next task in line. This may conclude the execution of a workflow or branch if there is no task defined after the `continue`. | - | `end` | Conclude the workflow execution and signal its completion with a `succeeded` status | + | `end` | Conclude the workflow execution and signal its completion with a `succeeded` status. | | `exit` | Complete the current parent `do` execution. This may conclude the execution of a workflow or branch if there is no task defined after the parent `do` scope. | - | `string` | Continues the workflow at the task with the specified name | - | `` | Redirect to a task declared within the scope | + | `string` | Continues the workflow at the task with the specified name. | + | `` | Redirect to a task declared within the scope. | Flow directives may only redirect to tasks declared within their own scope. They cannot target tasks at a different depth. @@ -226,10 +232,10 @@ A switch case defines a condition to evaluate and an action to execute when that **Try properties** - | Property name | Type | Required | Description | - | :----------------- | :---------------------- | :------- | :----------------------------------------- | - | `try` | map [string, task] | yes | The tasks to perform | - | `catch` | `catch` | yes | The errors to catch and how to handle them | + | Property name | Type | Required | Description | + | :----------------- | :---------------------- | :------- | :------------------------------------------ | + | `try` | map [string, task] | yes | The tasks to perform. | + | `catch` | `catch` | yes | The errors to catch and how to handle them. | **Try > Catch properties** @@ -237,12 +243,12 @@ A switch case defines a condition to evaluate and an action to execute when that | Property name | Type | Required | Description | | :----------------- | :----------------------- | :------- | :----------------------------------------------------------------------------------------------------------------------------------------------------------------- | - | `errors` | `errorFilter` | no | The definition of the errors to catch | - | `as` | string | no | The name of the runtime expression variable to save the error as. Defaults to 'error'. | - | `when` | string | no | A runtime expression used to determine whether or not to catch the filtered error | - | `exceptWhen` | string | no | A runtime expression used to determine whether or not to catch the filtered error | + | `errors` | `errorFilter` | no | The definition of the errors to catch. | + | `as` | string | no | The name of the runtime expression variable to save the error as. Defaults to `error`. | + | `when` | string | no | A runtime expression used to determine whether or not to catch the filtered error. | + | `exceptWhen` | string | no | A runtime expression used to determine whether or not to catch the filtered error. | | `retry` | `retryPolicy`
string | no | The `retry policy` to use, if any, when catching `errors`.
If a `string`, it must be the name of a retry policy defined in the workflow's reusable components. | - | `do` | map[string, task] | no | The definition of the task(s) to run when catching an error | + | `do` | map[string, task] | no | The definition of the task(s) to run when catching an error. | See the [Error properties](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#error) and [Retry policy properties](https://github.com/open-workflow-specification/specification/blob/main/dsl-reference.md#retry) for more details.