Starting a Pipeline Run

Use a Data Science pipeline run to run the code of the pipeline steps according to the workflow defined in a pipeline.

Pipeline creation sets the infrastructure and the actual use case steps, but the pipeline run runs the pipeline with the specified parameters. A pipeline run provisions the specified infrastructure, runs the pipeline steps, and then deprovisions and destroys the used resources when the pipeline ends.

You can view the pipeline steps, runs, and logs if they exist. You can also override steps and parameters.

These environment variables control the pipeline.

    1. On the Projects list page, select the project that contains the pipelines that you want to work with. If you need help finding the list page or the project, see Listing Projects.
    2. On the project details page, select Pipelines.
      All pipelines in the selected project are displayed in a table.
    3. On the project details page, select Pipelines.
    4. On the pipeline details page, select Pipeline runs.
    5. Select Start a pipeline run.
    6. On the Start a pipeline run page, select the compartment and enter a name for the pipeline run.
    7. To override configuration of a pipeline step, find the step under Pipeline step configuration override, and then from the Actions menu (three dots) for the step, select Edit. For field descriptions, see Creating a Pipeline.
    8. Override other configuration for this run by updating associated fields. For field descriptions, see Creating a Pipeline.
    9. Select Start.

      The pipeline run is in the Accepted state until the run begins, and then it changes to In Progress. When the run finishes, it's either Succeeded or Failed.

      The status of each pipeline step is listed. Pipeline steps are in the Waiting state until they run, and then they change to In Progress. When a step finishes, it's either Succeeded or Failed.

  • Use the oci data-science pipeline create command and required parameters to create (start) a pipeline run:

    oci data-science pipeline create --compartment-id <compartment_OCID> --pipeline-id <pipeline_OCID> ... [OPTIONS]

    For a complete list of parameters and values for CLI commands, see the CLI Command Reference.

    Example 1: JSON File

    Use the oci data-science pipeline-run create command and required parameters to create (start) a pipeline run:

    oci data-science pipeline-run create --from-json file://create_run.json

    The following example, create_run.json, starts a pipeline run by supplying values for the parameters defined in the pipeline:

    {
     "displayName": "Hello Run",
     "compartmentId": "ocid1.compartment.oc1.......",
     "projectId": "ocid1.datascienceproject.oc1.....",
     "pipelineId": "ocid1.datasciencepipeline.oc1.....",
                                
     # pamaters can be viewed as an interface that allows users to control the behavior of the pipeline
     "parametersOverride": {
      "greeting": "Hey",
      "name": "Alex"
     }
    }

    Example 2: JSON File with Overrides

    1. Start a pipeline run with:
      oci data-science pipeline-run create \
      --display-name <pipeline_run_name> \
      --compartment-id <compartment_ocid> \
      --project-id <project_ocid> \
      --pipeline-id <pipeline_ocid> \
      --configuration-override-details file://<optional_pipeline_run_configuration_override_json_file> \
      --log-configuration-override-details file://<optional_pipeline_run_logging_configuration_override_json_file>
    2. (Optional) Use this pipeline run configuration override JSON file to override the configurations defined on the parent pipeline:
      {
        "pipelineType": "DEFAULT",
        "maximumRuntimeInMinutes": 240,
        "commandLineArguments" : "test-arg",
        "environmentVariables": {
          "SOME_ENV_KEY": "<some_env_value_override>" 
        }
      }
    3. (Optional) Use this pipeline run logging configuration override JSON file to override the logging configuration defined on the parent pipeline:
      {
        "enableLogging": true,
        "enableAutoLogCreation": true,
        "logGroupId": "<log_group_ocid>"
      }
  • Run the CreatePipelineRun operation to create (start) a pipeline run.