Containers - Handbook - Scenario stages

Each project execution is executed in steps called stages. The purpose of such division is a logical separation of distinct procedures and, as an effect, clear and straightforward feedback.

The following stages are consecutively performed during the submission execution:

init Initialization

The first stage is devoted to the initial preparation steps.

Example procedures suitable for this stage:

  • creating and populating a database (or any different kind of storage)
  • generating randomized data

build Build

The build stage covers all the procedures related to building application files.

Example procedures suitable for this stage:

  • running make command for multi-file C++ projects with Makefile,
  • building jar file by executing mvn install command for Java projects,
  • fetching external dependencies, e.g., pip install in Python or composer install in PHP.

run Execution

Execution is the crucial stage in this workflow. After two previous stages (initialization and build), the application is ready for execution. All the dependencies are available and the code is built (if it was necessary).

Example procedures suitable for this stage:

  • running unit tests
  • launching web server
  • executing console program with input data stream.

test Evaluation

In case your project is designed as a code assignment or a puzzle, the evaluation is a proper stage for correctness verification.

Example procedures suitable for this stage:

  • checking output files generated during the run stage,
  • validation of unit testing report file.

post Finalization

The last stage is for summarizing the results of all the previous stages. It may also contain operations opposite to those performed in the initialization phase.

Example procedures suitable for this stage:

  • cleaning database,
  • gathering and packing generated files.

Note: Not every project requires all stages. If any step is not present, it will simply be omitted. Although, it is required to define at least one stage.

Stage scripts

A stage is usually defined in a form of a Python or a Bash script. Such scripts are placed among project files and provided in the project configuration (see scenarios.scenario.stages.stage). They are implementing

Note: Instead of bash script you can provide a stage as a shell command. This is especially useful for simple stages without complex logic behind (e.g. a shell command mvn test for the evaluation stage).

From now on, we assume that the run stage is defined as a bash script. Now, we will discuss the requirements and rules for stage scripts.

Common rules and requirements

All stage scripts except for the execution and the finalization should always execute correctly. This means:

Correctness is not required for the execution and finalization stages for two different reasons:

  • the execution (run) stage is dedicated to run the end user's code, so we expect that it may fail or exceed the time limit
  • the finalization (post) stage is mostly a cleanup procedure, and it is executed after establishing the final result

Initialization (init)

There is nothing specific for this stage. It is expected to perform preparation steps if there are any.

Build (build)

The build process should be performed in this step. The Content Manager should decide whether the build process was successful or not. There are two options

  • build process was successful - in this case there is nothing specific to be done,
  • build process failed - in this case the Content Manager should set build_error (BE) status using project utilities.

The build stage script has access to the $SE_PATH_RUNTIME_DATA environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the build stage script:

gcc -o program main.c

COMPILATION_EXIT_CODE="$?"

if [ $COMPILATION_EXIT_CODE -ne 0 ]; then
    echo "Build failed"
    se_utils_cli scenario_result status "BE"
else
    echo "Build successful"
fi

For this stage you can use:

  • CLI tool, for example:
    • access $SE_PATH_STAGE_INIT_STDOUT and $SE_PATH_STAGE_INIT_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "init" stage respectively,
  • Python tool, for example:
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "init" stage using se_utils.stage module.

Execution (run)

This stage is dedicated to executing the end user's code. The Content Manager should expect a wide range of program behaviors. In this case, no behavior is considered to be abnormal - the Sphere Engine Containers will handle many types of failures here (e.g., time limit exceeded, run-time error, excessive use of resources) and allow for further processing in the next stage.

For this stage you can use:

  • CLI tool, for example:
    • access $SE_PATH_STAGE_INIT_STDOUT and $SE_PATH_STAGE_INIT_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "init" stage respectively,
    • access $SE_PATH_STAGE_BUILD_STDOUT and $SE_PATH_STAGE_BUILD_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "build" stage respectively,
  • Python tool, for example:
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "init" stage using se_utils.stage module.
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "build" stage using se_utils.stage module.

Evaluation (test)

In this stage, the Content Manager has all the data required to make a final decision about the user's submission.

To make this decision you can use the following resources:

  • CLI tool, for example:
    • access $SE_PATH_STAGE_INIT_STDOUT and $SE_PATH_STAGE_INIT_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "init" stage respectively,
    • access $SE_PATH_STAGE_BUILD_STDOUT and $SE_PATH_STAGE_BUILD_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "build" stage respectively,
    • access $SE_PATH_STAGE_RUN_STDOUT and $SE_PATH_STAGE_RUN_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "run" stage respectively,
    • scenario result command for final result management,
  • Python tool, for example:
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "init" stage using se_utils.stage module,
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "build" stage using se_utils.stage module,
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "run" stage using se_utils.stage module,
    • se_utils.result module for final result management.

By design, the Sphere Engine Containers makes the final decision in accordance with the following rules:

  • if the "build" stage has ended with build error, then the "run" stage is omitted and the final status is set to "build error"
  • if the "build" stage is successful, then the "run" stage always executes, and:
    • if the "run" stage fails, then the final status is set to "run-time error"
    • if the "run" stage exceeds the time limit, then the final status is set to "time limit exceeded"
    • if the "run" stage is successful, then the final status is set to "success"

In order to overwrite the default behavior, the Content Manager can set the final result using:

  • se_utils.result module of Python package,
  • scenario result command of CLI tool. The evaluation stage script has access to the $SE_PATH_RUNTIME_DATA environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the evaluation stage script:

USER_OUTPUT_PATH="$SE_PATH_STAGE_RUN_STDOUT"
MODEL_OUTPUT_PATH="$SE_PATH_TEST_CASES/0.out"

if cmp --silent "$USER_OUTPUT_PATH" "$MODEL_OUTPUT_PATH"; then
  echo "[info] user output is identical to model output"
  SOLUTION_CORRECT="1"
else
  echo "[info] user output differs from model output"
  SOLUTION_CORRECT="0"
fi

if [[ "$SOLUTION_CORRECT" != "1" ]]; then
    echo "[info] setting final status as fail"
    se_utils_cli scenario_result status "FAIL"
fi

In this stage you can also set other final result parameters (final execution time, final score) using:

Finalization (post)

This stage is dedicated to summarizing and clean-up procedures. It is allowed to fail without failing the entire submissions, but this is not recommended. For potential information about failure during this stage please refer to:

  • stage finalization output,
  • stage finalization error,
  • debug log.

In this stage you can use:

  • CLI tool, for example:
    • access $SE_PATH_STAGE_INIT_STDOUT and $SE_PATH_STAGE_INIT_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "init" stage respectively,
    • access $SE_PATH_STAGE_BUILD_STDOUT and $SE_PATH_STAGE_BUILD_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "build" stage respectively,
    • access $SE_PATH_STAGE_RUN_STDOUT and $SE_PATH_STAGE_RUN_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "run" stage respectively,
    • access $SE_PATH_STAGE_TEST_STDOUT and $SE_PATH_STAGE_TEST_STDERR environmental variables; they are holding paths to files with the stdout and the stderr data from the "run" stage respectively,
  • Python tool, for example:
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "init" stage using se_utils.stage module,
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "build" stage using se_utils.stage module,
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "run" stage using se_utils.stage module,
    • access the stdout data (or path to file) and the stderr data (or path to file) from the "test" stage using se_utils.stage module.

Feedback from stages

The Sphere Engine Containers module handles the output data of each stage separately. When running in the Workspace, the execution details can be presented in the integrated terminal or hidden - depending on the specific project needs.

For each stage executed during submission's execution via API the output data (stdout) and error data (stderr) is captured and available to download after execution. Therefore, the following streams are available:

  • stage_init_output, stage_init_error,
  • stage_build_output, stage_build_error,
  • stage_run_output, stage_run_error,
  • stage_test_output, stage_test_error,
  • stage_post_output, stage_post_error.

For more information refer to:

  Contents