Containers manager's handbook - Execution stages

Each submission is executed in steps called stages. The purpose of such division is a logical separation of distinct procedures and, as an effect, clear and straightforward feedback.

The following stages are consecutively performed during the submission execution:

Name (code) Overview
Initialization
(init)
The first stage is devoted to the initial preparation steps.

Example procedures suitable for this stage:
  • creating and populating a database (or any different kind of storage)
  • generating randomized data
Compilation
(compilation)
The compilation stage covers all the procedures related to compiling application files.

Example procedures suitable for this stage:
  • running make command for multi-file C++ projects with Makefile
  • building jar file by executing mvn install command for Java projects
  • fetching external dependencies (e.g. pip install in Python or composer install in PHP)
Execution
(execution)
Execution is the crucial stage in this workflow. After two previous stages (initialization and compilation), the application is ready for execution. All the dependencies are available and the code is built (if it was necessary).

Example procedures suitable for this stage:
  • running unit tests
  • launching web server
  • executing console program with input data stream
Evaluation
(judge)
In case your project is designed as a code assignment or a puzzle, the evaluation is a proper stage for correctness verification.

Example procedures suitable for this stage:
  • checking output files generated during the execution stage
  • validation of unit testing report file
Finalization
(post)
The last stage is for summarizing the results of all the previous stages. It may also contain operations opposite to those performed in the initialization phase.

Example procedures suitable for this stage:
  • cleaning database
  • gathering and packing generated files

Note: Not every project requires all stages. If any step is not present, it will simply be omitted. Although, it is required to define at least one stage.

Stage scripts

A stage is usually defined in a form of a bash script. Such scripts are placed among project files and provided in the project configuration (see scenarios.scenario.stages.stage).

Note: Instead of bash script you can provide a stage as a shell command. This is especially useful for simple stages without complex logic behind (e.g. a shell command mvn test for the evaluation stage).

From now on, we assume that the execution stage is defined as a bash script. Now, we will discuss the requirements and rules for stage scripts.

Common rules and requirements

All stage scripts except for the execution and the finalization should always execute correctly. This means:

Correctness is not required for the execution and finalization stages for two different reasons:

  • the execution stage is dedicated to run the end user's code, so we expect that it may fail or exceed the time limit
  • the finalization stage is mostly a cleanup procedure and it is executed after establishing the final result

Initialization

There is nothing specific for this stage. It is expected to perform preparation steps if there are any.

Compilation

The compilation process should be performed in this step. The Content Manager should decide whether the compilation process was successful or not. There are two options

  • compilation process was successful - in this case there is nothing specific to be done
  • compilation process failed - in this case the Content Manager should create a file $RUNTIME/compilation_error

The compilation stage script has access to the $RUNTIME environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the compilation stage script:

gcc -o program main.c

COMPILATION_EXIT_CODE="$?"

if [ $COMPILATION_EXIT_CODE -ne 0 ]; then
    echo "Compilation failed"
    touch "$RUNTIME/compilation_error"
else
    echo "Compilation successful"
fi

In this stage you can also:

  • access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively

Execution

This stage is dedicated to executing the end user's code. The Content Manager should expect a wide range of program behaviors. In this case, no behavior is considered to be abnormal - the Sphere Engine Containers will handle many types of failures here (e.g. time limit exceeded, run-time error, excessive use of resources) and allow for further processing in the next stage.

In this stage you can also:

  • access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
  • access $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively

Evaluation

In this stage, the Content Manager has all the data required to make a final decision about the user's submission.

To make this decision you can use the following resources:

  • $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
  • $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively
  • $EXECUTION_STDOUT_FILE and $EXECUTION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "execution" stage respectively
  • the $SUBMISSION_STATUS environmental variable that holds the current submission status as a numerical value
  • the $EXECUTION_EXIT_CODE environmental variable that holds the exit code from the "execution" stage

By design, the Sphere Engine Containers makes the final decision in accordance with the following rules:

  • if the "compilation" stage has ended with compilation error, then the "execution" stage is omitted and the final status is set to "compilation error"
  • if the "compilation" stage is successful, then the "execution" stage always executes, and:
    • if the "execution" stage fails, then the final status is set to "run-time error"
    • if the "execution" stage exceeds the time limit, then the final status is set to "time limit exceeded"
    • if the "execution" stage is successful, then the final status is set to "success"

In order to overwrite the default behavior, the Content Manager can set the desired final status to the $RUNTIME/final_status file. The evaluation stage script has access to the $RUNTIME environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the evaluation stage script:

USER_OUTPUT_PATH="$EXECUTION_STDOUT_FILE"
MODEL_OUTPUT_PATH="model_output_data.txt"

if cmp --silent "$USER_OUTPUT_PATH" "$MODEL_OUTPUT_PATH"; then
  echo "[info] user output is identical to model output"
  SOLUTION_CORRECT="1"
else
  echo "[info] user output differs from model output"
  SOLUTION_CORRECT="0"
fi

if [[ "$SOLUTION_CORRECT" != "1" ]]; then
    echo "[info] setting final status as fail"
    echo "FAIL" > "$RUNTIME/final_status"
fi

In this stage you can also:

  • set the execution time of the submission by writing it to the file $RUNTIME/final_time; you should provide a single floating point value expressing the number of seconds
  • set the memory consumed by the submission by writing its value to the file $RUNTIME/final_memory; you should provide a single integer value expressing the number of kilobytes
  • set the score of the submission by writing its value to the file $RUNTIME/final_score; you should provide a single floating point value

Finalization

This stage is dedicated to summarizing and clean-up procedures. It is allowed to fail without failing the entire submissions, but this is not recommended. For potential information about failure during this stage please refer to:

  • stage finalization output
  • stage finalization error
  • debug log

In this stage you can also:

  • access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
  • access $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively
  • access $EXECUTION_STDOUT_FILE and $EXECUTION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "execution" stage respectively
  • access $JUDGE_STDOUT_FILE and $JUDGE_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "evaluation" stage respectively
  • access the $SUBMISSION_STATUS environmental variable that holds the current submission status as a numerical value
  • access the $EXECUTION_EXIT_CODE environmental variable that holds the exit code from the "execution" stage

Feedback from stages

The Sphere Engine Containers module handles the output data of each stage separately. When running in the Workspace, the execution details can be presented in the integrated terminal or hidden - depending on the specific project needs.

For each stage executed during submission's execution via API the output data (stdout) and error data (stderr) is captured and available to download after execution. Therefore, the following streams are available:

  • stage_init_output, stage_init_error
  • stage_compilation_output, stage_compilation_error
  • stage_execution_output, stage_execution_error
  • stage_judge_output, stage_judge_error
  • stage_post_output, stage_post_error

For more information refer to: