Containers manager's handbook - Execution stages

Each submission is executed in steps called stages. The purpose of such division is a logical separation of distinct procedures and, as an effect, clear and straightforward feedback.

The following stages are consecutively performed during the submission execution:

Stage scripts

A stage is usually defined in a form of a bash script. Such scripts are placed among project files and provided in the project configuration (see scenarios.scenario.stages.stage).

From now on, we assume that the execution stage is defined as a bash script. Now, we will discuss the requirements and rules for stage scripts.

Common rules and requirements

All stage scripts except for the execution and the finalization should always execute correctly. This means:

• the exit code (see Exit status) must be 0
• it has to finish within the time limit (see scenarios.scenario.stages.stage.timeout).
• it cannot be stopped or killed by a signal (see Signal IPC) If the stage script fails, the entire submission also fails with the "project error" status.

Correctness is not required for the execution and finalization stages for two different reasons:

• the execution stage is dedicated to run the end user's code, so we expect that it may fail or exceed the time limit
• the finalization stage is mostly a cleanup procedure and it is executed after establishing the final result

Initialization

There is nothing specific for this stage. It is expected to perform preparation steps if there are any.

Compilation

The compilation process should be performed in this step. The Content Manager should decide whether the compilation process was successful or not. There are two options

• compilation process was successful - in this case there is nothing specific to be done
• compilation process failed - in this case the Content Manager should create a file $RUNTIME/compilation_error

The compilation stage script has access to the $RUNTIME environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the compilation stage script:

gcc -o program main.c

COMPILATION_EXIT_CODE="$?"

if [ $COMPILATION_EXIT_CODE -ne 0 ]; then
    echo "Compilation failed"
    touch "$RUNTIME/compilation_error"
else
    echo "Compilation successful"
fi

In this stage you can also:

• access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively

Execution

This stage is dedicated to executing the end user's code. The Content Manager should expect a wide range of program behaviors. In this case, no behavior is considered to be abnormal - the Sphere Engine Containers will handle many types of failures here (e.g. time limit exceeded, run-time error, excessive use of resources) and allow for further processing in the next stage.

In this stage you can also:

• access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
• access $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively

Evaluation

In this stage, the Content Manager has all the data required to make a final decision about the user's submission.

To make this decision you can use the following resources:

• $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
• $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively
• $EXECUTION_STDOUT_FILE and $EXECUTION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "execution" stage respectively
• the $SUBMISSION_STATUS environmental variable that holds the current submission status as a numerical value
• the $EXECUTION_EXIT_CODE environmental variable that holds the exit code from the "execution" stage

By design, the Sphere Engine Containers makes the final decision in accordance with the following rules:

• if the "compilation" stage has ended with compilation error, then the "execution" stage is omitted and the final status is set to "compilation error"
• if the "compilation" stage is successful, then the "execution" stage always executes, and:
  • if the "execution" stage fails, then the final status is set to "run-time error"
  • if the "execution" stage exceeds the time limit, then the final status is set to "time limit exceeded"
  • if the "execution" stage is successful, then the final status is set to "success"

In order to overwrite the default behavior, the Content Manager can set the desired final status to the $RUNTIME/final_status file. The evaluation stage script has access to the $RUNTIME environmental variable. This variable holds the path to the directory used to store the files shared between stages.

Example of the evaluation stage script:

USER_OUTPUT_PATH="$EXECUTION_STDOUT_FILE"
MODEL_OUTPUT_PATH="model_output_data.txt"

if cmp --silent "$USER_OUTPUT_PATH" "$MODEL_OUTPUT_PATH"; then
  echo "[info] user output is identical to model output"
  SOLUTION_CORRECT="1"
else
  echo "[info] user output differs from model output"
  SOLUTION_CORRECT="0"
fi

if [[ "$SOLUTION_CORRECT" != "1" ]]; then
    echo "[info] setting final status as fail"
    echo "FAIL" > "$RUNTIME/final_status"
fi

In this stage you can also:

• set the execution time of the submission by writing it to the file $RUNTIME/final_time; you should provide a single floating point value expressing the number of seconds
• set the memory consumed by the submission by writing its value to the file $RUNTIME/final_memory; you should provide a single integer value expressing the number of kilobytes
• set the score of the submission by writing its value to the file $RUNTIME/final_score; you should provide a single floating point value

Finalization

This stage is dedicated to summarizing and clean-up procedures. It is allowed to fail without failing the entire submissions, but this is not recommended. For potential information about failure during this stage please refer to:

• stage finalization output
• stage finalization error
• debug log

In this stage you can also:

• access $INIT_STDOUT_FILE and $INIT_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "initialization" stage respectively
• access $COMPILATION_STDOUT_FILE and $COMPILATION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "compilation" stage respectively
• access $EXECUTION_STDOUT_FILE and $EXECUTION_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "execution" stage respectively
• access $JUDGE_STDOUT_FILE and $JUDGE_STDERR_FILE environmental variables; they are holding paths to files with the stdout and the stderr data from the "evaluation" stage respectively
• access the $SUBMISSION_STATUS environmental variable that holds the current submission status as a numerical value
• access the $EXECUTION_EXIT_CODE environmental variable that holds the exit code from the "execution" stage

Feedback from stages

The Sphere Engine Containers module handles the output data of each stage separately. When running in the Workspace, the execution details can be presented in the integrated terminal or hidden - depending on the specific project needs.

For each stage executed during submission's execution via API the output data (stdout) and error data (stderr) is captured and available to download after execution. Therefore, the following streams are available:

• stage_init_output, stage_init_error
• stage_compilation_output, stage_compilation_error
• stage_execution_output, stage_execution_error
• stage_judge_output, stage_judge_error
• stage_post_output, stage_post_error

For more information refer to:

• Submission results - especially the "Resulting data steams" section
• GET submission API method
• GET submission stream API method