- Sphere Engine overview
- Compilers
- Overview
- API integration
- JavaScript widget
- Problems
- Overview
- API integration
- JavaScript widget
- E-learning platforms
- Problem setter's handbook
- Problems archive
- RESOURCES
- Programming languages
- Submission streams
- Webhooks
- Disk operations
- Multi-file submissions
- Generating images
- Client libraries
- API Changelog
- FAQ
Each submission is executed in steps called stages. The purpose of such division is a logical separation of distinct procedures and, as an effect, clear and straightforward feedback.
The following stages are consecutively performed during the submission execution:
Name (code) | Overview |
---|---|
Initialization (init) |
The first stage is devoted to the initial preparation steps. Example procedures suitable for this stage:
|
Compilation (compilation) |
The compilation stage covers all the procedures related to compiling application files. Example procedures suitable for this stage:
|
Execution (execution) |
Execution is the crucial stage in this workflow. After two previous stages (initialization and compilation), the application is ready for execution. All the dependencies are available and the code is built (if it was necessary). Example procedures suitable for this stage:
|
Evaluation (judge) |
In case your project is designed as a code assignment or a puzzle, the evaluation is a proper stage for correctness verification. Example procedures suitable for this stage:
|
Finalization (post) |
The last stage is for summarizing the results of all the previous stages. It may also contain operations opposite to those performed in the initialization phase. Example procedures suitable for this stage:
|
Note: Not every project requires all stages. If any step is not present, it will simply be omitted. Although, it is required to define at least one stage.
Stage scripts
A stage is usually defined in a form of a bash script. Such scripts are placed among project files and provided in the project configuration (see scenarios.scenario.stages.stage).
Note: Instead of bash script you can provide a stage as a shell command. This is especially useful for simple
stages without complex logic behind (e.g. a shell command mvn test
for the evaluation stage).
From now on, we assume that the execution stage is defined as a bash script. Now, we will discuss the requirements and rules for stage scripts.
Common rules and requirements
All stage scripts except for the execution and the finalization should always execute correctly. This means:
- the
exit code
(see Exit status) must be0
- it has to finish within the time limit (see scenarios.scenario.stages.stage.timeout).
- it cannot be stopped or killed by a
signal
(see Signal IPC) If the stage script fails, the entire submission also fails with the "project error" status.
Correctness is not required for the execution and finalization stages for two different reasons:
- the execution stage is dedicated to run the end user's code, so we expect that it may fail or exceed the time limit
- the finalization stage is mostly a cleanup procedure and it is executed after establishing the final result
Initialization
There is nothing specific for this stage. It is expected to perform preparation steps if there are any.
Compilation
The compilation process should be performed in this step. The Content Manager should decide whether the compilation process was successful or not. There are two options
- compilation process was successful - in this case there is nothing specific to be done
- compilation process failed - in this case the Content Manager should create a file
$RUNTIME/compilation_error
The compilation stage script has access to the $RUNTIME
environmental variable. This variable holds the path to
the directory used to store the files shared between stages.
Example of the compilation stage script:
gcc -o program main.c
COMPILATION_EXIT_CODE="$?"
if [ $COMPILATION_EXIT_CODE -ne 0 ]; then
echo "Compilation failed"
touch "$RUNTIME/compilation_error"
else
echo "Compilation successful"
fi
In this stage you can also:
- access
$INIT_STDOUT_FILE
and$INIT_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "initialization" stage respectively
Execution
This stage is dedicated to executing the end user's code. The Content Manager should expect a wide range of program behaviors. In this case, no behavior is considered to be abnormal - the Sphere Engine Containers will handle many types of failures here (e.g. time limit exceeded, run-time error, excessive use of resources) and allow for further processing in the next stage.
In this stage you can also:
- access
$INIT_STDOUT_FILE
and$INIT_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "initialization" stage respectively - access
$COMPILATION_STDOUT_FILE
and$COMPILATION_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "compilation" stage respectively
Evaluation
In this stage, the Content Manager has all the data required to make a final decision about the user's submission.
To make this decision you can use the following resources:
$INIT_STDOUT_FILE
and$INIT_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "initialization" stage respectively$COMPILATION_STDOUT_FILE
and$COMPILATION_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "compilation" stage respectively$EXECUTION_STDOUT_FILE
and$EXECUTION_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "execution" stage respectively- the
$SUBMISSION_STATUS
environmental variable that holds the current submission status as a numerical value - the
$EXECUTION_EXIT_CODE
environmental variable that holds theexit code
from the "execution" stage
By design, the Sphere Engine Containers makes the final decision in accordance with the following rules:
- if the "compilation" stage has ended with compilation error, then the "execution" stage is omitted and the final status is set to "compilation error"
- if the "compilation" stage is successful, then the "execution" stage always executes, and:
- if the "execution" stage fails, then the final status is set to "run-time error"
- if the "execution" stage exceeds the time limit, then the final status is set to "time limit exceeded"
- if the "execution" stage is successful, then the final status is set to "success"
In order to overwrite the default behavior, the Content Manager can set the desired final status to the
$RUNTIME/final_status
file. The evaluation stage script has access to the $RUNTIME
environmental variable.
This variable holds the path to the directory used to store the files shared between stages.
Example of the evaluation stage script:
USER_OUTPUT_PATH="$EXECUTION_STDOUT_FILE"
MODEL_OUTPUT_PATH="model_output_data.txt"
if cmp --silent "$USER_OUTPUT_PATH" "$MODEL_OUTPUT_PATH"; then
echo "[info] user output is identical to model output"
SOLUTION_CORRECT="1"
else
echo "[info] user output differs from model output"
SOLUTION_CORRECT="0"
fi
if [[ "$SOLUTION_CORRECT" != "1" ]]; then
echo "[info] setting final status as fail"
echo "FAIL" > "$RUNTIME/final_status"
fi
In this stage you can also:
- set the execution time of the submission by writing it to the file
$RUNTIME/final_time
; you should provide a single floating point value expressing the number of seconds - set the memory consumed by the submission by writing its value to the file
$RUNTIME/final_memory
; you should provide a single integer value expressing the number of kilobytes - set the score of the submission by writing its value to the file
$RUNTIME/final_score
; you should provide a single floating point value
Finalization
This stage is dedicated to summarizing and clean-up procedures. It is allowed to fail without failing the entire submissions, but this is not recommended. For potential information about failure during this stage please refer to:
- stage finalization output
- stage finalization error
- debug log
In this stage you can also:
- access
$INIT_STDOUT_FILE
and$INIT_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "initialization" stage respectively - access
$COMPILATION_STDOUT_FILE
and$COMPILATION_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "compilation" stage respectively - access
$EXECUTION_STDOUT_FILE
and$EXECUTION_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "execution" stage respectively - access
$JUDGE_STDOUT_FILE
and$JUDGE_STDERR_FILE
environmental variables; they are holding paths to files with thestdout
and thestderr
data from the "evaluation" stage respectively - access the
$SUBMISSION_STATUS
environmental variable that holds the current submission status as a numerical value - access the
$EXECUTION_EXIT_CODE
environmental variable that holds theexit code
from the "execution" stage
Feedback from stages
The Sphere Engine Containers module handles the output data of each stage separately. When running in the Workspace, the execution details can be presented in the integrated terminal or hidden - depending on the specific project needs.
For each stage executed during submission's execution via API the output data (stdout
) and error data (stderr
) is
captured and available to download after execution. Therefore, the following streams are available:
stage_init_output
,stage_init_error
stage_compilation_output
,stage_compilation_error
stage_execution_output
,stage_execution_error
stage_judge_output
,stage_judge_error
stage_post_output
,stage_post_error
For more information refer to:
- Submission results - especially the "Resulting data steams" section
- GET submission API method
- GET submission stream API method