Containers
Handbook - Workspace initialization script

In some cases, the runtime environment of the Sphere Engine Containers module requires an initialization step to be performed immediately after startup. This document describes the workspace_init mechanism responsible for initialization, namely:

• the operation and applications of the initialization procedure,
• ways to define initialization for a project, workspace, and submission,
• implementation and debugging of workspace_init scripts.

General information

Technically, workspace_init is any shell script that is automatically executed as the last step of initializing the runtime environment (i.e., workspace or submission) of the Sphere Engine Containers module.

A simple example of a script:

#!/bin/bash
date > .workspace_creation_date
head -c 32 /dev/urandom | md5sum | awk '{print $1}' > .workspace_custom_hash

Basic use cases include:

• fetching small data from a remote repository,
• generating random unique test cases,
• varying the difficulty level of test scenarios by adding or removing test cases,
• minor updates to project dependencies,
• configuration settings,
• generating data that identifies the environment.

The above and entirely different use cases can be combined. However, when designing the workspace_init script, one must consider the technical limitations.

How to define the workspace_init script?

By default, the workspace_init script is defined per project in the "Workspace init" tab of the project settings available during editing, but it can be modified for both the workspace and the submission.

The workspace and submission will execute the initialization script defined for the project. However, if it has been overwritten, only the new script will be applied.

For the workspace, overwriting the script requires passing the optional parameter workspace_init in the API method POST /workspaces that creates the workspace. For submissions, it is the optional parameter workspace_init in the API method POST /submission that creates the submission.

Setting for the project

Assume our initialization script looks like the one below:

#!/bin/bash
echo "Hello Sphere Engine!"                # write to stdout
touch file_created_by_workspace_init       # create a file
>&2 echo "This goes to the error stream!"  # write to stderr

Setting the script for the project is done in the following steps:

1. Enter the project edit mode (Projects -> Edit (selected row)),
2. Go to the settings by clicking the Settings button,
3. Go to the Workspace init tab,
4. Place the script in the script code field,
5. Save the changes by clicking the Save button.

The effect of the changes can be tested by clicking the Reload project button. A new workspace will be launched for project editing, and all unsaved changes will be lost.

In the case of the script presented above, the result should be the creation of the file file_created_by_workspace_init in the main project directory.

Data from the stdout and stderr streams of the runtime script are available in the .sphere-engine/workspace_init_output and .sphere-engine/workspace_init_error files, respectively.

How to write workspace_init scripts?

The workspace_init script can be written in any interpreted language available in the runtime environment (e.g., bash, perl, python, etc.). To specify the appropriate interpreter, simply use the proper shebang.

Script using bash:

#!/bin/bash
echo "Hello Sphere Engine!"

Script using python:

#!/bin/python3
print('Hello Sphere Engine!')

The workspace_init script is executed in the working directory /home/user/workspace of the Sphere Engine Containers environment. This should be kept in mind, especially when using relative paths to files and directories.

Example:

#!/bin/bash
echo "Hello Sphere Engine!" > welcome_message.txt

In the above example, the workspace_init script will create a file welcome_message.txt (i.e., the file /home/user/workspace/welcome_message.txt) with the content "Hello Sphere Engine!".

The Sphere Engine Containers module expects the workspace_init script to be executed successfully, meaning that the exit code returned by the script will be equal to 0. Otherwise, the initialization will be considered unsuccessful. In that case, for the workspace, it will not be started, and for the submission, it will end with the status Project error .

Example:

#!/bin/bash
rm file_that_does_not_exist

The above script attempts to remove a file that does not exist. As a result, the workspace_init script ends with an exit code of 1, and the entire initialization process fails.

Debugging

The output data from stdout and the error stream stderr of the workspace_init script are available regardless of the success of the execution.

In the case of working in project edit mode, the data is available in the .sphere-engine directory:

• the stdout stream is located in the .sphere-engine/workspace_init_output file,
• the stderr stream is located in the .sphere-engine/workspace_init_error file.

In the case of workspaces managed via the API, data can be accessed using the method GET /workspaces/:id :

• the stdout stream is specified in the response with the key streams.workspace_init_output,
  • the link to download the file is specified by the key streams.workspace_init_output.uri,
• the stderr stream is specified in the response with the key streams.workspace_init_error,
  • the link to download the file is specified by the key streams.workspace_init_error.uri.

An example snippet of an API response:

(...),
"streams": {
    "workspace_init_output": {
        "size": 123,
        "uri": "https://<endpoint>/api/v1/workspaces/93086f6043984e63b696c710317ee6eb/workspace_init_output"
    },
    "workspace_init_error": {
        "size": 67,
        "uri": "https://<endpoint>/api/v1/workspaces/93086f6043984e63b696c710317ee6eb/workspace_init_error"
    }
},
(...)

Of course, it is possible to directly use the method GET /workspaces/:id/:stream with the parameters workspace_init_output and workspace_init_error, respectively. However, using the previously presented method allows for quick verification of whether the streams exist.

In the case of submissions sent via the API, access to the data generated by the workspace_init script can be obtained using the method GET /submissions/:id :

• the stdout stream is specified in the response with the key streams.workspace_init_output,
  • the link to download the file is specified by the key streams.workspace_init_output.uri,
• the stderr stream is specified in the response with the key streams.workspace_init_error,
  • the link to download the file is specified by the key streams.workspace_init_error.uri.

An example snippet of the API response is similar to that of the workspace. However, in this case, the identifier is the id of the submission, and the links use a separate method dedicated to submissions.

(...),
"streams": {
    "workspace_init_output": {
        "size": 123,
        "uri": "https://<endpoint>/api/v1/submissions/42/workspace_init_output"
    },
    "workspace_init_error": {
        "size": 67,
        "uri": "https://<endpoint>/api/v1/submissions/42/workspace_init_error"
    }
},
(...)

Again, in this case, it is also possible to use the direct API method GET /submissions/:id/:stream .

Limitations

The primary limitation of the workspace_init mechanism is the maximum execution time of the script, which is equal to 5 seconds. Exceeding this results in the failure of the entire workspace initialization.

Due to the time limit, it is not recommended to perform operations with long or uncertain execution times, such as:

• downloading large files from the internet,
• fetching large project dependencies,
• tool updates,
• building projects.

All of the above operations should be performed in project edit mode and then saved. The initialization process should focus on quick operations with predictable execution times.

Summary

To summarize the key information:

• the workspace_init script is executed as the last step in starting a new runtime environment (either workspace or submission),
• the global script definition is assigned to the project, but it can be overridden for the workspace or submission,
• the script runs in the project's main directory (i.e., /home/user/workspace),
• the system expects the script to execute successfully, meaning the exit code should be 0,
• the execution time limit for the script is 5 seconds,
• the stdout and stderr streams of the workspace_init script are available:
  • for workspaces in the response of the API method GET /workspaces/:id ,
  • for submissions in the response of the API method GET /submissions/:id .