Containers - Handbook - Project configuration

This is a technical reference for Sphere Engine Containers project configuration. You will find here a detailed description covering:

file structure,
parameters (meaning, type, validation rules, description),
examples.

The project configuration is specified by a single JSON file (config.json) of a well-defined structure.

General overview

Let's start by presenting the complete config.json configuration file based on an example. This is to demonstrate a general document structure.

{
  // configuration version
  "version": 2.1,

  // collection of run configurations
  "scenarios": {
    // first run configuration
    "scenario_1": {
      // whether to run this scenario by default
      "default": true,
      // whether to allow Internet connection
      "network": false,
      // list of services (e.g., database)
      "services": ["mysql"],
      // a sequence of consecutive workspace execution stages
      "stages": {
        // configuration of an "init" stage
        "init": {
          "command": null
        },
        // configuration of a "build" stage
        "build": {
          "command": null
        },
        // configuration of a "run" stage
        "run": {
          // a path to a script or shell command to run
          "command": ".sphere-engine/run.sh",
          // where should the data be redirected
          "redirect_streams_to": "console",
          // whether to skip this stage
          "skip": false,
          // how long should the stage be allowed to execute
          "timeout": 10
        },
        // configuration of a "test" stage
        "test": {
          "command": null
        },
        // configuration of a "post" stage
        "post": {
          "command": null
        }
      },
      "output": {
        // type of the workspace's output data
        "type": "file",
        // output data location
        "path": "my_output.csv",
        //  whether to present the output when the workspace loads
        "show_at_startup": false
      },
      // configuration for tests required if you use WeightedScoreTestsGrader
      // if you don't use WeightedScoreTestsGrader the key is not used
      // please see WeightedScoreTestsGrader docs for more information
      // tests weights are specififed as selector. Each selector can match specific unit tests
      "tests_weights": [
        {
          // optional name of the test case to match (or wildcard: "*")
          "name": "test_add_operation",
          // optional name of the test case classname to match (or wildcard: "*")
          "classname": "TestClass",
          // optional status of the test case to match (or wildcard: "*")
          "status": "ok",
          // score assigned by this selector
          "weight": 20
        },
        {
          "name": "other_test",
          "weight": 0
        }
      ],
      // collection of directories that should be collected and archived among other scenario execution results
      "auxdata": {
        "custom_dir1": "d1",
        "custom_dir2": "d2",
        "custom_dir3": "/tmp/xyz"
      },
      // options for webhooks
      "webhooks": {
        "workspace.scenario.finished": {
          "include_streams": ["auxdata", "tests_report", "stage_run_output"]
        }
      }
    },
    // second run configuration
    "scenario_2": {
      "default": false,
      ...
    }
  },

  // settings related to files
  "files": {
    // list of files that should be hidden in the file explorer
    "hidden": [".env", ".gitignore"],
    // list of files that should automatically open after launching the workspace
    "open_at_startup": ["README.md"],
    // list of read-only files
    "read_only": ["config.yaml"]
  },

  // any custom data of any type
  "custom": {
    "internal_tag": "3317c1863f8dd227c2e438e6cabf466b",
    "categories": ["simple programming", "data processing", "algorithms"]
  }
}

Detailed specification

In the following section we present detailed information regarding the config.json configuration file structure.

config.json (root)

A configuration file of the Sphere Engine Containers project.

type: object
required: yes
parameters:

Name	Type	Description	Required	Default
version	`number`	Version of the config file	Yes	-
scenarios	`object`	A collection of scenarios that will be run by the user of the workspace	Yes	-
files	`object`	Settings related to workspace files	No	`{}`
custom	`object`	Any custom data	No	`{}`

example:

{
  "version": 2.1,

  "scenarios": {
    "simple_run": {
      "stages": {
        "run": {
          "command": ".sphere-engine/run.sh"
        }
      }
    }
  }
}

root.version

Version of the config file.

type: number
required: yes
values: 2.1

example:

"version": 2.1

root.scenarios

A collection of scenarios that will be run by the user of the workspace.

type: object
required: yes
parameters:

Name	Type	Description	Required
_scenarioName1_	`object(scenario)`	Definition of scenario #1	No
...	...	...	...
_scenarioName2_	`object(scenario)`	Definition of scenario #N	No

example:

{
  "my_custom_scenario_1": {
    "stages": {
      "run": {
        "command": ".sphere-engine/run1.sh"
      }
    }
  },
  "my_custom_scenario_2": {
    "stages": {
      "build": {
        "command": ".sphere-engine/build.sh"
      },
      "run": {
        "command": ".sphere-engine/run2.sh"
      }
    }
  }
}

Note: It is necessary to define at least one scenario.

Note: You can use any custom name for a key in the scenarios JSON structure.

root.scenarios._scenarioName_

A definition of a single execution scenario.

type: object
required: yes
parameters:

Name	Type	Description	Required	Default
default	`boolean`	If set to true, this scenario will be run on Ctrl+Enter keyboard shortcut in the workspace	No	`false`
network	`boolean`	Should the workspace have access to the Internet	No	`true`
services	`array[string]`	Services needed in a scenario; e.g., database	No	`[]`
output	`object`	Settings related to scenario output data	No	`{}`
stages	`object`	Stages that will be run one after another in a given scenario	Yes	-
auxdata	`object`	Collection of directories that should be collected and archived among other scenario execution results	No	`{}`
tests_weights	`array[test_selector]`	List of test case selectors used to assign scores to test cases	No	`[]`
webhooks	`object`	Options for webhooks	No	`{}`

example:

{
  "stages": {
    "build": {
      "command": ".sphere-engine/build.sh"
    },
    "run": {
      "command": ".sphere-engine/run2.sh"
    }
  }
}

Note: It is necessary to define at least one stage.

root.scenarios._scenarioName_.default

Whether this scenario should be run on Ctrl+Enter keyboard shortcut in the workspace. If no scenario has a default value set, then the first scenario found is executed.

type: boolean
required: no
default: false

example:

"default": true

root.scenarios._scenarioName_.network

Should the workspace have access to the Internet.

type: boolean
required: no
default: true

example:

"network": true

root.scenarios._scenarioName_.services

Services needed in a scenario; e.g., database.

type: array[string]
required: no
default: []
available services:

Service name	Description	Details
`mysql`	MySQL database server	the port is available using the environment variable `$SE_NETWORK_PORT_MYSQL`
`mongodb`	MongoDB database server	the port is available using the environment variable `$SE_NETWORK_PORT_MONGODB`
`vnc`	VNC (Virtual Network Computing) server	the port is available using the environment variable `$SE_NETWORK_PORT_VNC`
`devserver`	HTTP server	the port is available using the environment variable `$SE_NETWORK_PORT_HTTP`

example:

"services": ["mysql", "mongodb"]

root.scenarios._scenarioName_.output

Settings related to scenario output data.

type: object
required: no
default: {}
parameters:

Name	Type	Description	Required	Default
type	`string`	Type of the workspace's output data	Yes	-
path	`string`	Path to the output file or url to the server	No	`_empty_`
show_at_startup	`boolean`	Whether to present the output when the workspace loads	No	`false`

example:

{
  "type": "file",
  "path": "my_output.csv",
  "show_at_startup": true
}

root.scenarios._scenarioName_.output.type

Type of the workspace's output data.

type: string
required: yes
values: console, file, tests, vnc, web

example:

"type": "web"

root.scenarios._scenarioName_.output.path

Path to the output file or url to the server.

type: string
required: no
default: _empty_

example:

"path": "my_output.csv"

Note: The value should contain relative or absolute path to the file or url on the server.

Note: The value can be omitted if the output type is set to tests.

root.scenarios._scenarioName_.output.show_at_startup

Whether to present the output when the workspace loads.

type: boolean
required: no
default: false

example:

"show_at_startup": true

root.scenarios._scenarioName_.stages

A sequence of consecutive workspace execution stages.

type: object
required: yes
parameters:

Name	Type	Description	Required	Default
init	`object(stage)`	Stage #1 - init	No	`{}`
build	`object(stage)`	Stage #2 - build	No	`{}`
run	`object(stage)`	Stage #3 - run	No	`{}`
test	`object(stage)`	Stage #4 - test	No	`{}`
post	`object(stage)`	Stage #5 - post	No	`{}`

example:

{
  "build": {
    "command": ".sphere-engine/build.sh"
  },
  "run": {
    "command": ".sphere-engine/run.sh"
  }
}

Note: It is necessary to define at least one stage.

root.scenarios._scenarioName_.stages._stageName_

Configuration of an individual stage.

stage_name: init, build, run, test, post
type: object
required: no
parameters:

Name	Type	Description	Required	Default
command	`string`	Path to a script or shell command to run	Yes	-
skip	`boolean`	Whether to skip the stage	No	`false`
timeout	`number`	Timeout of the stage in seconds	No	`30`
redirect_streams_to	`string`	Target location for redirecting the stage's output and error streams	No	`console`

example:

{
  "command": ".sphere-engine/stage_script.sh",
  "skip": false,
  "timeout": 10
}

root.scenarios._scenarioName_.stages._stageName_.command

Path to a script or shell command to run.

type: string
required: no

example:

"command": ".sphere-engine/stage_script.sh"

Note: The value should contain relative or absolute path to the Bash script or a shell command to run.

root.scenarios._scenarioName_.stages._stageName_.skip

Whether to skip the stage.

type: string
required: no
default: false

example:

"skip": true

root.scenarios._scenarioName_.stages._stageName_.timeout

Timeout of the stage in seconds.

type: string
required: no
default: 30

example:

"timeout": 90

Note: For the value, provide an integer representing the number of seconds.

root.scenarios._scenarioName_.stages._stageName_.redirect_streams_to

Target location for redirecting the stage's output and error streams. The default value is files for submissions created via API, and console for workspaces. To extract output or error streams from a workspace (e.g., via webhooks), use the both option.

In case of redirecting streams to files (i.e., files and both values), the resulting file locations are pre-defined. For detailed information about these file locations, kindly consult the "Environment" section in the CLI toolkit document or the "Stage" section in the Python toolkit document.

type: string
required: no
default: console
values: console, files, both

example:

"redirect_streams_to": "files"

root.scenarios._scenarioName_.auxdata

A collection of directories that should be collected and archived among other scenario execution results. It is provided as key: value pairs, where the key is a name under which the directory pointed by the value should be stored.

For more information, please refer to an extended description of this parameter.

type: object
required: no
default: {}

example:

{
  "custom_dir1": "d1",
  "custom_dir2": "d2",
  "custom_dir3": "/tmp/xyz"
}

root.scenarios._scenarioName_.tests_weights

A collection of selectors for WeightedScoreTestsGrader that specify what scores assign to what unit tests.

You can find complete weighted unit tests example here.

type: array
required: no
parameters:

Name	Type	Description	Required
[0]	`object(test_selector)`	Definition of unit test case selector #1	No
...	...	...	...
[N]	`object(test_selector)`	Definition of unit tests case selector #N	No

example:

[
  {
    "name": "test_add_operation",
    "weight": 20
  },
  {
    "name": "test_add_operation",
    "status": "failure",
    "weight": -20
  },
  {
    "name": "test_mul_operation",
    "weight": 20
  }
]

root.scenarios._scenarioName_.tests_weights[i]

A definition of a single test case selector. You can learn more here WeightedScoreTestsGrader.

type: object
required: yes
parameters:

Name	Type	Description	Required	Default
name	`string`	Test case name to be matched	No	`"*"`
classname	`string`	Test case class name to be matched	No	`"*"`
status	`"ok" \| "failure" \| "error"`	Test case status to be matched	No	`"ok"`
weight	`number`	Score that will be assigned to a test matched by this test case selector	Yes
example:

{
  "name": "test_add_operation",
  "status": "failure",
  "classname": "*",
  "weight": -20
}

Note: It is necessary to define weight, all other properties are optional.

root.scenarios._scenarioName_.webhooks

The options that can modify the behavior of webhooks.

type: object
required: no
parameters:

Name	Type	Description	Required
workspace.scenario.finished	`object`	sent when a scenario is finished	No

example:

{
  "workspace.scenario.finished": {...}
}

root.scenarios._scenarioName_.webhooks.workspace.scenario.finished

The workspace.scenario.finished webhook is sent when a scenario is finished.

type: object
required: no
parameters:

Name	Type	Description	Required	Default
include_streams	`array[string]`	Streams to be included in the webhook	No	`["auxdata", "tests_report"]`

available streams for include_streams parameter:

[
  "auxdata",
  "tests_report",
  "stage_init_output", "stage_init_error",
  "stage_build_output", "stage_build_error",
  "stage_run_output", "stage_run_error",
  "stage_test_output", "stage_test_error",
  "stage_post_output", "stage_post_error"
]

Important: Adding too many streams to the include_streams can have a negative impact on webhook delivery time. It is advisable to export only the necessary files.

example:

{
  "include_streams": [
    "tests_report",
    "stage_run_output"
  ]
}

root.files

Settings related to workspace files.

type: object
required: no
default: {}
parameters:

Name	Type	Description	Required	Default
hidden	`array[string]`	Files that will be hidden from the file explorer	No	`[]`
open_at_startup	`array[string]`	Files that will be open when the workspace loads	No	`[]`
read_only	`array[string]`	Files that are read only - cannot be modified in the workspace	No	`[]`

example:

{
  "hidden": [".env", ".gitignore"],
  "open_at_startup": ["README.md"],
  "read_only": ["config.yaml"]
}

root.files.hidden

Files that will be hidden from the file explorer.

type: array[string]
required: no
default: []

example:

"hidden": [".env", ".gitignore"]

Note: Each element of the list should contain relative or absolute path to the project file or directory that should be hidden in the file directory.

root.files.open_at_startup

Files that will be open when the workspace loads.

type: array[string]
required: no
default: []

example:

"open_at_startup": ["README.md"]

Note: Each element of the list should contain relative or absolute path to the project file that should automatically open during workspace's launch.

root.files.read_only

Files that are read only - cannot be modified in the workspace.

For more information, please refer to an extended description of this parameter.

type: array[string]
required: no
default: []

example:

"read_only": ["config.yaml"]

Note: Each element of the list should contain relative or absolute path to the project file whose permissions should be limited to read-only.

root.custom

Any data of any structure.

type: object
required: no
default: {}
parameters: any

example:

{
  "internal_tag": "3317c1863f8dd227c2e438e6cabf466b",
  "categories": ["simple programming", "data processing", "algorithms"]
}