Containers
Handbook - Project utilities - Final tests report

This is a technical reference for the Sphere Engine Containers final tests report.

You fill find here a detailed description covering:

  • description of the final tests report structure,
  • examples.

General overview

As discussed in the utilities overview, the Test Result Converter is a tool used for aggregating intermediate partial results into the final tests report. This report has a form of a standardized JSON document of a well-defined structure.

Let's start by presenting a complete report file based on an example. This is to demonstrate an overview of a general document structure.

[
  {
    "name": "test_example",
    "status": "ok",
    "time": 0.004770,
    "classname": "Tests.Unit.ExampleTest"
  },
  {
    "name": "test_api_index",
    "status": "error",
    "time": 0.232287,
    "classname": "Tests.Feature.ExampleTest",
    "error": "Assertion response_code==200 failed. Line 34"
  },
  {
    "name": "test_api_hello_name",
    "status": "failure",
    "time": 0.020445,
    "classname": "Tests.Feature.ExampleTest",
    "failure": "Assertion response_code==200 failed. Line 34"
  }
]

As you can see, the structure of this document consists of a list of single tests, each of which is an object with a number of primitive-type fields.

Detailed specification

In the following section, we present detailed specification regarding the structure of the final tests report.

report.json (root)

A final tests report file generated during tests stage during execution of the scenario.

It is a list of Test objects.

type: array[Test]
required: yes

example:

[
  {
    "name": "test_api_index",
    "status": "ok",
    "time": 3.14,
    "classname": "Tests.Feature.ExampleTest"
  },
  {
    "name": "test_api_list",
    "status": "ok",
    "time": 2.71,
    "classname": "Tests.Feature.ExampleTest"
  }
]

Test

Result of a single test.

type: object
parameters:

Name Type Description Required
name string Name of the test, usually name of the method that have performed the test Yes
status string Status of the test. Test passed (ok), test failed (failure or error) Yes
time number Time of the test execution in seconds No
classname string Name of the class that contains the test method No
failure string Error message generated by failed test No
error string Error message generated by errored test No

example:

{
  "name": "test_api_index",
  "status": "failure",
  "time": 0.5772,
  "classname": "Tests.Feature.ExampleTest",
  "failure": "Assertion response_code==200 failed. Line 34"
}

Note: Fields failure and error are mutually exclusive. If the value of the status field is set to failure, then the field failure is expected. Similarly, if the value of the status field is set to error, then the presence of the field error is expected.

Manual generation of final tests report

We already discussed the report specification. It allows for manual generation of custom reports.

The Sphere Engine Containers toolkit offers a number of built-in tools handling common use-cases. For example, we provide with the support for interpreting unit test results generating reports compatible with xUnit schema (e.g., jUnit, PHPUnit). Therefore, in such case, the final tests report can be generated automatically.

Another popular case uses a convention of a test given as an (input, model output) pair. In this convention, end-user's code output data is usually expected to match with the model output data being a part of the test. We provide support for such use-case as well, so the final tests report can also be generated automatically.

Nevertheless, we are aware there are lots of interesting use-cases that may require manual generation of the report. To achieve that, it's enough to fulfill these two requirements:

  • create a valid JSON document compatible with the specification,
  • place your document in the expected location:
    • if you use Python package: se_utils.environment.path.final_tests_report,
    • if you use CLI tool use the environmental variable $SE_PATH_FINAL_TESTS_REPORT.
  Contents