Instructions

Please refer to the main repository for how to use these tools here

CSA Certification Tool - CLI

CLI tool for using the CSA Test Harness

Requirements

1. Ubuntu Server OS
2. Python >= 3.10
3. Poetry installed (see: https://python-poetry.org/docs/#installation)

Setup

1. Open terminal in the root folder and run the command ./scripts/th_cli_install.sh
2. Change the url in config.json if required:

    "hostname" : "192.168.x.x" //Change this to your Raspberry Pi IP address/localhost for local development

3. Run th-cli --help to check available commands

Commands

Commands:
  abort-testing               Abort the current test execution
  available-tests             Get a list of available tests
  project                     Manage the application projects
  run-tests                   Create a new test run from selected tests
  test-run-execution          List test run executions and read logs
  test-runner-status          Get the current Matter test runner status

available-tests

Run th-cli available-tests to get a list of tests available in Test Harness, printed in YAML. For JSON respond, use th-cli available-tests --json .

run-tests

Run th-cli run-tests --tests-list <tests> [--title, -n <title>] [--config, -c <config>] [--pics-config-folder, -p <pics-config-folder>] [--project-id <ID>] [--no-color] [-- <extra-sdk-args>] to start a new test execution.

Required:

• --tests-list: Comma-separated list of test case identifiers (e.g. --tests-list TC-ACE-1.1,TC_ACE_1_3)

Optional:

• --title: Custom title for the test run. If not provided, the current timestamp will be used as the default.

• --config: Path to the JSON config file. Accepts two formats:

  • Config-only: {"network": {...}, "dut_config": {...}}
  • Full project format: {"name": "...", "config": {...}} (automatically extracts config)

  This allows the same JSON file to work with project create, project update, and run-tests commands. If not provided, the project's default configuration will be used.

• --pics-config-folder: Path to the folder that contains PICS files. If not specified, no PICS file will be used.

• --project-id: Project ID that this test run belongs to. If not provided, uses the default 'CLI Execution Project' in TH.

• --no-color: Disable all colors from the CLI's output text of this test run execution

• -- <extra-sdk-args>: Pass additional arguments directly to the SDK container Python tests. Use the double dash (--) separator followed by any SDK test arguments. These arguments will be added to every Python test execution in the run.

Example config files:

Config-only format:

{
  "network": {
    "wifi": {"ssid": "MyNetwork", "password": "MyPassword"}
  },
  "dut_config": {
    "pairing_mode": "ble-wifi",
    "setup_code": "20202021"
  }
}

Full project format (works with all commands):

{
  "name": "My Project",
  "config": {
    "network": {
      "wifi": {"ssid": "MyNetwork", "password": "MyPassword"}
    },
    "dut_config": {
      "pairing_mode": "ble-wifi",
      "setup_code": "20202021"
    }
  }
}

Passing Extra Arguments to SDK Tests:

You can pass additional arguments directly to the SDK container test execution using the -- separator. Everything after -- will be passed as-is to the Python test runner.

Examples:

# Enable trace logging for all tests in the run
th-cli run-tests -t TC-ACE-1.1 -- --trace-to json:log

# Pass boolean argument
th-cli run-tests -t TC-ACE-1.1,TC-ACE-1.2 -- --bool-arg flag:true

# Pass multiple extra arguments
th-cli run-tests -t TC-ACE-1.1 -- --timeout 60 --int-arg some-arg:1 --bool-arg flag:true

# Combine with other CLI options
th-cli run-tests -t TC-ACE-1.1 --config my-config.json --no-color -- --trace-to json:log

Common SDK test arguments you might want to use for example:

• --endpoint <value> - Choose the device endpoint
• --trace-to json:log - Enable detailed trace logging
• --timeout <seconds> - Override default test timeout
• --int-arg <name>:<value> - Pass integer argument to test
• --bool-arg <name>:<true|false> - Pass boolean argument to test
• --string-arg <name>:<value> - Pass string argument to test
• --hex-arg <name>:<hex-value> - Pass hex value argument to test

Note: These extra arguments are applied to ALL Test Cases in the test run. Invalid arguments could cause test failures, so ensure the arguments are valid for the SDK test framework.

test-run-execution-history

Run th-cli test-run-execution-history to fetch the history of test runs. Use --skip and --limit for pagination

Run th-cli test-run-execution-history --id {id} with a test run execution id to fetch the information for that test run.

For JSON respond, add --json to the command.

create-project

Run th-cli create-project --name {project name} --config {config file} to create a new project. Project name is required.

list-projects

Run th-cli list-projects to fetch projects. Use --skip and --limit for pagination. Use --archived to fetch archived projects only.

Run th-cli list-projects --id {id} with a project id to fetch the information for that specific project.

For JSON respond, add --json to the command.

delete-project

Run th-cli delete-project --id {id} to delete a project.

update-project

Run th-cli update-project --id {id} --config {config file} to update a project. Both parameters are required. Config must be a full test environment config file.

Command Colors

By default, the CLI application presents colored texts for all the available commands, specially for the log of test run executions from the th-cli run-tests command. If the users need to disable the colors from the tool's output, they may use one of the options presented below:

1. Use the option flag --no-color from the run-tests command to remove color for that execution (e.g th-cli run-tests --no-color -t TC-ACE-1.1)
2. Prepend the environment variable TH_CLI_NO_COLOR set to True to the run-test command (e.g TH_CLI_NO_COLOR=1 th-cli run-tests -t TC-ACE-1.1)
3. Export the environment variable and use the CLI normally on the same terminal instance (e.g Use export TH_CLI_NO_COLOR=1 before using the CLI commands)

For the item 3 above, it's possible to change permanently adding the TH_CLI_NO_COLOR=1 variable to the shell profile (e.g. ~/.bashrc). After that, resetting that terminal or any new one will present no color for all the CLI commands.

Development

The source files are organized in ./th_cli/.

Add new command

The project uses click to declare commands. To add a new command to the CLI:

• Add a new file in ./th_cli/commands
• Import the new command in ./th_cli/commands/__init__.py
• Import and add the new command to the root group in ./th_cli/main.py

VS Code Environment

This project comes with a pre-configured dev-container for VS Code. This takes care of all dependencies and configuring type-checker, linters and auto-formatting.

Test Harness API Client

A major component of the CLI is the calling the Test Harness API. We're auto-generating a python client for this API using fastapi_client based on the openapi.json published by the Test Harness backend.

To update our client:

• update openapi.json in the root of this project
• run ./scripts/generate_client (This requires Docker to be installed)

New Dependencies

The project dependencies are managed with Poetry. To add a new dependency, run poetry add <package-name>.

Checking Dependencies

The project dependencies may be scanned to look for known vulnerabilities or conflicts for all the poetry packages configured. For that, run the following command after the tool installation to verify the current status of the required dependencies:

• poetry run python ./scripts/check_deps.py

Linting and formatting

The GitHub Project will run linting with Black, Flake8 and mypy on PRs. But these are also available in convenient scripts:

• ./scripts/lint.sh
• ./scripts/format.sh

The VS Code dev-container is also configured to do this automatically on file save.