Docs › Testing › Pelion Edge ready test suite

Pelion Edge ready test suite

You can use the Pelion Edge ready test suite to test the full working functionality of the deployed Yocto-based or LmP builds and gateway software. The tests identify gaps, errors, or functionality missing the required target. The test cases are divided into different levels according to the functional areas:

Level 1: validates files, folders, partition mounting, partition size and some hardware requirements.
Level 2: checks for the required programs and services on the gateway.
Level 3: checks for the network interface and connectivity.
Level 4: runs smoke test cases to validate the environment.
Level 5: checks for all of the system services.

The tests complete with reports in CSV, JSON, XML and HTML.

Abilities

With the Pelion Edge ready test suite, you can:

Create test automation scripts.
Choose test cases and levels.
Run inside a physical gateway.
Run a single test case or a group of test cases.
Run the test suite on Linux machines with some configuration changes.

Other features include:

Scalability.
User- and business-friendly reporting.
Configurable running order of test cases.
Test reports in multiple formats (CSV, JSON, XML and HTML).

Platform

Pelion Edge (Yocto based or LmP Build).
- Raspberry 3B+
- i.MX8
- AVNET ZU3EG
Ubuntu (Linux)

Note: You must configure the suite to run it on Linux.

Requirements

Architecture

The diagram shows the architecture of the test suite. When the runner file starts, it requires a configuration file. The test suite executes all configured test cases.

perts_arch

Test levels

All test cases are in the test-cases folder. The test cases are divided into test levels. In each level, some categories run sequentially as defined in the configuration file.

Level	Function	Test case file or category
1: file system and specification tests	Tests all folders, files, binaries, hardware requirements and mounting of the partitions in the Yocto-based or LmP builds	- `slashDirectory` to check all files and folders required for Yocto-based or LmP builds. - `partition` to test the mounting and size of different partitions. - `Specification` to test the hardware.
2: program tests	Checks for the existence of the required programs on the gateway. The test cases are validated against the version number of each program. The validation information comes from the configuration file.	`program_and_service`
3: internet connectivity tests	Verifies the gateway connectivity to the internet	`network_and_connectivity`
4: smoke tests	Tests all smoke checks to make sure the test environment is ready to run functional test cases. If any of them fail, the program terminates, and the software is declared unstable. The test moves on to the next category only when all smoke tests pass.	- `edge_smoke`. - `kaas_smoke`.
5: system and acceptance tests	Evaluates the system's compliance with the business requirements, running all processes one by one, to verify the system meets the delivery criteria.	- `edge_services`. - `fluent_bit`.

Install

$ git clone https://github.com/PelionIoT/pelion-edge-ready-test-suite.git

Install npm if not installed:

$ wget https://registry.npmjs.org/npm/-/npm-7.5.3.tgz
$ tar -xzf npm-7.5.3.tgz
$ cd package
$ sudo node bin/npm-cli.js install --prefix /usr/local -gf ../npm-7.5.3.tgz

Note: By default LmP builds do not have npm.

To install kubectl, run:

$ cd pelion-edge-ready-test-suite
$ sudo ./scripts/install_kubectl.sh <api_url> <access_key>

Run

Build project
```
$ cd pelion-edge-ready-test-suite
$ npm install
```
Note: Use the access key and URL of the cloud on which you are testing the gateway.
Create configuration file by following the steps mentioned in Configure Section and run the test suite:
```
$ sudo node index.js -c config.json -e
```

Configure

The test suite runs according to a configuration file. For sample configuration files specific to Raspberry Pi, i.MX8 and Avnet, please see test-configs .

Create a config.json using these fields and format:

{
    "title": "pelion-edge-ready-test-suite",
    "version":"2.0.0",
    "device_type": "avnet",
    "tests_directory": "test-cases",
    "accountID": "<Pelion Cloud Account ID. Ex. -> 015c827e9fdf02420a010b1700000000>",
    "internal_id": "<Pelion Cloud Device ID. Ex. -> 0177b3ab0424000000000001001706ee>",
    "access_key": "<Pelion Cloud Access Key. Ex. -> ak_2MDE3N2FhN2ZhM.............>",
    "test_cases": [
        "test-cases/l1_file_system_and_specification_tests/slashDirectory.js",
        "test-cases/l1_file_system_and_specification_tests/specifications.js",
        "test-cases/l1_file_system_and_specification_tests/partition.js",
        "test-cases/l2_program_tests/program_and_services.js",
        "test-cases/l3_internet_and_connectivity_tests/network_and_connectivity.js",
        "test-cases/l4_smoke_tests/edge_smoke.js",
        "test-cases/l4_smoke_tests/kaas_smoke.js",
        "test-cases/l5_system_and_acceptance_tests/system_and_acceptance_tests.js"
    ],
    "mocha_timeout": 60000,
	"log_level": 5,
	"throttle_rate": 500,
	"create_virtual_devices": false,
    "generate_detailed_report": true,
    "internal_server_port": {
        "edge_core_port": 9101,
        "enterprise_server_port": 8080
    },
    "specifications": {
        "minimum_cpu_core": 1,
        "min_cpu_freq": 200000,
        "max_cpu_freq": 4200000,
        "device_mem": {
            "MemTotal": 900000,
            "MemFree": 30000,
            "MemAvailable": 400000
        },
        "entropy":200,
        "SOC_die_temprature": 80,
        "ledConfig": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['radioConfig']; Ex -> 01",
        "radioConfig": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['radioConfig']; Ex -> 00",
        "hardwareVersion": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['hardwareVersion']; Ex -> rpi3bplus",
        "apiAddress":"<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['cloudAddress']; Ex -> https://api.us-east-1.mbedcloud.com>",
        "gatewayServicesAddress":"<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['gatewayServicesAddress']; Ex -> https://gateways.us-east-1.mbedcloud.com"
    },
    "interface": {
        "wired": "eth0",
        "wireless": "wlan0"
    },
    "connectivity_dns": [
        "google.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['cloudAddress']; Ex -> api.us-east-1.mbedcloud.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['gatewayServicesAddress']; Ex -> gateways.us-east-1.mbedcloud.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['edgek8sServicesAddress']; Ex -> edge-k8s.us-east-1.mbedcloud.com",
        "<Run command - 'curl localhost:9101/status'; Fetch key value -> ['lwm2m-server-uri']; Ex -> lwm2m.us-east-1.mbedcloud.com"
    ],
    "config_path": {
        "identityJSON": "/userdata/edge_gw_config/identity.json",
        "mcc_config_working": "</userdata/mbed/mcc_config/WORKING>"; In case of PARSEC - "</userdata/mbed/mcc_config/pst>",
        "mcc_config_backup": "</userdata/mbed/mcc_config/BACKUP>"; In case of PARSEC - "</userdata/mbed/mcc_config/its>",
        "relay_term_config": "/wigwag/wigwag-core-modules/relay-term/config/config.json",
        "device_db_yaml_config": "/wigwag/etc/devicejs/devicedb.yaml",
        "td_agent_bit_conf": "/etc/td-agent-bit/td-agent-bit.conf"
    },
    "binary_path": {
        "edge_core": "/wigwag/mbed/edge-core"
    },
    "program_version": {
        "maestro": "0.1.0",
        "docker": "19.03.13-ce",
        "openssh": "OpenSSH_8.3p1",
        "openssl": "1.1.1",
        "node": "12.19.0",
        "edge_core": "0.15.0",
        "kernel": "5.4.0",
        "td-agent-bit": "1.3.5"
    },
    "partitions": {
        "/dev/mmcblk1p7": {
            "name": "Sysroot",
            "size": "960M"
        },
        "/dev/mmcblk1p3": {
            "name": "Config",
            "size": "8.9M"
        },
        "/dev/mmcblk1p5": {
            "name": "Flags",
            "size": "8.9M"
        },
        "/dev/mmcblk1p1": {
            "name": "Boot",
            "size": "26M"
        },
        "/dev/mmcblk1p6": {
            "name": "Cache",
            "size": "8.9M"
        },
        "devtmpfs": {
            "name": "Dev",
            "size": "457M"
        }
    },
    "fileSystem": [
        "bin",
        "boot",
        "dev",
        "etc",
        "home",
        "lib",
        "media",
        "mnt",
        "opt",
        "ostree",
        "proc",
        "run",
        "sbin",
        "srv",
        "sys",
        "sysroot",
        "tmp",
        "userdata",
        "usr",
        "var",
        "wigwag"
    ],
    "drivers_path": "/lib/modules/5.4.0-lmp-standard/kernel/drivers/",
    "drivers": [
        "bcma",
        "block",
        "bluetooth",
        "char",
        "crypto",
        "hid",
        "i2c",
        "input",
        "leds",
        "md",
        "media",
        "misc",
        "net",
        "remoteproc",
        "rpmsg",
        "spi",
        "ssb",
        "staging",
        "usb",
        "watchdog"
  ],
  "services": [
        "edge-core",
        "kubelet",
        "maestro",
        "edge-proxy",
        "docker",
        "pelion-relay-term",
        "td-agent-bit"
    ]
}

The configuration values may vary across gateways and operating systems. Please update accordingly.

Some of the essentials you can modify are:

mocha_timeout: the extended timeout in milliseconds for a Mocha test case. The default timeout for a Mocha test case is 2000ms. You can increase it by changing mocha_timeout based on your response time for the test case.
log_level: numeric value from 1 to 5:
- 1 shows only error logs.
- 2 shows error and warning logs.
- 3 shows an error, warning and information logs.
- 4 shows an error, warning, information and debug logs.
- 5 shows an error, warning, information, debug and trace logs.

Report

The test suite generates a detailed report in JSON, CSV, XML and HTML format. The report outlines the test cases run under each category and indicates whether they PASSED or FAILED. It also logs the execution time of each test case and shows the success rate with the total time taken by the test suite. The output files are in the suite_reports folder.

Troubleshooting

If you see this Python issue in npm install:

gyp ERR! stack ImportError: No module named site

Run this command:

$ unset PYHTONHOME
$ unset PYTHONPATH

Documentation