Docs › Testing › Pelion Edge ready test suite

Pelion Edge ready test suite

You can use the Pelion Edge ready test suite to test the full working functionality of the deployed LmP Yocto builds and gateway software. The tests identify gaps, errors, or functionality missing the required target. The test cases are divided into different levels according to the functional areas:

Level 1: validates files, folders, partition mounting, partition size and some hardware requirements.
Level 2: checks for the required programs and services on the gateway.
Level 3: checks for the network interface and connectivity.
Level 4: runs smoke test cases to validate the environment.
Level 5: checks for all of the system services.

The tests complete with reports in CSV, JSON, XML and HTML.

Abilities

With the Pelion Edge ready test suite, you can:

Create test automation scripts.
Choose test cases and levels.
Run inside a physical gateway.
Run a single test case or a group of test cases.
Run the test suite on Linux machines with some configuration changes.

Other features include:

Scalability.
User- and business-friendly reporting.
Configurable running order of test cases.
Test reports in multiple formats (CSV, JSON, XML and HTML).

Platform

Pelion Edge (LmP Build).
- Raspberry 3B+
- i.MX8
- AVNET ZU3EG
Ubuntu (Linux)

Note: You must configure the suite to run it on Linux.

Requirements

Architecture

The diagram shows the architecture of the test suite. When the runner file starts, it requires a configuration file. The test suite executes all configured test cases.

perts_arch

Test levels

All test cases are in the test-cases folder. The test cases are divided into test levels. In each level, some categories run sequentially as defined in the configuration file.

Level	Function	Test case file or category
1: file system and specification tests	Tests all folders, files, binaries, hardware requirements and mounting of the partitions in the LmP builds	- `slashDirectory` to check all files and folders required for LmP builds. - `partition` to test the mounting and size of different partitions. - `Specification` to test the hardware.
2: program tests	Checks for the existence of the required programs on the gateway. The test cases are validated against the version number of each program. The validation information comes from the configuration file.	`program_and_service`
3: internet connectivity tests	Verifies the gateway connectivity to the internet	`network_and_connectivity`
4: smoke tests	Tests all smoke checks to make sure the test environment is ready to run functional test cases. If any of them fail, the program terminates, and the software is declared unstable. The test moves on to the next category only when all smoke tests pass.	- `edge_smoke`. - `kaas_smoke`.
5: system and acceptance tests	Evaluates the system's compliance with the business requirements, running all processes one by one, to verify the system meets the delivery criteria.	- `edge_services`. - `fluent_bit`.

Install

$ git clone https://github.com/PelionIoT/pelion-edge-ready-test-suite.git

Install npm if not installed:

$ wget https://registry.npmjs.org/npm/-/npm-7.5.3.tgz
$ tar -xzf npm-7.5.3.tgz
$ cd package
$ sudo node bin/npm-cli.js install --prefix /usr/local -gf ../npm-7.5.3.tgz

Note: By default, LmP builds do not have npm.

To install kubectl, run:

$ cd pelion-edge-ready-test-suite
$ sudo ./scripts/install_kubectl.sh <api_url> <access_key>

Note: This script installs kubectl for Arm64 devices.

Run

Build project
```
$ cd pelion-edge-ready-test-suite
$ npm install
```
Note: Use the access key and URL of the cloud on which you are testing the gateway.
Create configuration file by following the steps mentioned in Configure Section and run the test suite:
```
$ sudo node index.js -c config.json -e
```

Configure

The test suite runs according to a configuration file. For sample configuration files specific to Raspberry Pi3, Raspberry Pi4, i.MX8, Dell and Avnet, please see test-configs .

Note: Fill AccountID, InternalID and Access Key values.

Create a config.json using these fields and format:

{
    "title": "pelion-edge-ready-test-suite",
    "version":"2.0.0",
    "device_type": "avnet",
    "tests_directory": "test-cases",
    "accountID": "<Pelion Cloud Account ID. Ex. -> 015c827e9fdf02420a010b1700000000>",
    "internal_id": "<Pelion Cloud Device ID. Ex. -> 0177b3ab0424000000000001001706ee>",
    "access_key": "<Pelion Cloud Access Key. Ex. -> ak_2MDE3N2FhN2ZhM.............>",
    "test_cases": [
        "test-cases/l1_file_system_and_specification_tests/slashDirectory.js",
        "test-cases/l1_file_system_and_specification_tests/specifications.js",
        "test-cases/l1_file_system_and_specification_tests/partition.js",
        "test-cases/l2_program_tests/program_and_services.js",
        "test-cases/l3_internet_and_connectivity_tests/network_and_connectivity.js",
        "test-cases/l4_smoke_tests/edge_smoke.js",
        "test-cases/l4_smoke_tests/kaas_smoke.js",
        "test-cases/l5_system_and_acceptance_tests/system_and_acceptance_tests.js"
        "test-cases/l5_system_and_acceptance_tests/kaas.js"
    ],
    "mocha_timeout": 60000,
	"log_level": 5,
	"throttle_rate": 500,
	"create_virtual_devices": false,
    "generate_detailed_report": true,
    "internal_server_port": {
        "edge_core_port": 9101,
        "enterprise_server_port": 8080
    },
    "specifications": {
        "minimum_cpu_core": 1,
        "min_cpu_freq": 200000,
        "max_cpu_freq": 4200000,
        "device_mem": {
            "MemTotal": 900000,
            "MemFree": 30000,
            "MemAvailable": 400000
        },
        "entropy":200,
        "SOC_die_temprature": 80,
        "ledConfig": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['radioConfig']; Ex -> 01",
        "radioConfig": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['radioConfig']; Ex -> 00",
        "hardwareVersion": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['hardwareVersion']; Ex -> rpi3bplus",
        "apiAddress":"<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['cloudAddress']; Ex -> https://api.us-east-1.mbedcloud.com>",
        "gatewayServicesAddress":"<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['gatewayServicesAddress']; Ex -> https://gateways.us-east-1.mbedcloud.com",
        "kaasServicesAddress": "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['kaasServicesAddress']; Ex -> https://edge-k8s.eu-west-1.mbedcloud.com"
    },
    "interface": {
        "wired": "eth0",
        "wireless": "wlan0"
    },
    "connectivity_dns": [
        "google.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['cloudAddress']; Ex -> api.us-east-1.mbedcloud.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['gatewayServicesAddress']; Ex -> gateways.us-east-1.mbedcloud.com",
        "<Run command - 'cat /userdata/edge_gw_config/identity.json'; Fetch key value -> ['edgek8sServicesAddress']; Ex -> edge-k8s.us-east-1.mbedcloud.com",
        "<Run command - 'curl localhost:9101/status'; Fetch key value -> ['lwm2m-server-uri']; Ex -> lwm2m.us-east-1.mbedcloud.com"
    ],
    "config_path": {
        "identityJSON": "/userdata/edge_gw_config/identity.json",
        "mcc_config_working": "</userdata/mbed/mcc_config/WORKING>"; In case of PARSEC - "</userdata/mbed/mcc_config/pst>",
        "mcc_config_backup": "</userdata/mbed/mcc_config/BACKUP>"; In case of PARSEC - "</userdata/mbed/mcc_config/its>",
        "relay_term_config": "/wigwag/wigwag-core-modules/relay-term/config/config.json",
        "device_db_yaml_config": "/wigwag/etc/devicejs/devicedb.yaml",
        "td_agent_bit_conf": "/etc/td-agent-bit/td-agent-bit.conf"
    },
    "binary_path": {
        "edge_core": "/wigwag/mbed/edge-core"
    },
    "program_version": {
        "maestro": "0.1.0",
        "docker": "19.03.13-ce",
        "openssh": "OpenSSH_8.3p1",
        "openssl": "1.1.1",
        "node": "12.19.0",
        "edge_core": "0.15.0",
        "kernel": "5.4.0",
        "td-agent-bit": "1.3.5"
    },
    "partitions": {
        "/dev/mmcblk1p7": {
            "name": "Sysroot",
            "size": "960M"
        },
        "/dev/mmcblk1p3": {
            "name": "Config",
            "size": "8.9M"
        },
        "/dev/mmcblk1p5": {
            "name": "Flags",
            "size": "8.9M"
        },
        "/dev/mmcblk1p1": {
            "name": "Boot",
            "size": "26M"
        },
        "/dev/mmcblk1p6": {
            "name": "Cache",
            "size": "8.9M"
        },
        "devtmpfs": {
            "name": "Dev",
            "size": "457M"
        }
    },
    "fileSystem": [
        "bin",
        "boot",
        "dev",
        "etc",
        "home",
        "lib",
        "media",
        "mnt",
        "opt",
        "ostree",
        "proc",
        "run",
        "sbin",
        "srv",
        "sys",
        "sysroot",
        "tmp",
        "userdata",
        "usr",
        "var",
        "wigwag"
    ],
    "drivers_path": "/lib/modules/<device_driver_version>/kernel/drivers/",
    "drivers": [
        "bcma",
        "block",
        "bluetooth",
        "char",
        "crypto",
        "hid",
        "i2c",
        "input",
        "leds",
        "md",
        "media",
        "misc",
        "net",
        "rpmsg",
        "spi",
        "ssb",
        "staging",
        "usb",
        "watchdog"
  ],
  "services": [
        "edge-core",
        "kubelet",
        "maestro",
        "edge-proxy",
        "docker",
        "pelion-relay-term",
        "td-agent-bit"
    ]
}

The configuration values may vary across gateways and operating systems. Please update accordingly.

Some of the essentials you can modify are:

mocha_timeout: the extended timeout in milliseconds for a Mocha test case. The default timeout for a Mocha test case is 2000ms. You can increase it by changing mocha_timeout based on your response time for the test case.
log_level: numeric value from 1 to 5:
- 1 shows only error logs.
- 2 shows error and warning logs.
- 3 shows an error, warning and information logs.
- 4 shows an error, warning, information and debug logs.
- 5 shows an error, warning, information, debug and trace logs.

Report

The test suite generates a detailed report in JSON, CSV, XML and HTML format. The report outlines the test cases run under each category and indicates whether they PASSED or FAILED. It also logs the execution time of each test case and shows the success rate with the total time taken by the test suite. The output files are in the suite_reports folder.

Troubleshooting

If you see this Python issue in npm install:

gyp ERR! stack ImportError: No module named site

Run this command:

$ unset PYHTONHOME
$ unset PYTHONPATH

End-to-end Edge Test Suite

You can perform end-to-end tests using the Python test suite for Pelion Edge end-to-end tests. The library is designed to be used with the pytest test framework.

Prerequisites

Before you can use the end-to-end test suite, you must have:

Python 3.6 or later.
pip.
venv (recommended).

sudo apt install python3-pip
sudo apt install python3-venv

kubectl.

Example:

snap install kubectl --classic
kubectl version --client

Set up

Clone repository.
Create virtual environment.
Upgrade pip.
Install requirements.

git clone https://github.com/PelionIoT/e2e-edge-test-suite.git
cd e2e-edge-test-suite
python -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install -r requirements.txt

Configure

Start Edge device.
Create a configuration file based on the environment config template:
Log in to the Pelion Device Management portal:
Create an access key in the portal.
Set the key in the configuration file.
Set the edge device ID in the configuration file.

Example configuration file (EU):

{
  "api_gw": "https://api.eu-west-1.mbedcloud.com",
  "api_key": "",
  "connection_type": "cloud",
  "edge_k8s_url": "https://edge-k8s.eu-west-1.mbedcloud.com",
  "gateways_url": "https://gateways.eu-west-1.mbedcloud.com",
  "device_id": "",
  "has_remote_terminal": true
}

Run

Run all tests:

pytest  --config_path=tests/config.json  --html=results.html

Customize

There are many ways to configure the test runs. Refer to the full pytest documentation for more information.

Tests

test_device_attributes.py

Device attribute tests use Pelion Device Management device directory APIs to get device information and device attributes, such as registration status.

Test name	Main functions	Notes
test_device_registered	Verify the device is registered.

test_k8s_kubectl.py

Kubernetes tests use the Kubernetes command-line tool, kubectl, to run commands against Kubernetes clusters. You must enable Edge Kubernetes in your Pelion Device Management account to run these tests successfully.

Test name	Main functions	Notes
test_node	Verify the device is registered in Kubernetes.
test_pod	Verify the test Pod can be launched.
test_pod_state	Verify the test Pod state is running.

test_lwm2m_resource.py

LwM2M resource tests use the Pelion Device Management connect API to read and write resource values to and from the device.

Test name	Main functions	Notes
test_put_lifetime	Verify the write-to-device resource works.	Resource '1/0/1'
test_get_lifetime	Verify the read-from-device resource works.	Resource '1/0/1'

test_remote_terminal.py

These tests verify the device remote terminal is functional.

Note: Not all edge gateway devices have remote terminal supported. Use the configuration file parameter has_remote_terminal to define whether this test is part of the set.

Test name	Main functions	Notes
test_remote_terminal	Verify communication via remote terminal works.

test_websocket_notifications.py

This test file uses the Pelion Device Management notifications API to register a websocket notification channel to receive notifications such as registration, registration update and resource notifications from the device.

Test name	Main functions	Notes
test_registration_update_notification	Verify the registration update notification is received.
test_registration_notification	Verify the registration notification is received after reboot.
test_notification_device_cpu_usage	Verify the notification from the device is received.	Resource: '/3/0/3320', supported by Snap-Pelion-Edge

License

For licensing details, please see the license agreement.

Documentation

Pelion Edge ready test suite

Abilities

Platform

Requirements

Architecture

Test levels

Install

Run

Configure

Report

Troubleshooting

End-to-end Edge Test Suite

Prerequisites

Set up

Configure

Run

Customize

Tests

test_device_attributes.py

test_k8s_kubectl.py

test_lwm2m_resource.py

test_remote_terminal.py

test_websocket_notifications.py

License