rpc_tool/README.md

# Matter RPC Verification Tool

## Overview

The Matter RPC verification tool can help verify if the Matter SDK (or Matter solution) has the required RPCs defined in the [Matter RPC guidelines document](https://docs.google.com/document/d/1uOFa_xLk7H03BnrTGNE5gV3TtiGoEfIE6Yh2fctSeMc), and offer a free stress test suite to help verify the SDK software quality and robustness.

With RPC implemented in the SDK (or the Matter solution), on-device automation testing becomes feasible, such as regression tests (ex: automatic execution every x hour) and stress tests (ex: 100 times device boot up test) can be easily enabled to catch bugs in an early stage.


## Features

1. Provides simple tests to verify if the SDK or the Matter solution contains all the required RPCs.
2. Provides stress test suites to test the quality and robustness of the SDK or the Matter solution.
3. Generates test result in compressed file for the above tests.


## Prerequisites

Make sure the host meets the following prerequisites before setting up the RPC tool.


### Host Minimum Requirements

1. Ubuntu LTS or any other Linux-based environments.
2. `Python3` installed with version `>=3.7`
3. Internet access.
4. Device debug interface (Typically UART over USB)


## Setup tool

1. Clone the repository at ```$HOME``` (or any desired place):
   ```
   $ cd $HOME
   $ git clone https://testsuite-smarthome-matter.googlesource.com/matter_rpc_verification
   ```

2. Change directory to the tool folder:
   ```
   $ cd $HOME/matter_rpc_verification/rpc_tool
   ```

3. Run the script to install packages:
   ```
   $ sh verify_rpc.sh build
   ```

   Or use the below command if you're running on Ubuntu:
   ```
   $ ./verify_rpc.sh build
   ```

## Usage

### Run verification test

To check if the SDK contains the required RPCs: `firmware_version` RPC, `reboot` RPC, `factory_reset` RPC, `descriptor cluster` RPC and `attribute_service` RPC.

1. Connect the device (flashed with the SDK you want to test) to the machine (the host where you installed this tool).

2. Inside the ```rpc_tool``` directory, run the script:
   ```
   $ sh verify_rpc.sh run
   ```

   Or use the below command if you're running on Ubuntu:
   ```
   $ ./verify_rpc.sh run
   ```

3. The tool will ask you to select the device to test, use arrow keys on the keyboard to select. (The tool only supports running 1 device at a time).

4. The tool will start testing the RPC functionality of the device, a successful test result looks like:

```
$ sh verify_rpc.sh run
[RpcTest] 06-09 11:42:34.896 INFO ==========> MatterRpcTest <==========
[RpcTest] 06-09 11:42:35.578 INFO [Test] test_rpc_attribute_service
[RpcTest] 06-09 11:42:36.624 INFO [Test] test_rpc_attribute_service PASS
[RpcTest] 06-09 11:42:36.629 INFO [Test] test_rpc_descriptor_cluster
[RpcTest] 06-09 11:42:37.657 INFO [Test] test_rpc_descriptor_cluster PASS
[RpcTest] 06-09 11:42:37.661 INFO [Test] test_rpc_factory_reset
[RpcTest] 06-09 11:42:38.686 INFO [Test] test_rpc_factory_reset PASS
[RpcTest] 06-09 11:42:38.691 INFO [Test] test_rpc_firmware_version
[RpcTest] 06-09 11:42:39.848 WARNING Failed to decode frame: frame check sequence failure; discarded 3962 bytes
[RpcTest] 06-09 11:42:40.880 INFO [Test] test_rpc_firmware_version PASS
[RpcTest] 06-09 11:42:40.884 INFO [Test] test_rpc_reboot
[RpcTest] 06-09 11:42:41.898 INFO [Test] test_rpc_reboot PASS
[RpcTest] 06-09 11:42:42.722 INFO Summary for test class MatterRpcTest: Error 0, Executed 5, Failed 0, Passed 5, Requested 5, Skipped 0
[RpcTest] 06-09 11:42:42.723 INFO Summary for test run RpcTest@06-09-2022_11-42-34-895:
Total time elapsed 7.827033718116581s
Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-09-2022_11-42-34-895"
Test results: Error 0, Executed 5, Failed 0, Passed 5, Requested 5, Skipped 0
```

A failing test result looks like (device is not responsive to some RPCs):

```
$ sh verify_rpc.sh run
[RpcTest] 06-09 11:48:32.649 INFO ==========> MatterRpcTest <==========
[RpcTest] 06-09 11:48:33.251 INFO [Test] test_rpc_attribute_service
[RpcTest] 06-09 11:48:34.318 INFO [Test] test_rpc_attribute_service PASS
[RpcTest] 06-09 11:48:34.323 INFO [Test] test_rpc_descriptor_cluster
[RpcTest] 06-09 11:48:35.354 INFO [Test] test_rpc_descriptor_cluster PASS
[RpcTest] 06-09 11:48:35.359 INFO [Test] test_rpc_factory_reset
[RpcTest] 06-09 11:48:36.373 INFO [Test] test_rpc_factory_reset PASS
[RpcTest] 06-09 11:48:36.374 INFO [Test] test_rpc_firmware_version
[RpcTest] 06-09 11:48:36.624 WARNING Failed to decode frame: frame check sequence failure; discarded 10 bytes
[RpcTest] 06-09 11:48:36.739 WARNING Failed to decode frame: frame check sequence failure; discarded 54 bytes
[RpcTest] 06-09 11:48:39.375 ERROR Exception occurred in test_rpc_firmware_version.
Traceback (most recent call last):
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 179, in _get_responses
    response = self._response_queue.get(True, timeout_s)
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/mobly/base_test.py", line 756, in exec_one_test
    test_method()
  File "/home/user/rainier_agent/rpc_tool/push_code/rainier-engprod/rpc_tool/./verify_rpc/rpc_tool.py", line 131, in test_rpc_firmware_version
    ack, device_info = self.dut.device_service.GetDeviceInfo()
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/impl.py", line 383, in call
    return self.invoke(
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 261, in wait
    return self._unary_wait(timeout_s)
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 142, in _unary_wait
    for _ in self._get_responses(timeout_s=timeout_s):
  File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 189, in _get_responses
    raise RpcTimeout(self._rpc, timeout_s)
pw_rpc.callback_client.errors.RpcTimeout: No response received for chip.rpc.Device.GetDeviceInfo after 3 s
[RpcTest] 06-09 11:48:40.380 INFO [Test] test_rpc_firmware_version ERROR
[RpcTest] 06-09 11:48:40.382 INFO [Test] test_rpc_reboot
[RpcTest] 06-09 11:48:40.395 WARNING Failed to decode frame: frame check sequence failure; discarded 41 bytes
[RpcTest] 06-09 11:48:41.457 INFO [Test] test_rpc_reboot PASS
[RpcTest] 06-09 11:48:42.390 INFO Summary for test class MatterRpcTest: Error 1, Executed 5, Failed 0, Passed 4, Requested 5, Skipped 0
[RpcTest] 06-09 11:48:42.401 INFO Summary for test run RpcTest@06-09-2022_11-48-32-648:
Total time elapsed 9.75229246309027s
Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-09-2022_11-48-32-648"
Test results: Error 1, Executed 5, Failed 0, Passed 4, Requested 5, Skipped 0
```

5. The test result will be compressed in a zip file ```(rpc_test_result-xxxx.zip)``` under the same directory, which contains the device's logs and test framework logs during execution.

```
$ ls
'rpc_test_result-2022-06-08 14:50:26.499469.zip'   verify_rpc   verify_rpc.sh
```


### Run stress test

To run stress test suite on the SDK (or Matter solution), verifying the software quality and robustness.
The tool will run 50 iterations of each RPC tests.

1. Connect the device to the machine.

2. Inside the ```rpc_tool``` directory, run the script:
   ```
   $ sh verify_rpc.sh stress
   ```

   Or use the below command if you're running on Ubuntu:
   ```
   $ ./verify_rpc.sh stress
   ```

3. Select the device you want to test, then the tool will start stress testing. A successful test result looks like:

```
$ sh verify_rpc.sh stress
[RpcTest] 06-08 11:58:13.719 INFO ==========> MatterRpcStressTest <==========
[RpcTest] 06-08 11:58:15.136 INFO [Test] test_rpc_attribute_service
[RpcTest] 06-08 11:58:16.205 INFO [Test] test_rpc_attribute_service PASS
[RpcTest] 06-08 11:58:16.209 INFO [Test] test_rpc_descriptor_cluster
[RpcTest] 06-08 11:58:17.234 INFO [Test] test_rpc_descriptor_cluster PASS
[RpcTest] 06-08 11:58:17.238 INFO [Test] test_rpc_factory_reset
[RpcTest] 06-08 11:58:18.255 INFO [Test] test_rpc_factory_reset PASS
[RpcTest] 06-08 11:58:18.260 INFO [Test] test_rpc_firmware_version
[RpcTest] 06-08 11:58:19.428 WARNING Failed to decode frame: frame check sequence failure; discarded 3962 bytes
[RpcTest] 06-08 11:58:20.435 INFO [Test] test_rpc_firmware_version PASS
[RpcTest] 06-08 11:58:20.441 INFO [Test] test_rpc_reboot
[RpcTest] 06-08 11:58:21.458 INFO [Test] test_rpc_reboot PASS
[RpcTest] 06-08 11:58:21.464 INFO [Test] test_reboot_0
[RpcTest] 06-08 11:58:22.479 INFO [Test] test_reboot_0 PASS
[RpcTest] 06-08 11:58:22.483 INFO [Test] test_reboot_1
[RpcTest] 06-08 11:58:23.494 INFO [Test] test_reboot_1 PASS
[RpcTest] 06-08 11:58:23.498 INFO [Test] test_reboot_2
[RpcTest] 06-08 11:58:24.516 INFO [Test] test_reboot_2 PASS
[RpcTest] 06-08 11:58:24.520 INFO [Test] test_reboot_3
[RpcTest] 06-08 11:58:25.535 INFO [Test] test_reboot_3 PASS

....

[RpcTest] 06-08 12:02:43.830 INFO [Test] test_attribute_service_48
[RpcTest] 06-08 12:02:44.877 INFO [Test] test_attribute_service_48 PASS
[RpcTest] 06-08 12:02:44.881 INFO [Test] test_attribute_service_49
[RpcTest] 06-08 12:02:45.945 INFO [Test] test_attribute_service_49 PASS
[RpcTest] 06-08 12:02:47.648 INFO Summary for test class MatterRpcStressTest: Error 0, Executed 205, Failed 0, Passed 205, Requested 205, Skipped 0
[RpcTest] 06-08 12:02:47.648 INFO Summary for test run RpcTest@06-08-2022_11-58-13-700:
Total time elapsed 273.94792108098045s
Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-08-2022_11-58-13-700"
Test results: Error 0, Executed 205, Failed 0, Passed 205, Requested 205, Skipped 0
```

4. The stress test result will also be compressed as a zip file under the same directory.