The Matter RPC verification tool can help verify if the Matter SDK (or Matter solution) has the required RPCs defined in the Matter RPC guidelines document, and offer a free stress test suite to help verify the SDK software quality and robustness.
With RPC implemented in the SDK (or the Matter solution), on-device automation testing becomes feasible, such as regression tests (ex: automatic execution every x hour) and stress tests (ex: 100 times device boot up test) can be easily enabled to catch bugs in an early stage.
Make sure the host meets the following prerequisites before setting up the RPC tool.
Python3 installed with version >=3.7Clone the repository at $HOME (or any desired place):
$ cd $HOME $ git clone https://testsuite-smarthome-matter.googlesource.com/matter_rpc_verification
Change directory to the tool folder:
$ cd $HOME/matter_rpc_verification/rpc_tool
Run the script to install packages:
$ sh verify_rpc.sh build
Or use the below command if you're running on Ubuntu:
$ ./verify_rpc.sh build
To check if the SDK contains the required RPCs: firmware_version RPC, reboot RPC, factory_reset RPC, descriptor cluster RPC and attribute_service RPC.
Connect the device (flashed with the SDK you want to test) to the machine (the host where you installed this tool).
Inside the rpc_tool directory, run the script:
$ sh verify_rpc.sh run
Or use the below command if you're running on Ubuntu:
$ ./verify_rpc.sh run
The tool will ask you to select the device to test, use arrow keys on the keyboard to select. (The tool only supports running 1 device at a time).
The tool will start testing the RPC functionality of the device, a successful test result looks like:
$ sh verify_rpc.sh run [RpcTest] 06-09 11:42:34.896 INFO ==========> MatterRpcTest <========== [RpcTest] 06-09 11:42:35.578 INFO [Test] test_rpc_attribute_service [RpcTest] 06-09 11:42:36.624 INFO [Test] test_rpc_attribute_service PASS [RpcTest] 06-09 11:42:36.629 INFO [Test] test_rpc_descriptor_cluster [RpcTest] 06-09 11:42:37.657 INFO [Test] test_rpc_descriptor_cluster PASS [RpcTest] 06-09 11:42:37.661 INFO [Test] test_rpc_factory_reset [RpcTest] 06-09 11:42:38.686 INFO [Test] test_rpc_factory_reset PASS [RpcTest] 06-09 11:42:38.691 INFO [Test] test_rpc_firmware_version [RpcTest] 06-09 11:42:39.848 WARNING Failed to decode frame: frame check sequence failure; discarded 3962 bytes [RpcTest] 06-09 11:42:40.880 INFO [Test] test_rpc_firmware_version PASS [RpcTest] 06-09 11:42:40.884 INFO [Test] test_rpc_reboot [RpcTest] 06-09 11:42:41.898 INFO [Test] test_rpc_reboot PASS [RpcTest] 06-09 11:42:42.722 INFO Summary for test class MatterRpcTest: Error 0, Executed 5, Failed 0, Passed 5, Requested 5, Skipped 0 [RpcTest] 06-09 11:42:42.723 INFO Summary for test run RpcTest@06-09-2022_11-42-34-895: Total time elapsed 7.827033718116581s Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-09-2022_11-42-34-895" Test results: Error 0, Executed 5, Failed 0, Passed 5, Requested 5, Skipped 0
A failing test result looks like (device is not responsive to some RPCs):
$ sh verify_rpc.sh run
[RpcTest] 06-09 11:48:32.649 INFO ==========> MatterRpcTest <==========
[RpcTest] 06-09 11:48:33.251 INFO [Test] test_rpc_attribute_service
[RpcTest] 06-09 11:48:34.318 INFO [Test] test_rpc_attribute_service PASS
[RpcTest] 06-09 11:48:34.323 INFO [Test] test_rpc_descriptor_cluster
[RpcTest] 06-09 11:48:35.354 INFO [Test] test_rpc_descriptor_cluster PASS
[RpcTest] 06-09 11:48:35.359 INFO [Test] test_rpc_factory_reset
[RpcTest] 06-09 11:48:36.373 INFO [Test] test_rpc_factory_reset PASS
[RpcTest] 06-09 11:48:36.374 INFO [Test] test_rpc_firmware_version
[RpcTest] 06-09 11:48:36.624 WARNING Failed to decode frame: frame check sequence failure; discarded 10 bytes
[RpcTest] 06-09 11:48:36.739 WARNING Failed to decode frame: frame check sequence failure; discarded 54 bytes
[RpcTest] 06-09 11:48:39.375 ERROR Exception occurred in test_rpc_firmware_version.
Traceback (most recent call last):
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 179, in _get_responses
response = self._response_queue.get(True, timeout_s)
_queue.Empty
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/mobly/base_test.py", line 756, in exec_one_test
test_method()
File "/home/user/rainier_agent/rpc_tool/push_code/rainier-engprod/rpc_tool/./verify_rpc/rpc_tool.py", line 131, in test_rpc_firmware_version
ack, device_info = self.dut.device_service.GetDeviceInfo()
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/impl.py", line 383, in call
return self.invoke(
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 261, in wait
return self._unary_wait(timeout_s)
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 142, in _unary_wait
for _ in self._get_responses(timeout_s=timeout_s):
File "/home/user/rpc_tool_venv/lib/python3.9/site-packages/pw_rpc/callback_client/call.py", line 189, in _get_responses
raise RpcTimeout(self._rpc, timeout_s)
pw_rpc.callback_client.errors.RpcTimeout: No response received for chip.rpc.Device.GetDeviceInfo after 3 s
[RpcTest] 06-09 11:48:40.380 INFO [Test] test_rpc_firmware_version ERROR
[RpcTest] 06-09 11:48:40.382 INFO [Test] test_rpc_reboot
[RpcTest] 06-09 11:48:40.395 WARNING Failed to decode frame: frame check sequence failure; discarded 41 bytes
[RpcTest] 06-09 11:48:41.457 INFO [Test] test_rpc_reboot PASS
[RpcTest] 06-09 11:48:42.390 INFO Summary for test class MatterRpcTest: Error 1, Executed 5, Failed 0, Passed 4, Requested 5, Skipped 0
[RpcTest] 06-09 11:48:42.401 INFO Summary for test run RpcTest@06-09-2022_11-48-32-648:
Total time elapsed 9.75229246309027s
Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-09-2022_11-48-32-648"
Test results: Error 1, Executed 5, Failed 0, Passed 4, Requested 5, Skipped 0
(rpc_test_result-xxxx.zip) under the same directory, which contains the device's logs and test framework logs during execution.$ ls 'rpc_test_result-2022-06-08 14:50:26.499469.zip' verify_rpc verify_rpc.sh
To run stress test suite on the SDK (or Matter solution), verifying the software quality and robustness. The tool will run 50 iterations of each RPC tests.
Connect the device to the machine.
Inside the rpc_tool directory, run the script:
$ sh verify_rpc.sh stress
Or use the below command if you're running on Ubuntu:
$ ./verify_rpc.sh stress
Select the device you want to test, then the tool will start stress testing. A successful test result looks like:
$ sh verify_rpc.sh stress [RpcTest] 06-08 11:58:13.719 INFO ==========> MatterRpcStressTest <========== [RpcTest] 06-08 11:58:15.136 INFO [Test] test_rpc_attribute_service [RpcTest] 06-08 11:58:16.205 INFO [Test] test_rpc_attribute_service PASS [RpcTest] 06-08 11:58:16.209 INFO [Test] test_rpc_descriptor_cluster [RpcTest] 06-08 11:58:17.234 INFO [Test] test_rpc_descriptor_cluster PASS [RpcTest] 06-08 11:58:17.238 INFO [Test] test_rpc_factory_reset [RpcTest] 06-08 11:58:18.255 INFO [Test] test_rpc_factory_reset PASS [RpcTest] 06-08 11:58:18.260 INFO [Test] test_rpc_firmware_version [RpcTest] 06-08 11:58:19.428 WARNING Failed to decode frame: frame check sequence failure; discarded 3962 bytes [RpcTest] 06-08 11:58:20.435 INFO [Test] test_rpc_firmware_version PASS [RpcTest] 06-08 11:58:20.441 INFO [Test] test_rpc_reboot [RpcTest] 06-08 11:58:21.458 INFO [Test] test_rpc_reboot PASS [RpcTest] 06-08 11:58:21.464 INFO [Test] test_reboot_0 [RpcTest] 06-08 11:58:22.479 INFO [Test] test_reboot_0 PASS [RpcTest] 06-08 11:58:22.483 INFO [Test] test_reboot_1 [RpcTest] 06-08 11:58:23.494 INFO [Test] test_reboot_1 PASS [RpcTest] 06-08 11:58:23.498 INFO [Test] test_reboot_2 [RpcTest] 06-08 11:58:24.516 INFO [Test] test_reboot_2 PASS [RpcTest] 06-08 11:58:24.520 INFO [Test] test_reboot_3 [RpcTest] 06-08 11:58:25.535 INFO [Test] test_reboot_3 PASS .... [RpcTest] 06-08 12:02:43.830 INFO [Test] test_attribute_service_48 [RpcTest] 06-08 12:02:44.877 INFO [Test] test_attribute_service_48 PASS [RpcTest] 06-08 12:02:44.881 INFO [Test] test_attribute_service_49 [RpcTest] 06-08 12:02:45.945 INFO [Test] test_attribute_service_49 PASS [RpcTest] 06-08 12:02:47.648 INFO Summary for test class MatterRpcStressTest: Error 0, Executed 205, Failed 0, Passed 205, Requested 205, Skipped 0 [RpcTest] 06-08 12:02:47.648 INFO Summary for test run RpcTest@06-08-2022_11-58-13-700: Total time elapsed 273.94792108098045s Artifacts are saved in "/tmp/logs/mobly/RpcTest/06-08-2022_11-58-13-700" Test results: Error 0, Executed 205, Failed 0, Passed 205, Requested 205, Skipped 0