README.md - local-agent - Git at Google

 # Local Agent

 ## Overview
 **Test Suite for Smart Home with Matter** is a developer platform to support 3rd-party partners build products using the [Matter](https://buildwithmatter.com/) technology, which integrates with the Google Nest ecosystem.

 To support partners in testing and certifying their products, **Test Suite for Smart Home with Matter** provides the tools and services to allow partners to comprehensively test their Matter products against the Google ecosystem in a scalable way.

 Local Agent, one of the major components of Smart Home Partner Test Suite, is a Python-based application which can be installed in the host machines at partners' laboratory. The local agent is required for running automated test suite. It is used to command and control the devices under test (hereinafter called DUT) and perform Matter protocol operations at the testing site. Note that partners won't need to use local agent if they only aim to run Manual test suites.

 In order to use local agent, partners will have to enable the Matter testability of their DUTs, please contact your **Google Developer Support**.

 Note that this is not an officially supported Google product.


 ## Features

 1. Linking and unlinking the local agent from the test suite web app.
 2. Commanding and controlling the devices from the test suite web app.
 3. Reporting local agent and connected device statuses to the backend.
 4. Collecting artifacts and logs from the connected devices and uploading to the backend.


 ## Prerequisites

 Make sure your machine meets the following prerequisites before setting up the local agent.

 ### Host Minimum Requirements

 1. Ubuntu LTS (Other Linux-based environments may work but aren’t officially supported.)
 2. 4GB RAM
 3. 1GB Hard disk space
 4. Internet access
 5. Device debug interface (Typically UART over USB)

 ### Matter extension packages

 The device control library we're using in local agent is GDM, also known as [Gazoo Device Manager](https://github.com/google/gazoo-device), which is also a python package for interacting with Matter devices.

 To allow local agent to talk to your Matter devices, you must build and add your own Matter device controllers to GDM. Please follow the extension [Matter controller codelab in GDM](https://github.com/google/gazoo-device/tree/master/examples/example_matter_extension_package) to build and install.

 Note that you'll have to install your extension packages under the same virtual environment that you create for setting up the local agent. Please contact your **Google Developer Support** if you are not sure how to implement your extension packages.


 ## Set up

 1. Clone the repository (at a desired place):
    ```
    $ git clone https://testsuite-smarthome-matter.googlesource.com/local-agent
    ```

 2. Create a dedicated virtual environment for developing local agent if you don't have one:
    ```
    $ python3 -m venv <venv-name>
    ```

    ```Important```: Make sure you're running a python version ```>= 3.7```, otherwise some packages are not supported.

 3. Activate the virtual environment:
    ```
    $ source <venv-name>/bin/activate
    ```

 4. Install the local agent package in your virtual environment:
    ```
    $ cd /path/to/{repo-base}/local-agent
    $ pip install .
    ```

 > For development, we might want to use `pip install -e .`.
 > (See documentation of the "-e" option [here](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-e))

 5. (Optional) Only needed if you want to register your own Matter extension packages:
    - Create an empty config file named ```local_agent_config.ini``` under ```~/.config/google/```.
    - Copy the content in [example_config.ini](https://testsuite-smarthome-matter.googlesource.com/local-agent/+/refs/heads/master/example_config.ini) to the ```local_agent_config.ini```.
    - Fill the names of your Matter controller extension packages (The extension you built in the above ```Matter extension packages``` section) in ```local_agent_config.ini```. See [example_config.ini](https://testsuite-smarthome-matter.googlesource.com/local-agent/+/refs/heads/master/example_config.ini#20) for references.


 ## Usage

 ### Run Local Agent

 You should be running the Smart Home testsuite from the frontend web app to link and use the local agent.

 1. On the web app's Test Plan page, you should see a ```Local Agent Setup``` button. Click on it, and if you haven’t linked your local agent before, you should see a linking code. Copy the linking code.

 2. On your host machine, activate the virtual environment where you installed the local agent, and start the local agent by simply running ```$ local-agent```.

 3. Once you’ve started your local agent, you should see a prompt that asks you for the linking code. Paste the linking code you copied from the web app. The local agent should start the linking process, and it will also start the device detection process, which should detect your DUTs that are connected to your host machine.

 4. A successful linking result will look like this:

 ```
 $ local-agent
 20211207 14:39:39.785 I local_agent.py:444 Start the linking process
 Linking Code:1d8eb371
 20211207 14:41:10.240 I local_agent.py:149 Local Agent is linked successfully.
 20211207 14:41:10.241 I local_agent.py:224 Reporting status to AMS.
 20211207 14:41:10.242 I local_agent.py:258 Polling JSON-RPC requests from AMS.
 Moving config files to the backup directory /usr/local/google/home/chingtzuchen/gazoo/gdm/conf/backup/backup-20211207-064110

 ##### Step 1/3: Detecting potential new communication addresses. #####

    detecting potential AdbComms communication addresses
    detecting potential DockerComms communication addresses
    detecting potential JlinkSerialComms communication addresses
    detecting potential PigweedSerialComms communication addresses
    detecting potential PtyProcessComms communication addresses
    detecting potential SshComms communication addresses
    detecting potential SerialComms communication addresses
 No read/write permission for these serial address(es): ['/dev/bus/usb/001/001', '/dev/bus/usb/001/005', '/dev/bus/usb/002/001']
    detecting potential UsbComms communication addresses
    detecting potential YepkitComms communication addresses
 'ykushcmd' is not installed. Cannot get Yepkit serials.
 Found 1 possible pigweedserialcomms connections:
    /dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00
 Found 2 possible serialcomms connections:
    /dev/bus/usb/001/002
    /dev/bus/usb/001/003
 Found 1 possible usbcomms connections:
    000440186694

 ##### Step 2/3 Identify Device Type of Connections. #####

 Identifying pigweedserialcomms devices..
 _dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt logging to file /tmp/_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt
 _dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt closing switchboard processes
 20211207 14:41:11.263 I local_agent.py:258 Polling JSON-RPC requests from AMS.
    /dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00 is a efr32matterlighting.
    pigweedserialcomms device_type detection complete.
 Identifying serialcomms devices..
 20211207 14:41:12.281 I local_agent.py:258 Polling JSON-RPC requests from AMS.
    /dev/bus/usb/001/002 responses did not match a known device type.
    /dev/bus/usb/001/003 responses did not match a known device type.
    serialcomms device_type detection complete.
 Identifying usbcomms devices..
    000440186694 responses did not match a known device type.
    usbcomms device_type detection complete.

 ##### Step 3/3: Extract Persistent Info from Detected Devices. #####

 Getting info from communication port /dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00 for efr32matterlighting
    efr32matterlighting_detect checking device readiness: attempt 1 of 1
    efr32matterlighting_detect starting GazooDeviceBase.check_device_ready
 Power cycling properties not yet detected.
    efr32matterlighting_detect health check 1/3 succeeded: Check power cycling ready.
    efr32matterlighting_detect waiting up to 3s for device to be connected.
    efr32matterlighting_detect health check 2/3 succeeded: Check device connected.
    efr32matterlighting_detect logging to file /tmp/_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt
 20211207 14:41:13.302 I local_agent.py:258 Polling JSON-RPC requests from AMS.
    efr32matterlighting_detect health check 3/3 succeeded: Check create switchboard.
    efr32matterlighting_detect GazooDeviceBase.check_device_ready successful. It took 0s.
    efr32matterlighting_detect starting Efr32MatterDevice.get_detection_info
    efr32matterlighting_detect Efr32MatterDevice.get_detection_info successful. It took 0s.
    efr32matterlighting_detect closing switchboard processes
 20211207 14:41:14.320 I local_agent.py:258 Polling JSON-RPC requests from AMS.

 ##### Detection Summary #####

    1 new devices detected:
       efr32matterlighting-6694

    3 connections found but not detected:
       /dev/bus/usb/001/002
       /dev/bus/usb/001/003
       000440186694

 If a connection failed detection, check https://github.com/google/gazoo-device/blob/master/docs/device_setup for tips

 Device                         Alias           Type                     Model                Connected
 ------------------------------ --------------- ------------------------ -------------------- ----------
 efr32matterlighting-6694       <undefined>     efr32matterlighting      PROTO                connected

 Other Devices                  Alias           Type                     Model                Available
 ------------------------------ --------------- ------------------------ -------------------- ----------

 1 total Gazoo device(s) available.
 20211207 14:41:15.339 I local_agent.py:258 Polling JSON-RPC requests from AMS.
 20211207 14:41:16.362 I local_agent.py:258 Polling JSON-RPC requests from AMS.
 20211207 14:41:20.455 W local_agent.py:269 The local agent is unlinked.
 20211207 14:41:20.456 I local_agent.py:285 Stopped polling RPC because stop event is set.
 20211207 14:41:20.456 I local_agent.py:249 Stopped reporting info because stop event is set.
 ```

 ### Check and Update Local Agent version

 You won't be able to run Local Agent if it's outdated (the current version is
 below the minimal required version showed on the web app).
 Check the version of your Local Agent:

 ```
 $ local-agent -v
 ```

 or

 ```
 $ local-agent -version
 ```

 If your Local Agent is outdated, you'll need to update the package manually:

 1. Go to the directory where you installed the local agent:
    ```
    $ cd /path/to/{repo-base}/local-agent
    ```

 2. Fetch and rebase to get the latest commit:
    ```
    $ git fetch origin
    $ git rebase origin/master
    ```

 3. Enable the virtual environment and install the latest Local Agent:
    ```
    $ source <venv-name>/bin/activate
    $ cd /path/to/{repo-base}/local-agent
    $ pip install .
    ```


 ## Troubleshooting

 ### I can't install the Local Agent on my host.

 Make sure you're using Ubuntu LTS on your host and python version is ```>= 3.7```.

 ### The Local Agent complained that the linking code is incorrect.

 Make sure your local agent is talking to the correct backend service:
 The local agent will use the official backend configuration if the ```AMS_HOST```, ```AMS_PORT```
 and ```AMS_SCHEME``` are not provided in ```local_agent_config.ini```, so for the external users,
 you should just simply leave these fields blank. For the internal users who deploy the backend
 by themselves, should probably check the backend configurations in ```local_agent_config.ini```.

 ### My Matter device is not detected by Local Agent.

 Local Agent uses Pigweed RPC debug interface for communicating with Matter devices, make sure
 your Matter device implements the Pigweed RPC endpoint and your Matter device class is already
 registered in GDM (step 5 in the ```Setup``` section). You can also try detecting the devices
 via GDM directly by following the [GDM instructions](https://github.com/google/gazoo-device#detecting-devices).

 ### What if my Matter device is accidentally shutdown during the test execution?

 If the device gets shutdown or rebooted during test execution, GDM may crash and you'll need to close
 the Local Agent and restart it again after the test execution. Local agent does not need to be re-linked..

 ### What if my host machine is accidentally shutdown during the test execution?

 Local agent will be closed forcely and you'll need to restart it. Local agent does not need to be re-linked.


 Please contact your **Google Developer Support** or **smarthome-testsuite-contrib@google.com** if you encounter any unexpected problems.
	# Local Agent

	## Overview
	Test Suite for Smart Home with Matter is a developer platform to support 3rd-party partners build products using the [Matter](https://buildwithmatter.com/) technology, which integrates with the Google Nest ecosystem.

	To support partners in testing and certifying their products, Test Suite for Smart Home with Matter provides the tools and services to allow partners to comprehensively test their Matter products against the Google ecosystem in a scalable way.

	Local Agent, one of the major components of Smart Home Partner Test Suite, is a Python-based application which can be installed in the host machines at partners' laboratory. The local agent is required for running automated test suite. It is used to command and control the devices under test (hereinafter called DUT) and perform Matter protocol operations at the testing site. Note that partners won't need to use local agent if they only aim to run Manual test suites.

	In order to use local agent, partners will have to enable the Matter testability of their DUTs, please contact your Google Developer Support.

	Note that this is not an officially supported Google product.


	## Features

	1. Linking and unlinking the local agent from the test suite web app.
	2. Commanding and controlling the devices from the test suite web app.
	3. Reporting local agent and connected device statuses to the backend.
	4. Collecting artifacts and logs from the connected devices and uploading to the backend.


	## Prerequisites

	Make sure your machine meets the following prerequisites before setting up the local agent.

	### Host Minimum Requirements

	1. Ubuntu LTS (Other Linux-based environments may work but aren’t officially supported.)
	2. 4GB RAM
	3. 1GB Hard disk space
	4. Internet access
	5. Device debug interface (Typically UART over USB)

	### Matter extension packages

	The device control library we're using in local agent is GDM, also known as [Gazoo Device Manager](https://github.com/google/gazoo-device), which is also a python package for interacting with Matter devices.

	To allow local agent to talk to your Matter devices, you must build and add your own Matter device controllers to GDM. Please follow the extension [Matter controller codelab in GDM](https://github.com/google/gazoo-device/tree/master/examples/example_matter_extension_package) to build and install.

	Note that you'll have to install your extension packages under the same virtual environment that you create for setting up the local agent. Please contact your Google Developer Support if you are not sure how to implement your extension packages.


	## Set up

	1. Clone the repository (at a desired place):
	```
	$ git clone https://testsuite-smarthome-matter.googlesource.com/local-agent
	```

	2. Create a dedicated virtual environment for developing local agent if you don't have one:
	```
	$ python3 -m venv <venv-name>
	```

	```Important```: Make sure you're running a python version ```>= 3.7```, otherwise some packages are not supported.

	3. Activate the virtual environment:
	```
	$ source <venv-name>/bin/activate
	```

	4. Install the local agent package in your virtual environment:
	```
	$ cd /path/to/{repo-base}/local-agent
	$ pip install .
	```

	> For development, we might want to use `pip install -e .`.
	> (See documentation of the "-e" option [here](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-e))

	5. (Optional) Only needed if you want to register your own Matter extension packages:
	- Create an empty config file named ```local_agent_config.ini``` under ```~/.config/google/```.
	- Copy the content in [example_config.ini](https://testsuite-smarthome-matter.googlesource.com/local-agent/+/refs/heads/master/example_config.ini) to the ```local_agent_config.ini```.
	- Fill the names of your Matter controller extension packages (The extension you built in the above ```Matter extension packages``` section) in ```local_agent_config.ini```. See [example_config.ini](https://testsuite-smarthome-matter.googlesource.com/local-agent/+/refs/heads/master/example_config.ini#20) for references.


	## Usage

	### Run Local Agent

	You should be running the Smart Home testsuite from the frontend web app to link and use the local agent.

	1. On the web app's Test Plan page, you should see a ```Local Agent Setup``` button. Click on it, and if you haven’t linked your local agent before, you should see a linking code. Copy the linking code.

	2. On your host machine, activate the virtual environment where you installed the local agent, and start the local agent by simply running ```$ local-agent```.

	3. Once you’ve started your local agent, you should see a prompt that asks you for the linking code. Paste the linking code you copied from the web app. The local agent should start the linking process, and it will also start the device detection process, which should detect your DUTs that are connected to your host machine.

	4. A successful linking result will look like this:

	```
	$ local-agent
	20211207 14:39:39.785 I local_agent.py:444 Start the linking process
	Linking Code:1d8eb371
	20211207 14:41:10.240 I local_agent.py:149 Local Agent is linked successfully.
	20211207 14:41:10.241 I local_agent.py:224 Reporting status to AMS.
	20211207 14:41:10.242 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	Moving config files to the backup directory /usr/local/google/home/chingtzuchen/gazoo/gdm/conf/backup/backup-20211207-064110

	##### Step 1/3: Detecting potential new communication addresses. #####

	detecting potential AdbComms communication addresses
	detecting potential DockerComms communication addresses
	detecting potential JlinkSerialComms communication addresses
	detecting potential PigweedSerialComms communication addresses
	detecting potential PtyProcessComms communication addresses
	detecting potential SshComms communication addresses
	detecting potential SerialComms communication addresses
	No read/write permission for these serial address(es): ['/dev/bus/usb/001/001', '/dev/bus/usb/001/005', '/dev/bus/usb/002/001']
	detecting potential UsbComms communication addresses
	detecting potential YepkitComms communication addresses
	'ykushcmd' is not installed. Cannot get Yepkit serials.
	Found 1 possible pigweedserialcomms connections:
	/dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00
	Found 2 possible serialcomms connections:
	/dev/bus/usb/001/002
	/dev/bus/usb/001/003
	Found 1 possible usbcomms connections:
	000440186694

	##### Step 2/3 Identify Device Type of Connections. #####

	Identifying pigweedserialcomms devices..
	_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt logging to file /tmp/_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt
	_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt closing switchboard processes
	20211207 14:41:11.263 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	/dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00 is a efr32matterlighting.
	pigweedserialcomms device_type detection complete.
	Identifying serialcomms devices..
	20211207 14:41:12.281 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	/dev/bus/usb/001/002 responses did not match a known device type.
	/dev/bus/usb/001/003 responses did not match a known device type.
	serialcomms device_type detection complete.
	Identifying usbcomms devices..
	000440186694 responses did not match a known device type.
	usbcomms device_type detection complete.

	##### Step 3/3: Extract Persistent Info from Detected Devices. #####

	Getting info from communication port /dev/serial/by-id/usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00 for efr32matterlighting
	efr32matterlighting_detect checking device readiness: attempt 1 of 1
	efr32matterlighting_detect starting GazooDeviceBase.check_device_ready
	Power cycling properties not yet detected.
	efr32matterlighting_detect health check 1/3 succeeded: Check power cycling ready.
	efr32matterlighting_detect waiting up to 3s for device to be connected.
	efr32matterlighting_detect health check 2/3 succeeded: Check device connected.
	efr32matterlighting_detect logging to file /tmp/_dev_serial_by-id_usb-Silicon_Labs_J-Link_Pro_OB_000440186694-if00_detect.txt
	20211207 14:41:13.302 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	efr32matterlighting_detect health check 3/3 succeeded: Check create switchboard.
	efr32matterlighting_detect GazooDeviceBase.check_device_ready successful. It took 0s.
	efr32matterlighting_detect starting Efr32MatterDevice.get_detection_info
	efr32matterlighting_detect Efr32MatterDevice.get_detection_info successful. It took 0s.
	efr32matterlighting_detect closing switchboard processes
	20211207 14:41:14.320 I local_agent.py:258 Polling JSON-RPC requests from AMS.

	##### Detection Summary #####

	1 new devices detected:
	efr32matterlighting-6694

	3 connections found but not detected:
	/dev/bus/usb/001/002
	/dev/bus/usb/001/003
	000440186694

	If a connection failed detection, check https://github.com/google/gazoo-device/blob/master/docs/device_setup for tips

	Device Alias Type Model Connected
	------------------------------ --------------- ------------------------ -------------------- ----------
	efr32matterlighting-6694 <undefined> efr32matterlighting PROTO connected

	Other Devices Alias Type Model Available
	------------------------------ --------------- ------------------------ -------------------- ----------

	1 total Gazoo device(s) available.
	20211207 14:41:15.339 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	20211207 14:41:16.362 I local_agent.py:258 Polling JSON-RPC requests from AMS.
	20211207 14:41:20.455 W local_agent.py:269 The local agent is unlinked.
	20211207 14:41:20.456 I local_agent.py:285 Stopped polling RPC because stop event is set.
	20211207 14:41:20.456 I local_agent.py:249 Stopped reporting info because stop event is set.
	```

	### Check and Update Local Agent version

	You won't be able to run Local Agent if it's outdated (the current version is
	below the minimal required version showed on the web app).
	Check the version of your Local Agent:

	```
	$ local-agent -v
	```

	or

	```
	$ local-agent -version
	```

	If your Local Agent is outdated, you'll need to update the package manually:

	1. Go to the directory where you installed the local agent:
	```
	$ cd /path/to/{repo-base}/local-agent
	```

	2. Fetch and rebase to get the latest commit:
	```
	$ git fetch origin
	$ git rebase origin/master
	```

	3. Enable the virtual environment and install the latest Local Agent:
	```
	$ source <venv-name>/bin/activate
	$ cd /path/to/{repo-base}/local-agent
	$ pip install .
	```


	## Troubleshooting

	### I can't install the Local Agent on my host.

	Make sure you're using Ubuntu LTS on your host and python version is ```>= 3.7```.

	### The Local Agent complained that the linking code is incorrect.

	Make sure your local agent is talking to the correct backend service:
	The local agent will use the official backend configuration if the ```AMS_HOST```, ```AMS_PORT```
	and ```AMS_SCHEME``` are not provided in ```local_agent_config.ini```, so for the external users,
	you should just simply leave these fields blank. For the internal users who deploy the backend
	by themselves, should probably check the backend configurations in ```local_agent_config.ini```.

	### My Matter device is not detected by Local Agent.

	Local Agent uses Pigweed RPC debug interface for communicating with Matter devices, make sure
	your Matter device implements the Pigweed RPC endpoint and your Matter device class is already
	registered in GDM (step 5 in the ```Setup``` section). You can also try detecting the devices
	via GDM directly by following the [GDM instructions](https://github.com/google/gazoo-device#detecting-devices).

	### What if my Matter device is accidentally shutdown during the test execution?

	If the device gets shutdown or rebooted during test execution, GDM may crash and you'll need to close
	the Local Agent and restart it again after the test execution. Local agent does not need to be re-linked..

	### What if my host machine is accidentally shutdown during the test execution?

	Local agent will be closed forcely and you'll need to restart it. Local agent does not need to be re-linked.


	Please contact your Google Developer Support or smarthome-testsuite-contrib@google.com if you encounter any unexpected problems.