Child pages
  • How to Run TestVectors Before Submitting a Switch to Stratum CCP
Skip to end of metadata
Go to start of metadata

Prerequisites

docker

curl 


Start Stratum

Stratum supports a range of switches. Details on how to run stratum can be found here.

Get Test Vectors

Test Vectors offer a compact way of defining test input/output. A Test Vector is defined as a set of Test Cases where each test case is defined as a set of Actions and Expectations. Actions are operations run on the switch sequentially, in parallel, or in random sequence. Expectations are expected behavior and start after all actions are triggered. The assumption here is the switch is a blackbox, so an action or an expectation is basically a set of Open API access or external stimulus. You can get testvectors here.

Create template_config, target, port_map

Along with test vector templates, target, port_map and template_config are three mandatory input files needed to run test vectors.

target.pb.txt stores the IP of the switch and port number on which stratum is accessible. Sample can be found here.

portmap.pb.txt contains the ingress and egress port information. This information is needed to run the tests in loopback mode. Sample can be found here. We need two portmap entries for these tests. Entry 1 should be ingress port number and entry 2 should be egress port number. More information on portmap can be found here.

template_config.json is used for rendering the templates and producing Test Vector files that match your switch target. Examples of template configuration files can be found here.     

Template config can be separated into three parts:

First 5 attributes are related to p4runtime test cases.

"Port1":70, 
"Port2":71, 
"Port1_Octal":"\"\\000\\106\"",
"Port2_Octal":"\"\\000\\107\"",
"CPU_Port_Octal": "\"\\000\\375\"",

Port1 is ingress port and port2 is egress port. Port1_Octal and Port2_Octal are octal values of port1 and port2 respectively. CPU_Port_Octal is the octal value of the CPU port of the switch. Appropriate CPU ports are as below:
cpu_port_bmv2 := 253
cpu_port_fpm := 253
cpu_port_mavericks := 320
cpu_port_montara := 192


Next attribute is related to e2e test cases.

"Interface2" :"4/0",

Interface2 attribute is interface name of port2. This interface is used to subscribe for port counters.


Final set of attributes are related to gNMI test cases. Below is the list of interfaces. This list should contain all the ports and respective values mentioned in the chassis_config.pb.txt used to start stratum. All the packet counters should be 0 as we run the gNMI tests in the beginning.

"Interfaces": [
{
"name": "1/0",
"ifindex": 68,
"enabled": true,
"operstatus": "DOWN",
"adminstatus": "UP",
"hardwareport": "",
"systemidmac": "11:22:33:44:55:66",
"systempriority": 1000,
"macaddress": "0:0:0:0:0:0",
"portspeed": "SPEED_100GB",
"negotiatedportspeed": "SPEED_40GB",
"configautonegotiate": false,
"stateautonegotiate": false,
"configforwardingviable": true,
"stateforwardingviable": true,
"confighealthindicator": "GOOD",
"statehealthindicator": "GOOD",
"inbroadcastpkts": 0,
"indiscards": 0,
"inerrors": 0,
"infcserrors": 0,
"inmulticastpkts": 0,
"inoctets": 0,
"inunicastpkts": 0,
"inunknownprotos": 0,
"outbroadcastpkts": 0,
"outdiscards": 0,
"outerrors": 0,
"outmulticastpkts": 0,
"outoctets": 0,
"outunicastpkts": 0
},
{
"name": "2/0",
"ifindex": 72,
"enabled": true,
"operstatus": "UP",
"adminstatus": "UP",
"hardwareport": "",
"systemidmac": "11:22:33:44:55:66",
"systempriority": 1000,
"macaddress": "0:0:0:0:0:0",
"portspeed": "SPEED_100GB",
"negotiatedportspeed": "SPEED_40GB",
"configautonegotiate": false,
"stateautonegotiate": false,
"configforwardingviable": true,
"stateforwardingviable": true,
"confighealthindicator": "GOOD",
"statehealthindicator": "GOOD",
"inbroadcastpkts": 0,
"indiscards": 0,
"inerrors": 0,
"infcserrors": 0,
"inmulticastpkts": 0,
"inoctets": 0,
"inunicastpkts": 0,
"inunknownprotos": 0,
"outbroadcastpkts": 0,
"outdiscards": 0,
"outerrors": 0,
"outmulticastpkts": 0,
"outoctets": 0,
"outunicastpkts": 0
}
]

Get tv-runner Binary Image, tvrunner.sh Script

tvrunner.sh script pulls the docker image if not available and executes the test vectors. The script can be found here. Run below command to get the tvrunner.sh script.

curl https://raw.githubusercontent.com/stratum/testvectors-runner/master/tvrunner.sh > tvrunner.sh

Send Pipeline Config

If you are running tests on a broadcom switch, run below command:

./tvrunner.sh --pull --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/bcm --tv-name PipelineConfig

If you are running tests on a tofino 32 port switch, run below command:

./tvrunner.sh --pull --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/tofino --tv-name PipelineConfig

If you are running tests on a tofino 64 port switch, run below command:

./tvrunner.sh --pull --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/tofino --tv-name PipelineConfig64

Setup Loopback Mode

For setting up ports in loopback mode, run below commands:

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/setup --tv-name Set_Loopback_Mode
./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/setup --tv-name Get_Loopback_Mode

Run Tests

If you are running tests on a broadcom switch, run below commands:

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/gnmi

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/e2e

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/p4runtime --tv-name PktIoOutDirectToDataPlaneTest

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/p4runtime --tv-name L3ForwardTest

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/p4runtime --tv-name PktIoOutToIngressPipelineAclRedirectToPortTest

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/p4runtime --tv-name RedirectDataplaneToDataplaneTest


If you are running tests on a tofino switch, run below commands:

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/gnmi

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/e2e

./tvrunner.sh --dp-mode loopback --target <path_to_target>/target.pb.txt --portmap <path_to_portmap>/portmap.pb.txt --template-config <path_to_template_config>/template_config.json --tv-dir <path_to_testvectors>/templates/p4runtime


Conclusion

If all the above steps are successful then your switch is ready to be included in the CCP.

  • No labels