Bosch performs automated regression testing and user testing using a real car instead of a simulated car. Their goal is to test the software as quickly as possible, both from the perspective of the test engineer and the user. The car is accessible remotely and team members can work without being in the car.
Pavel Hubenov, Senior Test Engineer at Bosch Bulgaria, shared his experiences with remote verification and validation of autonomous vehicles during the QA Challenge Accepted 2021.
The continuous integration testing process is based on an automated system that builds software every night and runs tests to monitor regressions, as Hubenov explained:
We have partial validation each night based on the latest changes implemented. In detail, this means that we test affected features overnight for regressions based on predefined priorities. It is possible that each night the configuration of the test set will be different. The results are dozens of reports to analyze. Another fully automated system with quite complicated criteria that reads, filters and analyzes, finally draws a conclusion for the whole test execution and all team members are informed in the morning.
Bosch follows the traditional V-cycle, except that they included a real car in the last two levels of cascade testing:
To ensure that our device will perform the same in a real car and on the Hardware-In-The-Loop bench, we realized that we need to include a real vehicle in our ongoing testing process.
Moreover, they organize additional test sessions from the user’s point of view during the day. Hubenov mentioned that they use vehicles with different levels of equipment from a commercial point of view to ensure that all model modifications operate the same. One of them is their “hero of the day” which is included in the automated process, Hubenov said.
The car is a special prototype car. It is not allowed to be driven on the road but only used in specially approved parking lots, Hubenov mentioned. It has all the necessary terminals to monitor the entire communication.
Having a modified vehicle as a testbed, they were able to cope with the challenges that the global Covid-19 pandemic brought to them, as Hubenov pointed out:
Having this unique bench has given us the flexibility to not stop working and to continue delivering the excellent quality that our customers deserve. To be mentioned: access is only from the internal Bosch network and is only granted to well-trained people from the project team.
InfoQ asked Pavel Hubenov about testing with a remote self-driving vehicle.
InfoQ: What challenges have you encountered while verifying and validating autonomous vehicles?
Pavel Hubenov: The challenges we face are not only related to autonomous vehicles. The automobile is a very sensitive field, because avoiding possible errors can save a human life. With the increasing complexity of features and the embedded software itself, test engineers can ensure excellent product quality and safety.
At Bosch Engineering Center Sofia, we realize that having highly qualified test engineers is essential, but it is not enough. We also work on the mindset of test engineers – doing their best to achieve top quality at every level of v-cycle by following company standards and processes and remembering that quality in all aspect is a must.
InfoQ: How did you automate and upgrade your tests with a real implemented car instead of a simulation?
Hubenov: Based on simulation tests, we decided to reuse the tests for the vehicle. We have redesigned and adapted the tests to run them in the car while waiting for well-known results. For example, the configuration on the Hardware-In-The-Loop simulation is configured with the actual gateway device and surrounded by simulated virtual nodes representing real vehicle ECUs. The concept of testing on HIL is to send queries to the device and wait for responses within a specified time interval. But, in the real situation in the car, real devices send these same requests and wait for the associated responses. So the test monitors based on those real requests and waits for triggered responses, with no real events on our side that could tip changes into the system.
In conclusion, for this type of test, the test itself is not a direct action generator, but just an event handler, synchronized with the transmitted events and measuring the expected time interval for responses. To control the whole system or affect the system communication, we only use the physical buttons of the car and the keyless remote with specially developed and implemented hardware equipment.
InfoQ: How does the remote test work?
Hubenov: In the automotive field, due to the huge volumes of testing we run for each delivery, we rely heavily on automated testing. Manual testing is mainly used for defect reproduction and as an additional activity to provide better quality. We rely on an automated nightly testing system to ensure stability and non-regression. Automotive testing is a 24/7 activity, and people can get tired. they need rest, but dyno’s can work for people all the time.
Remote access is allowed only for team members, regardless of location. Each team member has permission to reserve a slot for the use of the car and to perform manual or automated tests. Sometimes when a critical issue needs to be reproduced and resolved as soon as possible, the agenda is paused and team leaders are the decision makers. Cases like this are handled with collaboration between developers and validating members. They can control the vehicle using a computer and hardware specially designed for the robotics industry and adapted to our prototype vehicle.
InfoQ: What have you learned on your test automation journey?
Hubenov: I understand that when automation is a team effort, anything is possible. I am happy to share that I had my team behind my back and colleagues from other departments during my journey. We are around 450 highly qualified engineers at Bosch Engineering Center Sofia, and I know that I can count on each one of them when I need help or advice.
Our management strongly supports this friendly and positive environment and knows that we are all one team. I can share that I see one of my dreams come true. I can start looking for new challenges that might seem entirely impossible or unrealistic. I believe that as a team we can find a technical solution for every idea, even if it seems crazy at first glance. Each proactive suggestion is discussed, valued, and accepted or deferred by management based on project priorities and needs. As the process is dynamic, we need quick feedback to give an idea of our new life, and they know it perfectly. Communications between members and prospects at all levels are inspiring and positive. The whole family has the same futuristic vision to achieve our goals.