autoevolution
 

Did Euro NCAP Absolve Tesla in Possible Code Cheat? No, It Washed Its Hands

Euro NCAP did not absolve Tesla from possible code wrongdoing: it tried to absolve itself 13 photos
Photo: Euro NCAP/Markus Spiske/edited by autoevolution
GreenTheOnly discovered Tesla added an ANCAP support in its code. It already has Euro NCAP, I VISTA, and Korean NCAP support as wellGreenTheOnly discovered Tesla added an ANCAP support in its code. It already has Euro NCAP, I VISTA, and Korean NCAP support as wellTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classTesla Model Y Euro NCAP results show Dan O’Dowd and the world that FSD is top classANCAP said that it is looking into the Tesla code changes to identify its testing procedures
Pontius Pilate washed his hands to escape guilt when Jesus was condemned to die on the cross. The Roman governor doubted that the Nazarene deserved the death penalty but would be in trouble if he just released Him. In Euro NCAP’s recent issue with Tesla software, the organization said it found nothing wrong “so far.” That was a modern instance of hands washing, but not because Euro NCAP would regret condemning the EV maker.
If you are not familiar with this story – not the biblical one – the white-hat hacker GreenTheOnly bought some Tesla computers from crashed vehicles that contained videos of crash tests performed on Euro NCAP’s premises. They also had peculiar Autopilot software. The hacker regularly purchases MCUs and ICEs (the computers used by Tesla vehicles) to research Tesla’s code and computer hardware. That helped him reveal in May 2020 that the EV maker was dumping these computers without erasing sensitive personal data from its customers. Unclear if Tesla still does that.

When the hacker analyzed these computers, he discovered that they were “running non-production code.” More than that: “100% of observed crash/ADAS tested cars in EU/U.S. have one-off builds on specially provisioned computers.” In other words, these MCUs and ICEs only looked like regular ones: they were actually very different, as GreenTheOnly explained.

“Provisioning is the steps that happen when you get a new empty computer and ‘provision’ it to become a usable computer. That includes putting the correct firmware on it, various security keys, and such.”

According to the hacker, the computers from Euro NCAP’s tests had firmware that was not in the regular ones. The CPUs also presented “different security keys that allowed Tesla personnel to more freely log into that car, too, unlike production cars. In some cases, non-production firmware would also be allowed to be loaded, but not in all.”

Regarding the unusual code, GreenTheOnly announced on September 10 that it contained Australasian New Car Assessment Program (ANCAP) support. He already knew that the company had support elements for I VISTA (the Chinese testing grounds), Euro NCAP, and Korean NCAP, but only disclosed that when he talked about the latest code change he noticed. It is easy to understand why: after warning about what these crashed vehicles’ computers present, these traffic safety organizations may never get rid of them again. The support lines GreenTheOnly found included “separate settings per testing house.”

I sent Euro NCAP and ANCAP the following questions right on September 10:

“1 - Are these support lines in the code something Euro NCAP requested from Tesla to help it on its tests?
2 - If they are not, were these code elements something Euro NCAP was aware of or that Tesla warned Euro NCAP about?
3 - Now that Euro NCAP is aware of this support in the code for its Safety Assist tests, does it plan to study or reveal what it does?
4 - Can Euro NCAP analyze that software on its own, with its experts, or must it get help from other entities?
5 - What can Euro NCAP do if it confirms Tesla is tampering with its tests through its software?
6 - With vehicles that can use their location to adopt certain behaviors and receive over-the-air updates to change their parameters about almost anything, how is Euro NCAP prepared to avoid interferences in its tests?”


On September 13, I wrote an editorial warning about how crucial it was that these traffic safety organizations and regulators developed programming skills to make sure “computers on wheels” would play by the book. To be more precise, to guarantee they would not be modified to fare better in the evaluations these organizations perform. The very reason for these tests to exist is to allow a fair comparison between all products available on the market.

On September 29, I asked Euro NCAP the same questions again and never heard back from it. ANCAP also never replied to my original message: it only told the website Drive it was investigating the case.

To CNN, Euro NCAP declared it did not find evidence that Tesla was using geofencing to change its cars’ behavior in tests. The truth is that, if they had an internet connection, Tesla did not need to, as you’ll see as this story develops. The organization also said it did not find any evidence of wrongdoing. Beware of how Euro NCAP phrases that: not finding something does not mean it is not there, especially when you are careful enough to stress that you have not discovered it yet.

Euro NCAP also told CNN that “the recent addition of ANCAP to the code of Model Y coincides with the start of sales of that vehicle in Australasia.” The organization argued that the code would be necessary due to legislation and marking differences. CNN asked GreenTheOnly about that, and he found this explanation “strange:” Japan also has unique signs and markings, and there is no code change for the Japan New Car Assessment Program (JNCAP). The Model Y is also for sale in that country.

I must stress that the tested Model Y was made in Germany, which makes the excuse of sales in Australasia even more intriguing. In the end, it looks as though Euro NCAP was trying to justify Tesla’s procedures instead of investigating them.

The organization states that “any evidence found – before, during or after the tests – that casts doubt on the production status of the vehicles will automatically lead to an invalid result and follow-up investigations.” What about a different code and specially provisioned computers?

The story gets even more challenging from this point on. CarExpert also managed to get some answers from Euro NCAP. The organization told the Australian website that it trusts the results are fine because it “has been able to match the chronology of software updates to the selection and testing of vehicles.” Euro NCAP thoroughly described this chronology to CarExpert.

Summing up what the organization said, Tesla produced the cars and updated them at the end of the assembly line to make sure they all had the latest software. The entity then chose the vehicles it would test “randomly from the selection of cars offered by Tesla.”

Euro NCAP needs four vehicles for its tests. It either buys them from regular dealerships or selects them from a list when deliveries have not started yet. That’s what it did in Tesla’s case. However, the Model Y made in Germany has been on sale since March. Even with the three-month delay that may occur in publishing test results, Euro NCAP would have obtained the EV units in June (the test was published in September). The explanation is that the organization also selects cars from a list when automakers voluntarily want to participate in the tests.

Ironically, neither of these options would have helped it avoid software concerns with the EV maker. Tesla sells its cars directly to customers – without dealerships. Euro NCAP could never manage to buy vehicles “anonymously” from the company – as it says it does – unless it had some people do it in its place.

Why would none of these acquisition methods make any difference? Because Euro NCAP always informs the carmakers about the vehicle identification numbers (VINs) of the tested units. With regular cars, there is nothing an automaker can do after the vehicles leave their dealerships or factories. All that is left for them is to ask the organization to fit any equipment that will be in future production cars. With legacy carmakers, it did not matter if they knew the tested VINs or not.

Everything was in Euro NCAP’s control until OTA update capability entered the stage. Tesla may log into any of its cars. It often obtains information from crashed vehicles to help in investigations, even if Dutch forensic researchers discovered that the EV maker only discloses what is convenient for it. All Tesla needs to log into these cars is the VIN. Well, Tesla had them. Logging in was even more effortless with “specially provisioned” computers.

GreenTheOnly told me that the firmware 2019.16 28e1f16 was the one in which he first found the Euro NCAP support code. The organization tested a Model X and a Model 3 in 2019. They both achieved 94% ratings in Safety Assist, with a maximum rating of 100%.

If you are wondering why that started in 2019, GreenTheOnly said that “the addition of test labs coincides with (the) merge of the ‘vision only’ radar-less code.” Tesla began physically removing radars from its cars in 2021, but it may have started evaluating that possibility years before.

The hacker told me that firmware 2019.16 28e1f16 was in a whopping two or three deploys on Teslafi at the time: the EV maker put this software in very few vehicles. One possibility is that it was in all the EVs on the list Tesla offered Euro NCAP to select from, but this is not the most likely option. Can that be considered “production?” GreenTheOnly said that this is technically correct, “just barely anyone sees it.” He told me more about that:

“Weeks later, the actual wide release was 2019.16.2.1, containing (an) unknown amount of changes (there were also interim 2019.16.1.1 and 1019.16.2 and maybe even 2019.16.1 releases). You can google the release names, and there are pointers to TMC threads of people excitedly waiting for those releases and not getting them until the wide release.”

In other words, the firmware that GreenTheOnly confirmed to have a code “aware of different ADAS testing protocols” was eagerly awaited by Tesla customers. However, they only received a wide release, and that was weeks after Euro NCAP had tested the vehicles with the suspicious code on update 2019.16 28e1f16. I asked him about the chronology.

“On the surface, it matches what Euro NCAP seems to be saying. They get a build from Tesla, they test it out, then Tesla makes unknown changes to the code (assumed to be enhancements), and that's what gets deployed to the general public. Euro NCAP does not retest this code, though, and just assumes it won't get any worse than what they tested. Note that this is even before all the testing labs spelled out in the code (added sometime in June 2021 or so for the most part, and the Australian one, just recently).”

The rest of the chronology made things look even more confusing. According to Euro NCAP, after it got the vehicles it would test, the EV maker kept improving the software. It eventually froze its development “close to the start of official Euro NCAP testing.” When it was ready, Tesla started “to roll out the new software by OTA” updates. The organization says that the vehicles it would test were also updated with the frozen version of the firmware.

That means the software 2019.16 28e1f16 must have been installed in the tested cars in this last update. After that, they had their ADAS evaluated, crash tested, and their wrecks sold as scrap. For obvious reasons, no other update was performed. It was in that state and with that firmware that the computers eventually ended up in GreenTheOnly’s hands.

Everything gets more puzzling when Euro NCAP continues its chronology stating that Tesla deployed “the new software” before it published the results of the tests. This is another “new software,” if that is not clear. Weren’t the relevant new firmware frozen and updated before the tests occurred? What is this “new software” the organization mentions?

Euro NCAP is probably referring to the multiple OTA updates that Tesla releases, regardless of whether they are being tested by organizations or not. However, there is no purpose in mentioning them in this chronology if they have nothing to do with the code that recognizes ADAS testing. From what we learned from GreenTheOnly and the people Euro NCAP cared to talk to, the organization may not even know which OTA updates contain these codes.

Euro NCAP concluded its chronology by saying that it “tested cars with the same software that is used in production vehicles and which has been retrospectively deployed to previously-built vehicles.” As you have read, that is not entirely true.

Mind you that this is not an ordinary remark: it is a foundational principle of Euro NCAP’s tests. They only make sense if what is evaluated is in all production vehicles. If the organization detected that the tests were performed in modified cars, the evaluation would be void. GreenTheOnly proved that was the case when he discovered the computers had a special provision that “allows Tesla personnel to more freely log into that car.” Why?

After publishing the chronology, CarExpert wrote that “it appears that Tesla is actually using the crash body tests to further improve the vehicles.” Still, there are two mistakes in this conclusion. The first is that the unusual code relates to crash tests: GreenTheOnly made it very clear that they only apply to ADAS evaluations – Euro NCAP names them “Safety Assist” analysis. The second is classifying whatever Tesla did as an improvement.

As GreenTheOnly stressed, “the addition of test labs” to the code emerged when Tesla decided to get rid of radars, something that was widely criticized by safety experts. The EV maker claims that “Tesla Vision” will be all its cars need. What if the cameras get obstructed? What if they have no redundant sensors to keep the ADAS that relies on them working? Who benefits from having a car with fewer sensors, Tesla or its customers?

OTA updates are business as usual for the EV maker. Most software updates reach all vehicles, regardless of when they were manufactured. That does not mean they contain only enhancements. When these updates are restricted to a few thousand cars or even fewer ones, that is usually bad news.

One of these restricted updates capped the voltage of the cells in some Model S and Model X units. Their owners sued Tesla: it was accused of concealing fire risks and other issues with the battery packs in these cars. After two years, Tesla decided to settle. Elon Musk even tweeted that Tesla’s “policy is never to give in to false claims, even if we would lose, and never to fight true claims, even if we would win.” That was widely mocked even by the owners involved with the lawsuit. After all, if that were true, why did the company take two years to realize it was wrong?

Curiously, the settlement came when many of these vehicles were no longer under warranty. When the lawsuit was filed, all of them were still covered. Their owners are now getting even higher charging restrictions and battery pack replacement estimates of more than $20,000. The deal they accepted was to receive $625 as compensation for the voltage cap.

So far, what Euro NCAP said is that it did not find anything suspicious despite everything GreenTheOnly revealed. To CarExpert, the entity said that “it was unable to verify what the development code within the test vehicles contained.” It was Tesla who told Euro NCAP that “the elements of the development code that related to safety were deployed to customers as part of over-the-air updates.”

To be fair, Euro NCAP also did not put much effort into that: the hacker told me it never contacted him or tried to learn more about his findings. Even if it did, it would not have the means to investigate the software. GreenTheOnly states the organization has no way to verify if that is true.

“Unless Tesla gives them the source, how would they verify it? And even if Tesla gives them the source, how do they ensure what they were given matches the car unless they are allowed to self-build? It's not surprising they are at the mercy of the vendors here (not just Tesla).”

On top of that, Euro NCAP does not have any power to force Tesla to be transparent in this case. The EV maker could have tried to clarify everything, but it never made that effort. If you check the official communications from the company, it never even mentioned the episode. It is as if there was nothing suspicious about a limited-access software update and computers with special provisions. In fact, Tesla must be happy that Euro NCAP said it did not find anything wrong – and that the “so far” apparently did not come with any real effort to keep pursuing the truth.

Although the safety organization is the primary victim of anything wrong Tesla may be doing with its ADAS tests, it may also think it is better not to dwell on the subject anymore. After all, this situation exposes that all traffic safety organizations are ill-equipped to deal with “computers on wheels.” Things were much simpler when all they had to do was evaluate how much vehicle bodies deformed to protect their passengers, and it must be embarrassing to admit that. Like Pilate, Euro NCAP washed its hands to get absolution for itself.
If you liked the article, please follow us:  Google News icon Google News Youtube Instagram
About the author: Gustavo Henrique Ruffo
Gustavo Henrique Ruffo profile photo

Motoring writer since 1998, Gustavo wants to write relevant stories about cars and their shift to a sustainable future.
Full profile

 

Would you like AUTOEVOLUTION to send you notifications?

You will only receive our top stories