Safe and reliable production changes, and how Rivian recently got this wrong
60 points by kelp 1 year ago | 45 comments- froh 1 year ago> In this post I’ll be discussing a recent over-the-air (OTA) software update to Rivian vehicles that went badly. It is speculative; I have no insider knowledge of Rivian’s software, systems or practices.
and then it goes into canary testing, "pre flight tests" and rollback
- MattGrommes 1 year agoThose are very common practices and he's clear about saying "that probably means..." and "This seems like...". Seems like fine writing to me.
- froh 1 year agosure. my point is that the title doesn't say a thing about what it actually is about, and what the article is about indeed.
- froh 1 year ago
- MattGrommes 1 year ago
- gambiting 1 year agoTo be fair, Volvo did the same thing recently, so it's not just weird American startups that do this - Volvo never released the numbers on how many cars were affected, but I'm a member of several facebook groups for Volvo owners and it was just like an onslaught, people were posting daily warning others about not applying the latest software patch or it had a good chance of bricking your car. Absolutely no idea how that got released into the wild.
- kelp 1 year agoI hadn't heard about the Volvo one! I had a 2022 Volvo C40 before I got my Rivian R1T.
When I first got the Volvo the GPS and LTE connection would periodically stop working for a day or two. They pushed a fix for it. Later they added CarPlay, which wasn't there when I got the car. Good updates. But not as frequent at Rivian.
Was Volvo able to fix it with another OTA or did people have to go in for service?
- JumpCrisscross 1 year ago> had a 2022 Volvo C40
I tried to buy one, but the dealer in Montana was such a pain about it! How did you like it as a car?
- kelp 1 year agoI liked it a lot. Not great range, but new ones improved that and I still road tripped it all up and down the west coast. Sometimes regret selling it.
- kelp 1 year ago
- earthscienceman 1 year agoImagine being rich enough to buy high end cars every year and worry about OTA updates. What a world.
- ohthatsnotright 1 year agoBeats being so poor you can't afford a car, or the only car you can afford is slowly rusting itself away or one no-longer-mass-produced part away from making a repair out of your financial reach. I'd take bad OTA updates anyday.
- ohthatsnotright 1 year ago
- JumpCrisscross 1 year ago
- kelp 1 year ago
- aranchelk 1 year ago> With the botched 2023.42 update Rivian explained that they pushed the wrong build with the wrong certificates to customer vehicles. This made me immediately think they probably don’t have a canary fleet of vehicles that they roll out to first.
Sounds to me like they certainly could have a canary fleet, but if they do, they didn’t have sufficient process controls to only allow updates to the public after having gone through canary testing.
- steelframe 1 year agoSoftware is such a powerful tool that I understand motor vehicles having as much code in them as they do. What I don't want is for that software to be shoddy or for it to spy on me. I also want complete control over whether or when it changes, and I want to understand the nature of and reason for the updates, just as I do for my Linux laptop on which I use apt-listchanges before accepting upgrades.
For example:
apt-listchanges: Changelogs
---------------------------
bind9 (1:9.16.48-1) bullseye-security; urgency=high
...* New upstream version 9.16.48 - CVE-2023-4408: Parsing large DNS messages may cause excessive CPU load - CVE-2023-5517: Querying RFC 1918 reverse zones may cause an assertion failure when "nxdomain-redirect" is enabledglibc (2.31-13+deb11u8) bullseye; urgency=medium
...* debian/patches/any/local-qsort-memory-corruption.patch: Fix a memory corruption in qsort() when using nontransitive comparison functions.imagemagick (8:6.9.11.60+dfsg-1.3+deb11u2) bullseye; urgency=medium
And so forth. If something makes me raise an eyebrow I can go look at the source code to see what's up. I also like for upstream maintainers and other members of the community being able to do that same. Having that process in place helps keep everyone honest. Why not have this for my car's computers too?* Fix CVE-2021-3574: memory leak was found in TIFF coder * Fix CVE-2021-4219: a special crafted file could lead to a DOS. * Fix CVE-2021-20241 / CVE-2021-20243: divide by zero in some coders (Closes: #1013282)For the install I would rather download a signed image onto a USB drive and flash from that versus letting my car communicate with the mothership indiscriminately. I also want to downgrade at any time with a previous known-good image when there's something about the update that I don't like. For example, if it sends my car's console unit into a bootloop.
- kelp 1 year agoI've also often thought about what an open source car software stack might look like, but with different motivations. I'd love to be able to see more diagnostics about what the car is actually doing and to add 3rd party extensions.
For me, I don't want to have to tinker too much, but I want to be able to. I think the ideal would be something like SteamOS on Steam Deck where you can get into the system, and you can change or add things. But the default is just having it all take care of for you.
That said, cars have all sorts of regulations about how certain things work. I have no idea how any of the above ideas would interact with those regulations.
- nijave 1 year agoImo the current continuous update while letting customers beta test new updates starts to fall apart as the cost of the hardware increases.
Bricking am expensive smart phone is infuriating, but bricking an expensive household appliance or even more expensive automobile is a non starter.
The signed image on USB seemed to be the norm from maybe 2010-2020 but it seems cellular connectivity has gotten too cheap and telemetry too valuable...
- kelp 1 year agoIn the case of Rivian they have been pushing very meaningful improvements on a roughly monthly basis via OTA.
I got my R1T in June 2023 and since here are a few things they've improved, just off the top of my head, not bothering to look it up:
1. Significant improvement to ride quality via different / better suspension tuning.
2. Ability to schedule warming the cabin and pre-condition the battery
3. Completely redesigned the UX for setting drive modes and suspension height (for the better IMO)
4. Added a ton of car info, like battery temp, motor temp, and other info like altitude, various angles the vehicle is at (for off-roading), degrees the front wheels are turned
5. Added additional settings for ride softness / firmness (I got this update yesterday and haven't tried it yet)
When an update is ready I get a notification in the car and from the Rivian app on my phone. I can just hit apply and it installs it.
IMO a USB install would be a substantially worse experience and it would be much less likely that customers would actually install it.
But, for the type of person who just wants the car to stay the same as it was the day they bought it, and never change, it's not the vehicle for them. Personally I really like that it's continually improving and I don't have to go in for service or even go out to the truck to do an update.
- Prcmaker 1 year agoIt's not that I don't want improvements, I modify my cars for exactly that reason, but I want reliability. Improvement to the ride quality shouldn't be a manufacturers after-thought. UX adjustments are nice, adding further visibility to system features, great. OTA updates on systems impacting car functionality or safety, no. These things should be tested thoroughly enough before release to not require periodic updating. They should be stable and tested enough that an difficult to apply update is a reasonable cost. These are not the systems to fail and fix on repeat.
- steelframe 1 year ago> But, for the type of person who just wants the car to stay the same as it was the day they bought it, and never change, it's not the vehicle for them.
I never said I didn't want updates. What I said is that I want to understand what the updates are and then choose to upgrade or downgrade when and how I see fit. Or better yet make the updates OSS and then let me do my own builds with the features and functionality I prefer as they are developed.
One thing that is right is that a Rivian is not for me, for a lot of additional reasons.
- Prcmaker 1 year ago
- kelp 1 year ago
- kelp 1 year ago
- yazzku 1 year agoSoftware in cars and OTA is the stupidest thing in recent years. Like the damn laptop riddled with mediocre software wasn't already frustrating enough, let's fuck up cars too.
- dopylitty 1 year agoTo be fair cars as currently designed are a pretty stupid idea to begin with. Let’s just waste energy carrying around 5000lbs of car at ridiculous speeds to move around a 200lb person.
- dmoy 1 year agoIn this case it's even what, 7000 or 8500lbs car? Which is even crazier when you consider road wear scales on the 3rd or 4th power of weight.
(Edit: 4th power, damn) https://en.m.wikipedia.org/wiki/Fourth_power_law
So a 8500 car does 64x as much road wear as my small sedan.
- infecto 1 year agoWrong
https://twitter.com/ajisuzu1/status/1681123111364620294?s=46
TLDR; From a road wear perspective there is no real difference between a heavy EV and a lightweight smaller ICE.
Edit: Not sure why I get downvoted so heavily. It is just a fact that the weight difference between an EV and comparable ICE has no measurable difference to road wear.
People like the above poster just like to touch on the fourth power law but not how the calculations actually work.
ESAL is part of that calculation. A 5 axel semi has a ESAL of 2.35, a dumptruck ~4, a 3.5ton vehicle .004, a 3ton vehicle .002. When we are talking about the difference in hundreds of pounds between EV and ICE, there is no wear difference.
- infecto 1 year ago
- wilg 1 year agoThis is a terrible argument against cars. You could say the same for a train, let's waste energy carrying around 1,500,000 pounds of train to move around 120,000 pounds of passengers.
- 10000truths 1 year agoScooters and motorcycles are much more efficient in this regard, but uptake has been very limited in Western countries when compared to the pervasive use in Southeast Asia.
- dmoy 1 year ago
- csours 1 year agoSame things with phones. A phone should just make calls, it doesn't need software.
- TheBlight 1 year agoAfter the deluge of spam calls I receive daily I'm somewhat inclined to opt for the opposite direction. No more phonecalls and just using apps for communication.
- esalman 1 year agoPhone has become much more than a caller. It's become a cybernetic extension of our self, and are even used to validate our identity.
- TheBlight 1 year ago
- dopylitty 1 year ago
- giantg2 1 year ago[flagged]