Medical devices software testing overview
89 points by Totoradio 8 years ago | 22 comments- rubidium 8 years agoNote: this post is just the overview. There are a bunch more posts on the site (https://agilemeddev.com/) at the site that deep dive into each of the parts.
It seems to be a helpful framework to work from.
- kelvin0 8 years agoI'm surprised by the 'shallow' nature of the article, and it's obvious points (medical devices cannot fail). Almost seems self promotional.
I thought formal methods or at least advanced static analysis was going to be mentioned before I clicked on the URL.
- a3n 8 years agoAre formal methods and advanced static analysis used a lot in medical device software development?
- wordtothedevice 8 years ago"Formal methods" of verification for medical devices usually means writing detailed product requirements in prose that are traced to manual test cases. Unit test coverage on all the software I worked on was pretty low, and I never heard of anyone using formal program verification, but there were armies of manual testers that found way more bugs than any automated tests did anyway.
Source: I worked on anesthesia machines, cath lab equipment, and patient monitoring devices at big companies in the US.
- julienzaegel 8 years ago(I'm the author of the aforementioned article)
We tried using static verification with CodeContracts at the beginning of the project (2011). But this triggered us to litter the code with attributes dedicated to static verification only, which didn't spot many issues. The tool (which is a binary rewriter) was very slow (tripled compile time) and killed developer productivity and morale. It just wasn't worth it. We abandoned static verification, and later we abandoned CodeContracts altogether. Maybe it works with better tools or other languages than C#, but my experience with it is bad and I don't recommend it. With a statically typed language, maybe the best static verification tool is just the compiler.
- joe_the_user 8 years agoWell, a big question is "what is a medical device"?
Is that any object or software that comes in contact with patients?
I worked producing software that segmented images produced by a medical imaging device. The software would be built-in but it was the kind of thing one could have also have done exporting the images to Photoshop.
So the question of what the boundary between normal software and medical software is going to be rather important if medical software winds-up constrained by NASA level procedures.
(Obviously, implanted devices need very particular standards but other devices raise questions).
- seren 8 years agoIEC62304 requires that for risk management you decompose your sw in modules or units and assess the risk for each one.
Roughly a class A module or device even if malfunctioning can not cause harm. For example it could be a logging or debugging module. It is annoying if it is broken for you but should not harm the patient.
A class B malfunction can have minor or severe consequences but that should be reversible like a minor injury, it can not have long lasting effect.
Finally class C means irreversible effects (amputing the wrong leg, death). Misdiagnosis are often class C (unless your system is detecting common cold)
This analysis has to be done for SW you develop but also for SW you buy: anything that goes into the product is concerned. In your case I assume you do not sell the final product but this the responsibility of your customer to do the risk management. And if you are class C you have to provide a whole lot of evidence : requirements, tests, FMEA, tests results, detailed design and so on.
This does not really answer what is or what is not a medical device. As far as I know in most jurisdiction it is linked to "claim".
If you are claiming that you system is curing or doing the diagnosis of some ailment, this is likely a medical device. On the other hand, if you do not claim anything (like homeopathy or food supplement), you are not regulated. That's why most of the time health mobile app do not claim to be able to diagnose anything.
- wmkn 8 years agoFor the EU what is a 'medical device' is pretty well described in 93/42/EEC which is the law describing the medical CE process (http://eur-lex.europa.eu/LexUriServ/LexUriServ.do?uri=CONSLE...)
There are quite a bit of edge cases though, so the definition of medical device actually has a guidance document that gives a bit more context (http://ec.europa.eu/DocsRoom/documents/10278/attachments/1/t...).
> I worked producing software that segmented images produced by a medical imaging device. The software would be built-in but it was the kind of thing one could have also have done exporting the images to Photoshop.
Segmentation software could certainly be considered a medical device if it has a medical purpose. For instance, if the segmentation is used to produce a diagnosis or a measurement. If you would like to sell it to medical professionals you would be required to have a medical CE for the software. Photoshop might have the same functionality, but it does not have a medical CE, so in principle it should not be used by medical professionals.
- davidw 8 years agoSomething like that might cause a misdiagnosis or cause a doctor to not see something, if done incorrectly.
OTOH, it's certainly not an invasive device or something where someone dies if it fails.
There are different categories of medical device, IIRC. The last place I worked did non-invasive diagnostic devices.
- seren 8 years ago
- daveguy 8 years agoYes. And they are required by the FDA:
http://www.fda.gov/MedicalDevices/DeviceRegulationandGuidanc...
Therac-25 is an example of what can happen when medical devices are not held to a high standard:
- jschwartzi 8 years agoThat document is guidance. In it they suggest static analysis and formal methods, but they do not require them. You might get asked questions during a 510k if you don't explicitly call them out, though.
Therac-25 is actually an example of how a bad risk analysis and lack of appropriate mitigations for the level of concern can lead to patient injury or death. The design of the device meant that a single-fault condition in software could create a hazardous situation for the patient. When they revised the design for software control they removed a hardware mitigation.
- Waterluvian 8 years agoI feel like "engineer overconfidence" is actually laziness.
Any time I've felt inclined to say it's fine, wasn't overconfidence. It was laziness to go through the whole process of re-validating something.
This seems like an embarrassing admission, but it was part of me learning how to be a programmer. The rookie me wanted to cut corners.
- jschwartzi 8 years ago
- wordtothedevice 8 years ago
- a3n 8 years ago
- geoelectric 8 years agoI know it was a throwaway, but be careful about the "untested code never works" comment. It plays into the fallacy that if it works, it's adequately tested.
You talk a lot about testing for specified requirements, but not so much about looking for undesired behavior outside the requirements. That's where the fallacy bites you. It's especially pernicious when you start to firm up a working prototype or are inheriting a legacy production release with a history of good behavior, but which is about to change execution context in some way. Changing the context means hitting unexplored territory and there be dragons.
Of course, you're a professional tester (as am I), and I know this is obvious for you. But for the average PHB or engineer-driven organization trying to prioritize, it's important to be pretty clear about this:
Untested code works great in the 95% case--this is one of the big reasons QA has a credibility problem sometimes, because they're spending $$$ to confirm it already works. It's the 5% case you need to worry about. It can kill your product or, in your case, the patient.
- julienzaegel 8 years ago(I'm the author of the aforementioned article)
Yes it was a throwaway :-) But I mean "untested" as not tested at all, by any means; code that has went through automated testing (unit tests and the like) has been tested quite a bit, so when QA receives it, the defect rate is not so overwhelming (5% or so as you say). I maintain that the code I write almost never works if I don't write automated tests for it (which is why I always write tests now :-))
You're right about the importance of "looking for undesired behavior outside the requirements". I'll think about an update of the article about manual tests. Thanks for the input!
- geoelectric 8 years agoI was QA prior to the advent of mandatory unit test for commit (I'm actually an SDET or tools/infra SWE nowadays, depending on hat) and, even then, untested code was mostly fine, as evidenced by the fields of green "PASS" spreadsheets.
Those spreadsheets either meant it was good when you got it or you did a crappy job testing it, and either way it was fine before being tested. Regression testing in particular is an exercise in generating "PASS" results, and historically QA practice has been dominated by regression testing.
Luckily, SW practices have changed enough that (as you say) most code is at least minimally tested by the developer and, maybe more importantly, some minimal level of unit testing is expected now as a basic professional standard. Hopefully we never move backwards again.
- geoelectric 8 years ago
- julienzaegel 8 years ago
- Beltiras 8 years agoI just started work at a medical software company. This was a godsend.
- thefastlane 8 years agomedical software and agile development ... what could go wrong?
- officialchicken 8 years agoNot too much actually. You need to have a design history file (DHF) and a Failure-Mode Engineering Assesment (FMEA) document as part of your "Quality" system for FDA submission, usually at the time the project starts. Defining and using agile methodologies forces those documents/proceedures to be more in-sync with the software system and any unit/system/e2e testing and CI. It also formalizes the testing process - bringing some structure to the agility of your chosen methodology.
- switchtodecaf 8 years agoAAMI TIR-45 ("Guidance on the use of agile practices in the development of medical device software") offers guidance by mapping agile practices to regulatory requirements, and there are many other good examples for implementing agile (small "a") methods in other highly-regulated industries.
- Jtsummers 8 years agoAgile itself is perfectly fine for this sort of software. It's a philosophy and approach, not a set of dogmatic rules.
We applied agile methods to developing safety systems for aircraft (something of a pilot project within the company, looking for a "better way"). It was quite effective, focusing on iterative development, small changes integrated frequently, frequent delivery to the customer for verification/validation, etc.
- cema 8 years agoWell, a lot could, of course. But the word agile is used rather loosely these days, and I think good advice can come from a blog with (almost) an arbitrary name.
- agumonkey 8 years agoThere's a balance. Slow and heavy development might create unadequate systems that cause more harm than good to doctors and patients. Proper iteration time might avoid this.
- officialchicken 8 years ago