Valencell President and Co-Founder Steven F. LeBoeuf, Ph.D.
After surviving several pre-sub correspondence letters from the FDA seeking 510(k) clearance for Valencell’s cuffless, no-calibration fingertip blood pressure monitor (BPM),Not FDA approved.Not sold in the United States as of the publication date of this article), I have become familiar with the FDA. Good Machine Learning Practice (GMLP) for Medical Device Development: Guiding Principles;1 I love this book for at least two reasons. 1) It’s an incredibly fast read, and 2) It’s spot on. This guidance document outlines 10 guiding principles, each of which is important in the development of medical devices powered by machine learning (ML). This article summarizes the 10 principles and provides real-world examples of how they can be adopted.
1. Multidisciplinary expertise across the entire product lifecycle
The important point is that data science does not work alone when it comes to developing machine learning models for medical devices. It is important that a multidisciplinary team of scientists, engineers, clinicians, and regulatory experts is assembled with a deep understanding of the intended use and regulatory context. clinical A problem that is being solved. We emphasize “clinical” here. Data science teams tend to fixate on optimizing (and sometimes over-optimizing) machine learning models rather than ensuring that a complete system-level clinical solution is fit for its intended use. is. In addition to building a multidisciplinary team within Valencell, we assigned representatives from each major discipline to meet regularly to adjust efforts and update timelines as needed.
2. Good software engineering and security practices
Medical technology teams must have good software engineering practices, data quality assurance, data management, and robust cybersecurity practices. At Valencell, our approach to this was to track data from the beginning to the end of the product development cycle. By doing this, you can uncover risks and mitigations for maintaining data integrity and security. For example, our products must autonomously retrieve various diagnostics and metadata from users in order to receive services in the field. Ensuring the security and integrity of this data became part of the development process.
3. Focus on your target patient population
It is important to ensure the generalizability of machine learning models across the patient population of interest, confirm the usefulness of the model, and identify where model performance may deteriorate. Regulators want assurance that machine learning models are accurate across all ages, weights, and heights within the range of indications for use. This guarantee is especially important for non-invasive medical wearables, where metadata is incorporated into machine learning models to help improve estimation accuracy.
4. Independence of training dataset and test set
I have seen this golden rule broken time and time again, both in peer-reviewed publications and in FDA-approved medical devices. Machine learning models always show higher accuracy on the data set used for training than on the real data set. Medical devices that employ machine learning are ultimately judged by the performance of subjects not used for model development, so subjects used to train the model cannot be used for clinical validation studies. But training and testing independence goes beyond the data set itself. It’s also important to ensure that your machine learning solution is independent of your test environment. For example, Valensel’s upcoming clinical validation study will require collecting data from participants in three of her geographically distinct locations. The reason is that even if test-training subject independence is ensured, there is always a risk that the training and test datasets will be confounded by environmental factors that may bias the clinical validation results. .
5. Reference dataset based on best available methods
The key is to develop machine learning models with sufficiently robust reference datasets to facilitate generalization across the patient population of interest. Don’t make the beginner’s mistake of choosing a reference dataset that is too narrow to ensure generalizability of the model across a wide patient population (as we did in the initial feasibility study). To achieve the desired accuracy across the population of interest, reference data sets must be enriched to include a more diverse representation of subjects.
6. Model design according to the intended use of the device
The clinical benefits and risks must be well understood so that clinically meaningful performance goals can be derived while remaining safe and effective within the intended range of use. Blood pressure is not just “one thing”, it can mean many things depending on the use case, so for us, the focus was on the use case of making spot checking of blood pressure easier and more convenient, rather than passive monitoring. It was important to guess. -Beat monitoring, night monitoring, etc.
7. Focus on the performance of teams that combine humans and AI
Humans are always part of medical device solutions. Even if a doctor doesn’t have to interpret the results, at least the patient has to interact with the device. The development of ML solutions must consider this factor, especially regarding risks and mitigations. It is important to recognize that end users may misuse devices and to autonomously detect these occurrences and prevent false measurements where possible. Storyboarding the human-machine interaction is critical to developing appropriate mitigations.
8. Demonstration of device performance in clinically relevant conditions
It must be demonstrated that the solution is safe and effective for the clinically relevant intended use within the indication of use. This guideline is easier to meet when the clinical relevance is already established, such as in the pursuit of a 510(k).
9. Ability to provide clear and important information to users
While this bit of guidance may seem despicably obvious in medical device development, special considerations arise when applied to devices that employ ML. For example, ML models may behave differently between different subgroups or within different clinical workflows. This must be effectively communicated to the end user. Consider employing sensor analytics that can determine where differences in model performance may be and communicate that to your end users.
10. Monitor model performance over time and manage retraining risks
If the product employs a static ML model, no retraining/tuning is required by the user. In contrast, many ML solutions are designed to learn and improve over time, but they are not without the risk of overfitting, unintended bias, and data drift. As ML models adapt over time, these risks must be managed within the user base.
reference
About the author:
Dr. Steven LeBoeuf is President and Co-Founder of. Valensel. He is an inventor with over 100 patents in the field of wearable biomedical sensing, and an innovator in his wearable PPG sensor, which is incorporated into millions of wearables currently on the market. there is. Before he founded Valencell in 2006, Mr. Leboeuf was a senior scientist at General Electric Company and a biosensor project leader where he worked on solid-state materials, multiwavelength optoelectronic devices, high-power electronics, nanostructured materials and devices, We worked on innovations in biochemical sensor systems. LeBoeuf received his Ph.D. He earned a bachelor’s degree in electrical engineering from North Carolina State University and a bachelor’s degree in electrical engineering and mathematics from Louisiana Tech University.