Machine Learning (aka Artificial Intelligence) has put a spotlight on the importance of algorithms. While the term algorithm is ancient, meaning “a process or set of rules to be followed in problem-solving operations,” the modern use within refers to the complex way that the software behaves.
Before broad adoption of Machine Learning in other industries, algorithms in diagnostic and medical device software typically have been well-defined computer instructions that solve well-defined problems or compute results. This allowed developers to define the sequence and use bench and clinical data to verify and validate the computation. Once the computations matched the expected results, the algorithm was locked in, in preparation for FDA approval and market launch.
Machine Learning algorithms are, instead, complex, cross-linked, statistical models, linking inputs and outputs in ways that are difficult, if not intractable, for even the developers to fully understand. The benefit of Machine Learning algorithms is the ability to find patterns and data relationships that rule-based algorithms were too limited to identify, particularly in very large data sets.
This profound shift over the past decade has industry requesting FDA guidance on how to submit devices and diagnostic tools for approval when they utilize Machine Learning. The FDA’s Digital Health office has engaged with the public on numerous efforts to include the SaMD pilot participants providing their perspective. I, as an Entrepreneur in Residence (EIR) – Digital Health Expert – along with my industry colleagues, have been called upon to provide insight from industry processes utilizing Machine Learning. Additionally, the FDA has held several public forums, such as the February 25-26, 2020 “Workshop on the Evolving Role of Artificial Intelligence in Radiological Imaging” and the seven-hour patient engagement meeting last October; the FDA has also participated in standards organizations as well as international forums. The same concerns and questions invariably arise regarding the quality of the data used to develop the statistical Machine Learning models (called “training”). Not all health and clinical data is equal. Training algorithms on poor data can lead to inaccurate and even harmful results, while high-quality data can accelerate diagnostic data collection and decision-making processes, leading to a higher rate of efficacy.
The Digital Health office compiled feedback from over the years, numerous inputs and sources, to propose an action plan to evolve industry and the agency on Machine Learning. The plan addresses submissions with a tailored regulatory framework “Predetermined Change Control Plan” with a goal to publish draft guidance in 2021. Additionally, the FDA is committed to continuing to harmonize the Good Machine Learning Practice (GMLP) development, as well as other goals, to improve the use of Machine Learning.
While GMLP harmonization is critical, Machine Learning processes still require quality data. It is not that in the past good data was not required. However, the models (and, more importantly, the persons evaluating the models) had the ability to evaluate outliers and determine whether the outlier was an artifact of poor data or an accurate representation of product poor performance to justify removing the poor data. Yet, in Machine Learning, the poor data can be made into “noise” in the data, precisely because the models are generated without direct human review or understanding.
Companies that are seeking to position themselves for the future, by using Machine Learning to develop quality products, utilize real-world performance, identify manufacturing efficiencies, manage cyber security, or improve client engagement, first need to ensure that their data governance and software development lifecycle processes are levering best practices to establish the foundation for the harmonized GMLP.
Lachman Consultants is prepared to support its clients in evaluating opportunities and establishing GMLP processes for Machine Learning within devices and diagnostics, as well as part of complex manufacturing activities, to identify potential vulnerabilities that may have an adverse effect on patient safety.
