AI and reality – diagnosing otitis media is a real challenge

Let’s pretend for a moment that you receive a call from one of your college roommates who thanks to his family connections has become a venture capitalist in California. His group is considering investing in a start-up that is developing a handheld instrument that it claims will use artificial intelligence to diagnose ear infections far more accurately than the human eye. He wonders if you would like to help him evaluate the company’s proposal and offers you a small percentage of the profits for your efforts should they choose to invest.

Your former roommate has done enough research on his own to understand that otitis media makes up a large chunk of a pediatrician’s workload and that making an accurate diagnosis can often be difficult in a struggling child. He describes his own experience watching a frustrated pediatrician attempting to remove wax from his child’s ear and eventually prescribing antibiotics “to be safe.”

You agree and review the prospectus, which includes a paper from a peer-reviewed journal. What you discover is that the investigators used more than 600 high-resolution images of tympanic membranes taken “during operative myringotomy and tympanostomy tube placement” and the findings at tympanocentesis to train a neural network.

Once trained, the model they developed could differentiate with 95% accuracy between an image of a tympanic membrane that covered a normal middle ear from one that merely contained fluid and from one that contained infected fluid. When these same images were shown to 39 clinicians, more than half of which were pediatricians and included both faculty-level staff and trainees, the average diagnostic accuracy was 65%.

The prospectus includes prediction that this technology could easily be developed into a handheld instrument similar to a traditional otoscope, which could then be linked to the operator’s smartphone, giving the clinician an instant treat or no-treat answer.

Now, remember you have nothing to lose except maybe a friendship. How would you advise your old college roommate?

My advice to your college buddy would be one of caution! Yes, there is a potential big upside because there is a real need for a device that could provide a diagnostic accuracy that this AI model promises. While I suspect that AI will always be more accurate in diagnosis using static images, I bet that most people, clinicians and nonclinicians, could improve their accuracy by linking photos with diagnoses with an hour of practice.

However, evaluating a high-resolution photograph taken through an operative scope inserted into the cerumenless ear canal of a sedated, afrebrile child is several orders of magnitude less difficult than the real-world environment in which the diagnosis of otitis media is usually made.

If the venture capitalists were still interested in getting into the otitis media marketplace, you might suggest they look into companies that have already developed image capture otoscopes. At this point I could only find one on the Internet that was portable and it certainly isn’t small-child friendly. Once we have a tool that can capture images in real-world situations, the next step is to train AI systems to interpret them using the approach these researchers have developed. I bet it can be done. It will be only a matter of time ... and money.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Other than a Littman stethoscope he accepted as a first-year medical student in 1966, Dr. Wilkoff reports having nothing to disclose. Email him at [email protected].

Let’s pretend for a moment that you receive a call from one of your college roommates who thanks to his family connections has become a venture capitalist in California. His group is considering investing in a start-up that is developing a handheld instrument that it claims will use artificial intelligence to diagnose ear infections far more accurately than the human eye. He wonders if you would like to help him evaluate the company’s proposal and offers you a small percentage of the profits for your efforts should they choose to invest.

Your former roommate has done enough research on his own to understand that otitis media makes up a large chunk of a pediatrician’s workload and that making an accurate diagnosis can often be difficult in a struggling child. He describes his own experience watching a frustrated pediatrician attempting to remove wax from his child’s ear and eventually prescribing antibiotics “to be safe.”

You agree and review the prospectus, which includes a paper from a peer-reviewed journal. What you discover is that the investigators used more than 600 high-resolution images of tympanic membranes taken “during operative myringotomy and tympanostomy tube placement” and the findings at tympanocentesis to train a neural network.

Once trained, the model they developed could differentiate with 95% accuracy between an image of a tympanic membrane that covered a normal middle ear from one that merely contained fluid and from one that contained infected fluid. When these same images were shown to 39 clinicians, more than half of which were pediatricians and included both faculty-level staff and trainees, the average diagnostic accuracy was 65%.

The prospectus includes prediction that this technology could easily be developed into a handheld instrument similar to a traditional otoscope, which could then be linked to the operator’s smartphone, giving the clinician an instant treat or no-treat answer.

Now, remember you have nothing to lose except maybe a friendship. How would you advise your old college roommate?

My advice to your college buddy would be one of caution! Yes, there is a potential big upside because there is a real need for a device that could provide a diagnostic accuracy that this AI model promises. While I suspect that AI will always be more accurate in diagnosis using static images, I bet that most people, clinicians and nonclinicians, could improve their accuracy by linking photos with diagnoses with an hour of practice.

However, evaluating a high-resolution photograph taken through an operative scope inserted into the cerumenless ear canal of a sedated, afrebrile child is several orders of magnitude less difficult than the real-world environment in which the diagnosis of otitis media is usually made.

If the venture capitalists were still interested in getting into the otitis media marketplace, you might suggest they look into companies that have already developed image capture otoscopes. At this point I could only find one on the Internet that was portable and it certainly isn’t small-child friendly. Once we have a tool that can capture images in real-world situations, the next step is to train AI systems to interpret them using the approach these researchers have developed. I bet it can be done. It will be only a matter of time ... and money.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Other than a Littman stethoscope he accepted as a first-year medical student in 1966, Dr. Wilkoff reports having nothing to disclose. Email him at [email protected].

Let’s pretend for a moment that you receive a call from one of your college roommates who thanks to his family connections has become a venture capitalist in California. His group is considering investing in a start-up that is developing a handheld instrument that it claims will use artificial intelligence to diagnose ear infections far more accurately than the human eye. He wonders if you would like to help him evaluate the company’s proposal and offers you a small percentage of the profits for your efforts should they choose to invest.

Your former roommate has done enough research on his own to understand that otitis media makes up a large chunk of a pediatrician’s workload and that making an accurate diagnosis can often be difficult in a struggling child. He describes his own experience watching a frustrated pediatrician attempting to remove wax from his child’s ear and eventually prescribing antibiotics “to be safe.”

You agree and review the prospectus, which includes a paper from a peer-reviewed journal. What you discover is that the investigators used more than 600 high-resolution images of tympanic membranes taken “during operative myringotomy and tympanostomy tube placement” and the findings at tympanocentesis to train a neural network.

Once trained, the model they developed could differentiate with 95% accuracy between an image of a tympanic membrane that covered a normal middle ear from one that merely contained fluid and from one that contained infected fluid. When these same images were shown to 39 clinicians, more than half of which were pediatricians and included both faculty-level staff and trainees, the average diagnostic accuracy was 65%.

The prospectus includes prediction that this technology could easily be developed into a handheld instrument similar to a traditional otoscope, which could then be linked to the operator’s smartphone, giving the clinician an instant treat or no-treat answer.

Now, remember you have nothing to lose except maybe a friendship. How would you advise your old college roommate?

My advice to your college buddy would be one of caution! Yes, there is a potential big upside because there is a real need for a device that could provide a diagnostic accuracy that this AI model promises. While I suspect that AI will always be more accurate in diagnosis using static images, I bet that most people, clinicians and nonclinicians, could improve their accuracy by linking photos with diagnoses with an hour of practice.

However, evaluating a high-resolution photograph taken through an operative scope inserted into the cerumenless ear canal of a sedated, afrebrile child is several orders of magnitude less difficult than the real-world environment in which the diagnosis of otitis media is usually made.

If the venture capitalists were still interested in getting into the otitis media marketplace, you might suggest they look into companies that have already developed image capture otoscopes. At this point I could only find one on the Internet that was portable and it certainly isn’t small-child friendly. Once we have a tool that can capture images in real-world situations, the next step is to train AI systems to interpret them using the approach these researchers have developed. I bet it can be done. It will be only a matter of time ... and money.

Dr. Wilkoff practiced primary care pediatrics in Brunswick, Maine, for nearly 40 years. He has authored several books on behavioral pediatrics, including “How to Say No to Your Toddler.” Other than a Littman stethoscope he accepted as a first-year medical student in 1966, Dr. Wilkoff reports having nothing to disclose. Email him at [email protected].