Lecture: AI-Integrated Screening Models in India: Lessons for the World

>> DR. ANAND VINEKAR: Hello, everybody, good morning, good evening, in different parts of the world. Thank you for joining us for this really exciting webinar by Cybersight. I’m joined by my friend and colleague, Dr. Shalinder Sabherwal from Delhi. I’m Dr. Ananda Vinekar, both of us are from India. Both of us are just clinical clinician ophthalmologists and much of the work that we might talk about may actually be going above our heads. We may not even be able to answer some of your technical questions, because AI is such a black hole, and sometimes we don’t even know how it’s doing, what it’s doing. However, our intention is to present work that has a very specific India and lower middle income country context. And therefore there’s a lot of great work going on in AI, both in diabetic retinopathy, glaucoma, and retinopathy of maturity across the world and we may not be able to do justice in all those models in our talks. A quick roundup of how we’ll conduct this. I’m going to speak about AI models and retinopathy of maturity for 15 minutes followed by Dr. Sabherwal who will talk about AI models for diabetic and diabetic retinopathy. That will leave us anywhere between 15 and 25 minutes of Q&A. You can keep feeding in your questions. Some of you have already done that. We hope we’ll be able to take as many questions as possible after both our talks. We might even do it after our individual talks. So without wasting more time, let me share my screen and begin my presentation. Okay. I hope that is visible and I’m audible, so let me begin. So I’m going to be talking about AI models, integrating it into the real world for retinopathy of maturity. Firstly, AI is such a big buzzword. We have AI running on our phones, maybe even traffic signals. So why AI in ROP? Is it just a cool thing to do? It’s certainly cool, but a little more than that. There are more reasons for getting AI into ROP. Firstly, the numbers are absolutely staggering. Look at where India is, a decade later, from 3.5 million, we’re still over 3 million preterm babies born annually. Look at the other countries as well. What’s happening here is there are tons of babies who are being born premature, who survive, and of course who needed to be screened, and then there are fewer and fewer ROP specialists to do it, which basically means that it might come down to imaging, and imaging therefore naturally lends itself to AI models. Firstly, why is there such a big ROP problem? This is an editorial we wrote. Other than the fact that blindness is a big deal if screening programs are not in place, the fact today that there is much better survival of preterm babies has added to the burden. As I mentioned, it’s really all about imaging and images which are lending itself for the AI model. Let’s look at the new international model for ROP which came just about three years back. In the preamble itself of that classification, it talks about one of the reasons why the new classification was required. And look what it says. The innovations in ophthalmic imaging. Right from understanding the notch, which is a disease going into the posterior part of the macula, to the fact that plus disease and preplus disease, a term that came in 2005, is so variable that we must agree to disagree. And to further complicate our lives, we’ve started addressing things like mild preplus, preplus, severe plus, and so on. Well, at least the ICROP3 classification decided we must now accept the fact that today plus disease is a spectrum, from no plus at all there could be very obvious plus. And to help us understand the vascularity or vascular abnormalities, we look at green which is none at all, to red. Some of the stages of ROP, this is not a talk that will address that in detail, in just in the context of AI, when there’s a simple line, stage 1, stage 2 is a slightly thicker line, which has volume and is raised. Stage 3 is newer neovascularization. You also have the notch, you have other abnormalities like popcorn. And there are combinations thereof. The other scary part of ROP which we’re seeing more and more often now in middle income countries, especially in India, is aggressive ROP, the older word being posterior aggressive ROP. The second reason imaging is becoming so much more commonplace is the fact that we now have affordable and portable cameras. Let’s take the Neo, it is more portable, lightweight and has a better resolution. In this publication we compared the RetCam family of cameras with the Neo. You can use your cellphone onto devices, great for photo documentation. Still not completely integrated into a telemedicine platform. The point being we’re going to have tons and tons of image as camera costs go south. In the format of Cybersight we’ll have a few polling questions. This is up there for you to read, I’ll allow you 30 seconds after which the Cybersight team will put out the results of your poll. So with regards to plus disease, is it easy to differentiate — it is easy to differentiate arterial and venous tortuosity from each other, preplus and plus disease are distinct and each other, preplus disease is sufficient to make a call for treatment, and the P score provides some objectivity in understanding the gradient. Go ahead and put your answers and we’ll see the results. So 24% — it’s almost equally divided. That’s exactly the purpose of this poll, to tell you that we now have some objectivity in place. The fact that we were struggling with plus disease across the world, all of them said that experts hardly agree on what is plus disease. This publication came out only last month from a few of us in the ICROP group where we hope that some amount of objectivity will come in, from P1 which is no plus at all, to P9 which is very obvious plus. We hope that in some way we will be able to communicate with each other by saying today it is P3, last week it was P2, and so on. Regression and reactivation is for us to get the complete understanding of this ROP process. So regression has we can understand is the disease coming down on its own, and unfortunately after anti-VEGF there are different patterns of regression. Reactivation is when the disease come back. The disease can come back in several different forms, either in the form of abnormal plus disease, or may the reappearance as stage 3. The second poll related to the last three slides, go regression and reactivation. The patterns are similar after laser or anti-VEGF, incomplete regression is more common after laser than anti-VEGF, the timing of recurrence after anti-VEGF depends on the drug used, and reactivation is always characterized with reappearance of plus and ridge tissue. We’ll give you 30 seconds for the poll. Great. That’s the question up for you and your format, in case you’re following this format on your cellular phones, you’ll also see this coming up on your screen. Okay. Let’s have the result. Wonderful. Okay. So the point, again, it’s almost equally divided, but the timing of the recurrence depends on the drug, okay? And all of the others are false, which is something we look forward to going into as the talk happens and perhaps a bit in the Q&A as well. The journey of artificial intelligence, at least in our program which is in its 18th year, stems from the fact that we started screening more and more, as more cameras, both the RetCam and the Neo started getting used, we have more and more images, close to 100,000 every month. We realized as this program was scaling up into different states and countries, we’ll be dealing with a ton of images which we must know how to decode. About a decade back when we were dealing with some low quality images at certain times in ROP, we figured that maybe there are algorithms to better understand or pull out more clinically useful information. So we always had this problem where we couldn’t see the vessels reaching the end or some amount of capillary nonperfusion which is not picked up, sometimes the resolution is bad and quantification was not possible. So this is work we did a company that used contrast limited adaptive histogram equalization techniques. I’m just going to cut to the chase to say that we used three filters, color enhancement, which allowed us to see capillary nonperfusion, gray enhance which allowed us to see the ridges raised and vessels ended. This is by trial and error, as you can see, sometimes it gave too much noise, like in image 2 or 4, and sometimes too little noise. The details of the clinical picture was being missed. The vesselness measure was interesting, we were able to see smaller capillaries that extended beyond where the clinical picture showed it would. Let me illustrate with an example. This is a case of posterior zone 1 of aggressive ROP. If you run our algorithms of C protocol and more importantly the vessel protocol, you can see the vessels extend much beyond. These are probably capillaries, they were always there, just being missed because of the detail. Close to 15 to 20% of these images that were classified as zone 1 or even zone half or posterior zone 1 were reclassified as zone 2 and vice-versa as well because we were able to see finer details extending more anteriorly. So that’s really the advantage of doing that. Now comes a segmentation and quantification, this was work done with an engineering college in Bangor. Our idea was to look at the subtype of aggressive posterior ROP in those days. We looked at not only the tortuosity, but in zone 2, because posterior ROP was said to occur in zone 1 and zone 2 posterior. What’s important is we were able to quantify it. Plus disease we spoke about. What about detection? This is another bit of work we did with two independent engineering institutions in Bangalore and in southern India. We looked at plus disease and tried to use adaptive segmentation. Here, since preplus was two quadrants in those days by definition, more than two quadrants was plus disease, we looked to see if it could be extrapolated to the entire image. This is work done by a group which helped detect the zones. Even when the macula was not fully developed. Let’s understand that by definition of zone, when we said the center of the macula, if the macula itself is not well-developed, how do you make that distinctions, that’s where they made automatic detection of all zones. That’s been a very good tool that can be used clinically. In terms of three-dimensional reconstruction, this is work that we worked with, again, engineering colleges in Bangalore and outside, let’s look at this case of retinal detachment which is stage 4A, potentially 4B, and the way the retina is ballooned, we reconstructed the image in three dimensions to understand how it was lifted. Remember, these images were being taken by health care workers. They would be helped by understanding whether stage 2, flatter, or stage 3, by definition extra retina. Using deep learning methods, this is work we did in technology, they started using poor quality images to enhance them, process them, work on their ridge using ridge detection methods, filtering them, looking at them under different filters, then focusing on the region of interest and then comparing it with the ground group and then looking at the ones with poor quality. Basically, all of that, what it did was in images that were very obvious stage 2 or with ridge, we extrapolated the information we learned from there and were able to pick it up in images that did not have good quality but were having the disease itself. This was another deep learning approach which we did the Institute of Technology. Here they adopted a different format where first we labeled it and then there was some pseudo labeling based on the algorithm itself and then they fed the true and the pseudo labeling back into the algorithm to see where the accuracy improved. It was very good, 99%, with great range of sensitivity as well. The images that were misqualified, the idea being even with a poor quality image, can we extract clinically useful information. This is the publication compared with other similar models of deep learning using the deep learning network. In automatic detection, this was a single person funded Ph.D. where her algorithm basically learned stage 2 and was able to differentiate it from a stage 3. As you can see, the accuracy, in the early ’90s, and they’re only improving as we put integrated into the real world. This is work done by Pete Campbell’s group in association with the groups in India, Nepal, and Mongolia. Again, the idea was to see whether AI can help a population to reduce the number of unnecessary screening visits by flagging them at a different score and can they identify high risk infants who need to be more closely monitored who might potentially go into treatment requiring ROP. They showed this is possible across ethnicities as well. This is our own work where we used AI across three public institutions, where became of we said each of those models works in their own ecosystem. Can we have the leader of that system and the people whom they train, are they on sync with the way they classify the ground truth? That was the idea on whether AI can be used for training, and training across the board under different circumstances. And as you can see, there was the green polygrams were we agreed and the red, where we disagreed. Early technicians are learning. By using plus only, there was a good accuracy, and by using ridge detection only, there was decent accuracy. By combining the two we can say the algorithm shows the baby needs treatment or is normal. The technicians in the out field where validated by ROP specialists sitting in the main center. Our work was to focus on the true negatives and positives and reduce the false positives and false negatives. Here as you can see, it worked across the RetCam and Neo. I’m going to use the interface to play, this is how the live video works. This is part of an ongoing grant with government of India’s buyback department. We have some milestones to meet by next year and we hope the algorithm will be ready. As you can see, it’s going to be device agnostic. Here what we’re doing quite manually is uploading the right eye and left eye and submitting it for prediction. As you can see shortly, in a few seconds, the only thing that the speed would improve as it gets integrated into the real world, and as you can see, the report shows up in a few seconds. And this is exactly live, this video has been not been edited, this is how long it will take in the current scenario. You can see it will generate a report where the heat map will show you where the disease is and it will tell you whether this baby needs to be referred or treated or is completely normal. You can see this is early stage 3 in zone 2 posterior with plus disease or preplus disease. It’s not telling you all of that as of now but it’s telling you this baby needs referral. As you can see, this is really good, it’s almost 100% when you compare it with the experts. It drops a bit when you compare it with your technicians who are screening in the outreach. I’m going to end with the fact that we can add the fundus images, several more modalities. I bring to your attention angiography which is coming of age now because of anti-VEGF. OCT angiography could lend itself to artificial intelligence. As of now the devices are not specific to pediatric and therefore much more work needs to be done. And of course OCD which has come up, we have been doing some work in OCT of infants where we’ve shown some of those layers may not appear at all and they can code whether these babies will have poor or normal vision. And I’m going to end with some exciting fact that we are now almost able to detect predict whether our babies will develop ROP as they undergo screening by taking their tears during the first or second screening, examining their biomarkers, and then as these biomarkers increase or decrease, we’re able to predict whether these babies will eventually will develop ROP or even need treatment. Some of our publications that address this, this can very much lend itself into the AI algorithm because we’re looking at a systemic risk factor. So an image along with a risk factor along with the biomarker could be the way where we could predict these babies on developing ROP or not. In the first paper we looked at angiogenic factors. In our second publication we looked at the interplay of markers. In our third publication we looked at the role of vitamin D in VEGF modulation. So summarize, advances in affordable imaging has really changed the way we are diagnosing, screening, treating, and following up these babies with ROP. The fact that there are so many babies to be screened and AI and machine learning, is going to be here to stay. And as real world integration happens, I think we are going to be seeing in the next few years a paradigm shift in the way we are going to utilize AI. I’m going to end with this final polling question. All of these answers are right, none are wrong. It’s for me to understand where do you think the maximal focus will be, which will help us find our own AI models. A last dig at this before I hand over the podium to Dr. Sabherwal. Okay. Can we see the results? Okay. So I can see there’s a little more on the fact that it is going to reduce costs and scale up the AI model, wonderful. Thank you so much for listening. I’m going to stop share. And we are now going to invite Dr. Shalinder Sabherwal who is going to talk about the other part of AI in glaucoma and diabetic retinopathy. >> DR. SHALINDER SABHERWAL: Thank you, Dr. Vinekar. So I think my talk is more on the AI-integrated community model for diabetic retinopathy and glaucoma. And it’s basically impact. It’s not going to be as technical because fortunately most of the algorithms have been developed and they’ve been working well. There have been published studies showing the accuracy of this detection for diabetic retinopathy and glaucoma, especially the referable part of it, so I will not go into the technical part. I would rather go into how we’re using it and how it can impact the community. I have to refine the context of the community setting. Just before that I would just like to roll out this poll. Just to know how what are we seeing, especially in India, because we’ve been working in an Indian context, what do you think is the sort of people with diabetes who do not know their status? And the second question is about those who know their status, how many would have not ever got their eyes screened, and this is all published data, and what percentage of glaucoma might be undetected in India. I’ll wait to see the responses to this poll. So I think for the first one, this is basically to highlight the need. And I’ll come to this subject later. This was done by 2015 and 2019. We found between 30 to 40% of people actually did not know about their diabetes status. And almost 90% had never got their eyes tested, even among those who knew they were diabetics. Some studies show up to 90% of people with glaucoma might be undetected. Before going into the topic, I would just like to share our community context. We worked at Dr. Shroff’s hospital, and we worked at the community level by doing door to door screenings and referring people with the need for eye care, the vision centers are the primary eye care centers. We have a technician and optometrist who do a comprehensive eye examination facilitated with the equipment required. They may refer to the next level which would be a surgical center, might be a certain number of kilometers away from the vision centers. They do charity work including management of diabetic retinopathy and glaucoma. We have a care center in Delhi which would do research and training. So why, going to this AI-integrated model for community care? In our own context, actually, we did a survey which is quite standardized in terms of finding a prevalence for blindness and its causes in a defined catchment. We did this around the care center that we have in the most populous state in India. The catchment was 2 million people. What we found was not very different, in fact even worse than what was the national averages. 50% of people did not know, they were not aware their diabetes. 85% of people never got their eye tested even if they knew they were diabetics. 90% among women. Glaucoma can be undetected in India. Both these conditions are not amenable to screening. What happens is we have to use every opportunity available for opportunistic screening for these conditions. That’s where the context comes in. I think before I proceed I would still we are still learning. We have consolidated our finding for the last six months. Out of 100 primary eye care centers, right now we’re in 20% of those centers which are covered where the interventions are at vision center level. We already have a technician there. It’s a referral point from the community and it’s closer to where people are and it doesn’t have an ophthalmologist. So what we have found is the training on this, using fundus on phone, and the training is pretty easy. What we’ve found in our experience, we can easily train a technician in a day or two, and then it’s only about refining the technique, and the training includes the training on the platform, as well as actually capturing good quality pictures. Is it easier than training the technician on 90D or retina cup? Definitely it is. But more than the easy part, it also standardizes the training, especially at the level where there is not a technician. We found it possible to train. Coming to the images, if you have an ideal condition, I’ll come to that too, you get good pictures. The picture on the left shows the hemorrhages, may or may not be diabetes, but it did definitely not a normal fundus picture. The picture on the right shows a disc which can be labeled as glaucomatous. It shows the signs available on the photo even if it is not very clear. As for the reports, you do get sort of automated reports. Here there is always a word of caution, it’s not diagnosis, it’s screening, and it is indicated, and it can actually highlight where you’re seeing the lesions for the technician. For glaucoma, it can point out the disc, it is seeing — how it is seeing the disc and how it is evaluating it. Also the fiber layer in different quadrants. Once you get the report, another sort of thing we notice is sometimes the images are not ideal. You can still bypass the sort of the quality check there. And it will give you a report saying that although you have bypassed the quality check, so you have to be cautious, but still it will tell you whether it is a referable case or not, which sometimes is pretty useful depending on the conditions that we find at the vision center. We do not have to dilate patients at the vision center which many of us are doing at primary. As for the data availability, we can show you what vision center is doing, where the stations are. You can actually filter down data for each vision center to go into the depth of the data. With the new sort of dashboards being developed, it is supposed to be becoming more visual. And it can give you more data in terms of what exactly what you are asking for. So it is getting more granular. This is sort of our experience, we have consolidated in the last five or six months the number of patients screened have been increasing, and so has been the number of patients seen and detected for possible diabetic retinopathy, glaucoma and AMD. As for our data, what we are finding is about 12% with diabetic retinopathy, glaucoma up to 7%, for AMD around 3%. Coming to true positive, when you actually filter them down to see what happens at the hospital, for the condition itself, we are finding the sensitivity of around 66%. Exactly the positive prediction rate that is there. If I actually look at only whether they were referable or not, that rate goes to 92%. That means that person definitely has a lesion for which they should go to the hospital. When we roll out the program in public health prospective, we have to have some indicators for monitoring the program. The first thing that we try to see is what percent of patients are being at least screened, out of all attending the vision center above age 35 or more. But looking at this in isolation doesn’t work in the real world scenario. You have to assess the people, depending on a crowded vision center, you have to match it with what the technician is seeing daily and facilitate a junior person who can actually do the screening or a second technician is required. Some of them are not screened because they come for cataract surgery followup. Some of them are feeling cataract grade 1. Charging problems, and only single vision technicians are available. These are facts that come from the real world, limiting the utilization. Then comes the monitoring of quality part of it. Fortunately or unfortunately, right now for them there is no benchmark, because it will total depended on type of people or patients that are coming to you in a real world sort of situation and with different sizes of pupil and sort of cataract grades. We try to compare for each and every vision center, we feel they should not be very different. How many actually ended up attending the referral made is a basically quality indicator for any public health program, that means how many were actually connected to care. That is wavering, and I’ll come to that, how we can address that. Then it comes to true positive, it can be an indication of the level of pictures being taken or the training which has been given. Another thing which the Medios remedial platform helps with, what percentage of patients, you had to bypass the quality to get the report. This could be due to multiple reasons. It could just be because of pupil size and the quality is still not there but the indication for a referable case. It could be because of early cataract. But it could also be because of training and looking at these variations around the vision centers, we are in sort of planning a second rate of training where we are only going to basically focus on the quality of images being taken by a technician. And some of the centers now are actually thinking of should we go and dilate these patients where we are actually bypassing quality, instead of just pushing them forward. That’s something which has to be tested still. Sort of the major goal for us right now is increasing compliance to referral. We’re starting to put these pictures on the ophthalmology platform we use. And talking to the patient, to motivate them more to come, because there is no other sort of facilitation given to the patient to come to the hospital which might be 50, 70 kilometers away from the vision center. There is some ongoing research, basically more on the operation side. One is that we are looking at is basically a multicenter project to look at the efficacy of these tools at the vision center. So apart from true positive, how many actually ended up coming in the real world? That is basically the output of the whole intervention. And the second thing we found interesting was that would we use the camera after dilating the pupil to make sure they screen the fundus before cataract surgery so that you’re not dependent on an ophthalmologist to make the case for cataract surgery and the patient can proceed from the vision center to the hospital with more surety of being fitted for cataract surgery. Whenever we talk about AI in real world we have to be careful about certain things which can be a roadblock. One is about data confidentiality. We should not be sharing any sort of confidential information on the patient even on a platform given by the company or which is not your own secure. Consent has to be taken from the patient if you’re taking images. At the end of the day you have to know that still it is a screening tool, so no diagnosis should be offered on the basis of the screening. So at the end I would like to thank the supporters of these cameras and program. We hope to cover all our vision centers soon and hopefully learn more from the ongoing research. Thank you. >> DR. ANAND VINEKAR: Thank you so much, Dr. Sabherwal. I think we can now take questions. We have a good 15 to 18 minutes. Let me pull up some of them. And let’s probably take them in order in which they have been asked. The ones that are addressed to me, I’ll go ahead and reply, and then Shalinder can take the ones address to him. Is AI based screening covered with health insurance? Not yet. In fact AI is still not part of mainstream ROP diagnosis or even screening. So the question of getting it covered by insurance is probably going to be the next step, which leads to another question on the medical legal aspects, but first, the second question is, Shalinder, to you. What are the medical legal aspects in AI based diabetic retinopathy screening? >> DR. SHALINDER SABHERWAL: I think there are no clear-cut guidelines. I would say what I have mentioned in the last sort of slides is to make sure that, one, this is not a diagnostic tool, it is a screening tool. Consent for taking a photograph is a must. And you have to make sure the data remains secure. Because health care data is actually the top notch priority, and you have to make sure it remains on your own platform which is secure and not shared with any confidential information. Maybe I’ll stop there and there might be more. This is what I’m aware. >> DR. ANAND VINEKAR: Great, thank you. India has a new DPDT law. As far as imaging is concerned in ROP, we use it as patient information, we have 20 sometimes preserve — and there are time periods for which the last days, how long you need to keep these reports, but technically an ROP image should be kept until the child turns into an adult, in fact until the age of 21, because it’s 18 years plus three, and that’s what some of the older laws did and we’re still waiting for clarification of the rules. The next question is, what is the accuracy of the diabetic retinopathy and glaucoma screening with AI, are there any misdiagnosis reports? >> DR. SHALINDER SABHERWAL: Thanks for the question. For the published reports, yes, for diabetic retinopathy and glaucoma, whatever is published, it’s more than 95% sensitivity. So that is what is available in literature. As far as, as I said, research is still ongoing, I don’t have my published data right now to share, but our cases, like I said, we are finding somewhere between 60 to 70% patients actually being referred, actually needing that referral for that condition. If I say for all the conditions together, it’s quite reasonable in terms of public health intervention. >> DR. ANAND VINEKAR: Thank you. The next question is addressed to me. It says basically — it refers to the slide in which I showed the vesselness measure. It says I showed region zone 1 and then the segmentation algorithm showing the capillaries are beyond zone 1 and it actually reached zone 2. Did I have a ground truth for the model to learn and did I confirm these findings with an algorithm for clearer images. Okay, so great question. This work actually was published in 2012, 2013, even before AI was a thing, we were doing mostly machine learning and doing segmentation and better understanding poor quality images. However, at that time the ground truth that we used was aclinical, and in certain cases fluorescein angiography. We were able to see where the vessels were. The image was poor quality for several reasons. Germane to the ROP problem, it precludes great images taken, and that’s when AI algorithms might work. I just hope that that answered the question. The next question is to you, Shalinder, is FDA clearance required, if yes, what are the FDA-approved AI applications available for DR? >> DR. SHALINDER SABHERWAL: Sorry, I’m not aware of this, but I can find this out. >> DR. ANAND VINEKAR: Yeah, I think AI, definitely I can say for ROP, that’s work in progress. There are some models that have been applied for, and I think if it has to become mainstream, it will require more of that certification. This is a general question I think for both of us, perhaps. What is the cost of screening — I think it refers to you because you had spoken about the public end model. >> DR. SHALINDER SABHERWAL: I think in our context we have still not calculated that. There are some studies that show not the costing but it shows cost effectiveness of the screening. But I think the research they’re doing right now would actually lead us to. It’s not screening for one person but connecting a true positive case to care. That will complete the cycle. Hopefully down the line we’ll have those results. Right now it’s very difficult to say in our context. As per the theoretical cost effectiveness as published in some western studies, it shows it’s cost effective in terms of the detection rate and what comes out of it. >> DR. ANAND VINEKAR: A followup question on your model, is the next question. What are the other health care delivery use cases in using AI for DR and glaucoma apart from the vision center screening? >> DR. SHALINDER SABHERWAL: That’s an excellent question, I should have included that, because talking in my context where we have a very vertical sort of ophthalmology program, but a true sort of intervention will be how we can use the cameras in the noncommunicable diseases clinics or the hospital where the physicians should be able to use that, instead of waiting for these people to come to an eye care center and they don’t have a complaint. It has to be within a clinic, because going door to door doesn’t make sense right now, looking at the prevalence and the cost of the camera. But I would say that we have been using general health setups and that will be even more cost effective as compared to a vertical vision center or primary care center. >> DR. ANAND VINEKAR: Just to follow up on that, to make it successful it will have to work with the public health delivery system. What are your thoughts on how to integrate it with that, because that’s a separate challenge itself. >> DR. SHALINDER SABHERWAL: Absolutely. I think it makes it pretty easy now, we were struggling when there was no AI-integrated into that, because even if you put a camera in a nonclinical disease clinic where there is no ophthalmologist in the picture, you can lose the patient. The AI report would guide you with the referral condition, and that’s all we require at a screening level. It is very useful to have the AI where the ophthalmologist cannot be directed. With the India scenario, the primary health care centers, I think it is a great sort of usage, potential for that. >> DR. ANAND VINEKAR: And I think we answered the next question, what is the future of AI in India. ‘ Not just about finding the true positive but to actually integrate it for treatment. AI is great, like you showed, in the peripheral rural centers, we can pick up patients who need treatment. But what I meant in my first question was helping to create it in the public delivery system on getting pa patient into the closest center where this treatment is available. It could be the community health center. It could be district headquarters. Of course it could be attached to a center in the closest city. And of course this is a question more to your public health knowledge than the AI itself, but I thought it would complete the ecosystem. What do you think about that? >> DR. SHALINDER SABHERWAL: Again, thanks for the question. There is no easy answers there. There are two or three things I can think of straight away. One, how do you train the technician to use the images? Because now you have something in your hand to show the patient. Earlier, we didn’t have that. So how do you use that for change? That is something which the technology platforms can also work on, because they’re already coming out with these reports. Second, with technology being quite standard in most of the networks, how do you connect them directly with the clinician on the basis of positive reports and can clinicians counsel them better? It may or may not require a followup, some of the cases will have a very clear picture. We know it doesn’t require anything to be done right now, it may not even need that referral to the hospital. I think that is something that can be indicating connecting to care. >> DR. ANAND VINEKAR: I think you brought up a very important point of using the mobile phone. Geotagging them, making sure they are connected to the closest health care worker, and of course integrating it into the public delivery system. Okay. Let’s move to the next question. I presume this is a generic question for both of us, or maybe it’s me and I don’t have the answer so maybe you can help me. What are the current models for the use of the eye in detection of astigmatism early in children that can give accurate measurements. I’m not aware of any such model. Are you? >> DR. SHALINDER SABHERWAL: Not aware. >> DR. ANAND VINEKAR: So I’m not sure how that might work. In terms of just image, so this will be an example of what I said where we’re using a metadata of the patient’s systemic conditions. Similarly if it’s not just an image, obviously astigmatism is a numerical value as much as it probably is a value we’re getting out of the cornea. Both of these will have to be integrated into an AI model. It’s not just an image. I think we’re looking at models beyond imaging. I don’t think we’ve come of age, we’re not sure of anything that’s worked in the outreach. The next question is, what level of training do technicians have or need, are they usually from the health care background or is there no specific trend? Also which fundus cameras are being used and what is the greatest obstacle of increasing compliance? I know there are three questions, but training of technicians, which fundus cameras, and the greatest obstacle to increasing compliance. So I think I’ve already discussed, compliance is always an issue in any public intervention. There are certain positives in this in terms of the polite to put those images and directly talking to the clinician. Coming to the other sort of question which was about the need for the technician to be qualified, frankly we have tried with both. At division center we have the luxury of having a litigation technician. So we have used trained technicians for two years and all that. In the previous program we actually trained some community health workers who had no background in ophthalmology or optometry and it’s about taking a good image and how to train them to take a good image, and that’s all they have to do with the AI already coming in. That is not a very big obstacle in terms of requiring a technical person to take these images. And we just use many cameras, with the AI-integrated we just use the Medios fundus on phone. >> DR. ANAND VINEKAR: I would like to add about the same questions with relevance to ROP. The ROP model, we have a training model to train what we call technicians, but we know we can train optometrists, nurses, you can train them from anywhere between two to six weeks and accredit them. As far as cameras go, in the ROP we are unfortunately limited with the RetCam family and the Neo family and others that are integrated with the mobile phone. As more and more cameras come up, and we are hopeful in the next one or two years there are at least three or four cameras that I’ve seen in the version that it’s going to come out, we hope that will also, our AI algorithms will be camera agnostic. And as far as the increasing compliance, he mentioned it’s a problem, for us it’s an even bigger problem in ROP, but as he very rightly said, images can be used for patient education. And when you show the mother or the father or the family the image with the ROP and you show them that there is avascularity and there’s a ridge, then they’re convinced, and they become part of the decision process, and when you call them back, they will come. What are the sensitivity and specificity measures with regard to ROP? In the journey we had, our initial sensitivity was in the late ’80s and early ’90s and in our most recent — the AI Medios, it is showing well, 100% concurrence with the ROP specialist. At least in the late ’90s is where we’ve reached insensitivity. For ROP we’re really putting in our money for the sensitivity, because we don’t want to miss any true positives because a baby that needs treatment must be picked up. Okay. Next question. Considering AI models are trained on a specific population, an algorithm developed in India would need to be adapted for use in North America. That’s a great question. Some of it has been answered by the work that I projected with Pete Campbell, it’s an American device algorithm that was tested in Nepal, India, and Mongolia, and they showed that across ethnicities, something developed for the Caucasian population is working well in other ethnicities as well. And I think the reverse will be true. The ones that we’ve developed in India will need to be validated and tested in western ethnicities as well. A quick answer is yes, it will work, and already some work has gone proving that. Shalinder, this one is for you. Was human-based DR and glaucoma screening at vision centers already before you introduced AI, or did the availability of AI to facilitate introduction of the screening services for the first time? >> DR. SHALINDER SABHERWAL: An excellent question. And we were not doing this earlier. Only because we had done a small sort of pilot where patients with everything in place, still we are struggling with compliance and we are trying to see how we can improve that beyond 30%. Earlier what used to happen is we were depend on somebody reporting on those images. And we were not leaving that to the technician. And to get a clinics on sort of — suddenly, even to see those images and report, was almost an impossible task in a very operational hospital. And so we never rolled that out as a program, because it was almost impossible to contact the patient later. And we found that the pilot was not working to contact them later. And so AI has definitely enabled or encouraged us to do that now and hopefully scale it up in the future. >> DR. ANAND VINEKAR: We have about four minutes before we end, so we’ll take the next few questions quickly. The one question is actually asked to you and me, so I’ll go first. Can you please share the training, validation, and testing data set numbers, are these models approved by regulatory bodies and have these models been compared with approved models internationally? The training and validation goes from 35,000 to 100,000 images. Are these models approved by regulatory bodies, not yet for ROP, but our last work is being done with government of India, we hope that itself will be some sort of validation. Have these models been compared with approved models internationally, not yet. There has been a western model in various ethnicities. Shalinder, now to you to answer those questions with respect to DR. >> DR. SHALINDER SABHERWAL: For DR, it has received approval, I’m not sure about glaucoma. These studies that have been done are published and they showed good results. Regulation, I think I’ll have to check whether they have that for glaucoma right now. The DR, definitely, referral for DR, they have received regulatory approval. >> DR. ANAND VINEKAR: Can AI detect ridge formation, the answer is yes. All our algorithms have done exactly that. Is the data set being used to train each AI based on the local population only, how many images have been needed to train the AI to this level? You can go first. >> DR. SHALINDER SABHERWAL: Sorry, I don’t have the specific numbers. We can get back to this question, asking — >> DR. ANAND VINEKAR: Our publication will carry the numbers. Obviously the more we put in for the training, the better the validation set is. I just shared my answer in the previous question so I won’t repeat the numbers again. There is a question not related to AI but to vitreous hemorrhage, maybe we’ll take that offline, I know you’re able to share this, for the lack of time. We have probably a minute left. Is AI applicable, feasible for low and middle income countries? Yes, definitely. And how he showed it in the rural outreach, for ROP, that’s exactly where we think it’s got bang for the buck, imagine the number of babies that need to be screened and the lack of expertise, that’s where AI can fit in. Are these applications used outside India, how do you ensure the data set used in training the AI is not skewed based on rating and ethnicity. I think he’s answered that in terms of the algorithms being used across the world. In India, I also told you, there are systems that have been used on different ethnicities, but yes, there would be a cross-ethnicity validation required as new softwares are developed. Do new softwares require an active Internet? Short answer, no. In your case, Shalinder? >> DR. SHALINDER SABHERWAL: No, they don’t, if you’re not sending the images anywhere, that’s fine. >> DR. ANAND VINEKAR: I think we’ve come to the end of our hour. All other pending questions I think Cybersight team will send and you say we hope we will be able to answer that offline. Before we end, thank you so much for being with us. I can see there was a very large number of participants online, and I know this is now shared onto the Cybersight website, and all of you can view it later, and we are open to receiving questions from you even after this webinar ends. Thank you, Cybersight team, thank you, Shalinder, and on behalf of the team here, thank you very much, have a great day, from whichever part of the world you are viewing us. Thank you once again. >> DR. SHALINDER SABHERWAL: Thank you.