- cross-posted to:
- autism@lemmy.world
- cross-posted to:
- autism@lemmy.world
AI-screened eye pics diagnose childhood autism with 100% accuracy::undefined
You must log in or register to comment.
100% ? That’s a fucking lie. Nothing in life is 100%
Are you 100% sure of that?
A convolutional neural network, a deep learning algorithm, was trained using 85% of the retinal images and symptom severity test scores to construct models to screen for ASD and ASD symptom severity. The remaining 15% of images were retained for testing.
It correctly identified 100% of the testing images. So it’s accurate.
100% accuracy is troublesome. Literally statistics 101 stuff, they tell you in no uncertain terms, never, never trust 100% accuracy.
You can be certain to some value of p. That number is never 0. .001 is suspicious as fuck, but doable. .05 is great if you have a decent sample size.
They had fewer than 1000 participants.
I just don’t trust it. Neither should they. Neither should you. Not at least until someone else recreates the experiments and also finds this AI to be 100% accurate.
What they’re saying, as far as I can tell, is that after training the model on 85% of the dataset, the model predicted whether a participant had an ASD diagnosis (as a binary choice) 100% correctly for the remaining 15%. I don’t think this is unheard of, but I’ll agree that a replication would be nice to eliminate systemic errors. If the images from the ASD and TD sets were taken with different cameras, for instance, that could introduce an invisible difference in the datasets that an AI could converge on. I would expect them to control for stuff like that, though.
I would expect them to control for stuff like that, though.
What was the problem with that male vs female deep-learning test a few years ago?
That all the males were earlier in the day, so the sun angle in the background was a certain direction, while all the females were later in the day, so the sun was in a different angle? And so it turned out that the deep-learning AI was just trained on the window in the background?
100% accuracy almost certainly means this kind of effect happened. No one gets perfect, all good tests should be at least a “little bit” shoddy.
Definitely possible, but we’ll have to wait for some sort of replication (or lack of) to see, I guess.
Yeah, exactly. They’re reporting findings. Saying that it worked in 100% of the cases they tested is not making a claim that it will work in 100% of all cases ever. But if they had 30 images and it classified all 30 images correctly, then that’s 100%.
The article headline is what’s misleading. First, it’s poorly written - “AI-screened eye PICS DIAGNOSE childhood autism.” The pics do not diagnose the autism, so the subject of the verb is wrong. But even if it were rephrased, stating that the AI system diagnoses autism itself is a stretch. The AI system correctly identified individuals previously diagnosed with autism based on eye pictures.
This is an interesting but limited finding that suggests AI systems may be capable of serving as one diagnostic tool for autism, based on one experiment in which they performed well. Anything more than that is overstating the findings of the study.
You need to report two numbers for a classifier, though. I can create a classifier that catches all cases of autism just by saying that everybody has autism. You also need a false positive rate.
True, but as far as I can tell the AUROC measure they refer to incorporates both.
Yup, you’re right, good catch 🙂
They talk about collecting the images - the two populations of images were collected separately. It’s probably not 100% of the difference, but it might have been enough to push it up to 100%
Then somebody’s lying with creative application of 100% accuracy rates.
The confidence interval of the sequence you describe is not 100%
From TFA:
For ASD screening on the test set of images, the AI could pick out the children with an ASD diagnosis with a mean area under the receiver operating characteristic (AUROC) curve of 1.00. AUROC ranges in value from 0 to 1. A model whose predictions are 100% wrong has an AUROC of 0.0; one whose predictions are 100% correct has an AUROC of 1.0, indicating that the AI’s predictions in the current study were 100% correct. There was no notable decrease in the mean AUROC, even when 95% of the least important areas of the image – those not including the optic disc – were removed.
They at least define how they get the 100% value, but I’m not an AIologist so I can’t tell if it is reasonable.
Yeah, from the way they wrote, it sounds to me they indirectly trained on the test set
Not even your statement?
Could we reasonably expect an AI to something right 100% if a human could do it with 100%?
Could you tell if someone has down syndrome pretty obviously?
Maybe some kind of feature exists that we aren’t aware of
Except death and taxes
taxes
- Only if you’re poor
Hold the fuck up. What exactly is the marker?
A big problem with this type of ai is they are a black box.
We don’t know what they are identifying. We give it input and it gives output. What exactly is going on internally is a mystery.
We don’t know what they are identifying. We give it input and it gives output. What exactly is going on internally is a mystery.
Counterintuitively that’s also where the benefit comes from.
The reason most AI is powerful isn’t because its can think like humans, its because it doesn’t. It makes associations that humans don’t simply by consumption of massive amounts of data. We humans tell it “Here’s a bajillion sample examples of X. Okay, got it? Good. Now here’s 10 bajillion samples we don’t know if they are X or not. What do you, AI, think?”
AI isn’t really a causation machine, but instead a correlation machine. The AI output effectively says “This thing you gave me later has some similarities to the thing you gave me before. I don’t know if the similarities mean anything, but they ARE similarities”.
Its up to us humans to evaluate the answer AI gave us, and determine if the similarities it found are useful or just coincidental.
Sure, but if we could take the model generated by the AI and convert it into a set of quantifiable criteria - i.e., what is being correlated - we could use our human abilities of associative thought to gain an understanding of why this correlation may exist, possibly leading to better understanding of Autism overall.
Incidentally to train AI, you need a bajillion samples of X and a bajillion-plus samples of not-X.
Sample set of 2
This is great. Article explains the method and sample size. This could be a great tool, and I hope it can be applied to any age. Many people who are on the spectrum and are high functioning can go most of their lives without a diagnosis while struggling to understand why the world feels so different to them.
This is particularly useful, since it would be easy to mass deploy. A quick photo, during a childhood checkup, and it can be easily checked. It doesn’t need to be focused, so could catch a lot more, less obvious cases.
As an autistic myself, an early diagnosis would have potentially helped a lot. This would still be true of those who mask well.
Sensitivity or specificity? Sensitivity is easy, just say every person is positive and you’ll find 100% of true positives. Specificity is the hard problem.
deleted by creator
Bull.Shit.
Define the criteria, have it peer reviewed and diagnosed, or else we will ALL be diagnosed with Autism soon enough.
