Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado This isn't AI, but my oncologist moved offices about ten years ago. I happened to see my records that had been transferred electronically. To my astonishment, the records stated that I had been using illegal drugs in the past. Right away, I got to my doctor and asked him how this untrue statement had got into my health records. He apologized and said sometimes people's records got "mixed up" when they were moved around electronically. This could have been quite a problem for me had I not caught it, because I was a professional pilot and had to get a letter from this doctor about my overall health every time I applied for a new medical certificate that allowed me to fly.
-
The implication then, is if you are using AI reports or notes, you MUST proofread them line by line before submitting them to the official record. You are, after all, the one signing the note. You are the one who is staking your license/certification on the information being accurate. If the AI lied, and you sign their note, you are wrong. It’s like if I signed a medical student note without any corrections or hint that it was a medical student note, except that most students don’t hallucinate.
@mcnado
Which begs the question: if you have to go through the output, line by line scanning for errors, is there really an efficiency gain to be had?
People being people, there will be corners cut and there will be deaths/false convictions as a result. I just hope that accountability will fall on the people using these tools instead of on the tools themselves.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado Yes, isn't it great? They can put whatever lies they want in the report, and if they get caught then it's the AI's fault. Bring on the weaponized vehicles and agonizer gloves!
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
I think it important to understand why.
AI reply engines are programmed to encourage user interaction. That means after an analysis of your input they will construct a reply that makes you feel good using the kind of language you use. Watch for the qualifiers: "excellent", Well done", etc.
AI "makes up everything". Each instance is independent of previous ones outside your present "interaction". The engine parses your request to set up reply parameters then its database for answers and assigns each part a probability and a reply that is random phonemes is just as likely as a direct quote from a Supreme Court ruling.
Ask for references and check them out.
Ask for challenging replies that make you think. Again, they all need to be referenced. -
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
-
The implication then, is if you are using AI reports or notes, you MUST proofread them line by line before submitting them to the official record. You are, after all, the one signing the note. You are the one who is staking your license/certification on the information being accurate. If the AI lied, and you sign their note, you are wrong. It’s like if I signed a medical student note without any corrections or hint that it was a medical student note, except that most students don’t hallucinate.
-
The implication then, is if you are using AI reports or notes, you MUST proofread them line by line before submitting them to the official record. You are, after all, the one signing the note. You are the one who is staking your license/certification on the information being accurate. If the AI lied, and you sign their note, you are wrong. It’s like if I signed a medical student note without any corrections or hint that it was a medical student note, except that most students don’t hallucinate.
@mcnado Exactly. Last year I was on a large multiagency call regarding new Veteran Services grants. At the end someone sent out their AI notes and I compared it with my own notes. I had to email the group to let them know that not only had the assistant botched the discussion of complex claiming, anyone following them would probably be committing a federal crime.
-
@mcnado
Which begs the question: if you have to go through the output, line by line scanning for errors, is there really an efficiency gain to be had?People being people, there will be corners cut and there will be deaths/false convictions as a result. I just hope that accountability will fall on the people using these tools instead of on the tools themselves.
-
@mcnado *ahem*
While the lay term may be hallucinate, the precise term is confabulate
@autolycos @mcnado Thank you. To hallucinate, you must first have some perception of reality.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado LLM’s have a success rate of 80%. That’s the same as the “pull out” method of birth control. Intelligent people don’t rely on these kinds of things for important decisions.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado
Research papers written by AI contain references (because AI knows it has to include them), but it doesn't know they have to be real references... study analysing 2.5 million papers identified approximately 146,900 hallucinated citations in 2025 alone -
The implication then, is if you are using AI reports or notes, you MUST proofread them line by line before submitting them to the official record. You are, after all, the one signing the note. You are the one who is staking your license/certification on the information being accurate. If the AI lied, and you sign their note, you are wrong. It’s like if I signed a medical student note without any corrections or hint that it was a medical student note, except that most students don’t hallucinate.
@mcnado Yep. I suspect that if you’re actually progressing carefully, this AI note process is not actually faster than writing the report yourself. The human doctor just provides a sort of liability crumple zone to absorb lawsuits when the AI makes errors.
-
The implication then, is if you are using AI reports or notes, you MUST proofread them line by line before submitting them to the official record. You are, after all, the one signing the note. You are the one who is staking your license/certification on the information being accurate. If the AI lied, and you sign their note, you are wrong. It’s like if I signed a medical student note without any corrections or hint that it was a medical student note, except that most students don’t hallucinate.
@mcnado I’m a radiologist so I’m writing reports all day. Actually generating the report is not the time consuming part of the process. What takes time is THINKING about the case.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado Probabilistic computing has no place in high-trust sectors.
-
@mcnado Yep. I suspect that if you’re actually progressing carefully, this AI note process is not actually faster than writing the report yourself. The human doctor just provides a sort of liability crumple zone to absorb lawsuits when the AI makes errors.
@garland this.
-
@rpmik I can’t watch it. Jumped off the couch and fled the room. Too damn real.
-
@mcnado
Which begs the question: if you have to go through the output, line by line scanning for errors, is there really an efficiency gain to be had?People being people, there will be corners cut and there will be deaths/false convictions as a result. I just hope that accountability will fall on the people using these tools instead of on the tools themselves.
@crispius to actually proofread a note for me would take much longer than writing it.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado For a while I saw ads on YouTube for AI note-taking apps where the narrator was talking about how they struggled to take notes in meetings and it was so much easier now.
My view of this was that said narrator needed to learn proper note-taking skills. (One of my duties on my current job is taking notes for meetings related to an esoteric subject in a regulated industry.)
-
@mcnado *ahem*
While the lay term may be hallucinate, the precise term is confabulate
Thanks for this important correction. As any psychiatrist will agree, a 'hallucination' has a quite specific technical meaning that carries diagnostic weight. The correct term for AI producing nonsense is 'confabulate' which is also a psychopathology with diagnostic implications.
-
@mcnado
Research papers written by AI contain references (because AI knows it has to include them), but it doesn't know they have to be real references... study analysing 2.5 million papers identified approximately 146,900 hallucinated citations in 2025 aloneLets be clear - the AI did not 'hallucinate'. It confabulated the invented referrences; no sensory processing mistake occurred. It doesn't (yet) have a sensory system.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login