Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
@mcnado For a while I saw ads on YouTube for AI note-taking apps where the narrator was talking about how they struggled to take notes in meetings and it was so much easier now.
My view of this was that said narrator needed to learn proper note-taking skills. (One of my duties on my current job is taking notes for meetings related to an esoteric subject in a regulated industry.)
-
@mcnado *ahem*
While the lay term may be hallucinate, the precise term is confabulate
Thanks for this important correction. As any psychiatrist will agree, a 'hallucination' has a quite specific technical meaning that carries diagnostic weight. The correct term for AI producing nonsense is 'confabulate' which is also a psychopathology with diagnostic implications.
-
@mcnado
Research papers written by AI contain references (because AI knows it has to include them), but it doesn't know they have to be real references... study analysing 2.5 million papers identified approximately 146,900 hallucinated citations in 2025 aloneLets be clear - the AI did not 'hallucinate'. It confabulated the invented referrences; no sensory processing mistake occurred. It doesn't (yet) have a sensory system.
-
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
-
-
@drdr6262 @AH_99 @mcnado
#GenAISlop machines don’t do any of those things, they merely calculate a parametized statistical probability output, based on a vast uncurated corpus, that one character follows another in combination with white spaces in a sequence that is likely to be understodd by a human requesting it. There’s no confabulation, no hallusinations, just a big con job. -
@mcnado For a while I saw ads on YouTube for AI note-taking apps where the narrator was talking about how they struggled to take notes in meetings and it was so much easier now.
My view of this was that said narrator needed to learn proper note-taking skills. (One of my duties on my current job is taking notes for meetings related to an esoteric subject in a regulated industry.)
@MisuseCase @mcnado 100%
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
The actual problem isn't hallucination, it's that the output looks authoritative whether or not it is. A confused human writes a confused sentence. The model writes confident prose either way, and that's what gets the report signed.
-
Whether you are in #lawenforcement or #medicine it is imperative that you understand this simple concept: AI engines that write reports will put information in the report that the AI engine thinks is what you want to see in the report.
Doesn’t matter if that information is true. Doesn’t matter if the referenced facts or events actually occurred, were discussed, or are present in the record. The engine will “hallucinate” — make it up entirely — whatever the engine decides fulfills the prompt.
It's going on averages and data weights, not at all "thinking" or knowing what's important.
Hello! It looks like you're interested in this conversation, but you don't have an account yet.
Getting fed up of having to scroll through the same posts each visit? When you register for an account, you'll always come back to exactly where you were before, and choose to be notified of new replies (either via email, or push notification). You'll also be able to save bookmarks and upvote posts to show your appreciation to other community members.
With your input, this post could be even better 💗
Register Login