@deirdrebeth

"Would you find that AI's choices to be equivalent to yours"

So far, there is a fairly high inter-rater reliability between how the AI codes texts and how I, or other researchers do. The variance is also very useful - mostly, I dismiss the things the AI codes and I didn't, but almost every time, the AI codes something I missed and I then code that.

Among fellow researchers and trainers, this has been a consistent experience. The AI helps do better research

/3

@deirdrebeth

" or would you be fuming at someone pretending to be a trainer and releasing a training manual that's full of easy to spot errors?"

I would fume at anyone who produces a shitty textbook or bad research
But this is not what is happening. We aren't standing back an just letting the AI rip, but rather using it to do some of the scutwork, and to crosscheck our work, and then editing and fixing bits it missed or got wrong.

Its like having an eager undergrad intern that needs oversight
@deirdrebeth

@mloxton

I find it very telling that here, even in what is apparently your best case scenario, the reasons you claim to use generative AI for qualitative analysis do not include "writing better code".

All you can ever do is say "yeah I know it makes mistakes. I don't care because it's cheap and fast."

@deirdrebeth

@Lana

From his response a few hours ago:

"In over a year of intensively testing and using AI Assist in MAXQDA, I have had zero occurrences of hallucination, and every coding or summarization or topic discovery it makes comes with a reference to document and line number. If it doesn't find the topic in the text, it says so"

and also:
"I never use any AI tools for the content itself."

@mloxton

@mloxton
@deirdrebeth

Okay but "and then editing and fixing bits it missed or got wrong" implies that it does, in fact, regularly hallucinate and make errors or omissions.

@mloxton

I find it very telling that here, even in what is apparently your best case scenario, the reasons you claim to use generative AI for qualitative analysis do not include "writing better code".

All you can ever do is say "yeah I know it makes mistakes. I don't care because it's cheap and fast."

@deirdrebeth

@Lana
Lana, you are mixing up two different topics.

I have in over two years of using it daily and during extensive testing, not seen a single case of AI Assist in MAXQDA hallucinate.

When used for coding, as I said, the IRR is high, but not perfect (which would be unexpected), but still useful because it occasionally spots something I didn't code and should have. That leads to analysis with higher validity - a better product

@deirdrebeth

@Lana

I think you may be under the impression that we are talking about "code" as in Java, Python, or R, and therefore "writing better code". This is not what I am talking about. We are talking about qualitative research and coding text segments.

The AI indeed helps increase coding construct validity, and therefore have better research results.
I used it in this way here, for example: https://www.medrxiv.org/content/10.1101/2025.03.04.25320693v1

I am happy to go deeper into how this works if you are interested
@deirdrebeth

@deirdrebeth @mloxton

>"I have never once seen it hallucinate"

>"I have to go back and fix things it missed or got wrong"

Both of these cannot be true simultaneously.

@deirdrebeth @mloxton

You seem to be missing the point we are all saying. And at this point, I'm certain it's on purpose because nobody could be this dense. Whether you're talking about computer code, braille, or your little orphan Annie decoder ring doesn't matter.

The point is, when you're using AI, you're using it for these things:
- speed
- cost

And not for this thing:
- the quality of the output

And we know that, because, even in your best case scenario, you say things like "it's like a fast, unpaid intern who needs oversight because they sometimes miss things or make mistakes."

@Lana

Oh Dear God
Hallucination is one kind of error, amongst many kinds of error.
It has never yet hallucinated, but is sometimes makes other errors.

When using AI Assist in MAXQDA, sometimes it misidentifies an implication, and I need to fix that, and sometimes its coding is too broad, and I need to fix that too. Sometimes the fix is just adjusting code range, sometimes it is rewo9rding and tightening up code definitions

@deirdrebeth

@Lana

Lana, I think you just want to be obnoxious because I have repeatedly stipulated that using AI does ALL THREE THINGS in research.

Maybe you didn't understand what the term "validity" implied, so let's restate this cleanly:

Using AI Assist:
- Reduces cost
- Saves time
- Improves quality

It also does an additional thing by expanding CAPACITY, and that is it allows me to tackle research that was otherwise impractical or impossible.

@deirdrebeth

@deirdrebeth @mloxton

I'm really not interested in having an argument over definitions.

The point is, you are using AI for these things:
- speed
- cost

And not for these things:
- the quality of the output

And we know that because you keep saying how, even in your best case scenario, the generative AI that you use needs constant oversight because it misses things, makes coding mistakes, or needs rewording. Whether you call that a hallucination error or some other kind of error is irrelevant to the broader point, and you know that. You cannot be this dense.

@deirdrebeth @mloxton

So it improves quality by making mistakes and needing constant oversight.

Sure buddy.