We'll see how I feel in the morning, but for now i seem to have convinced myself to actually read that fuckin anthropic paper

"AI" is not actually a technology, in the way people would commonly understand that term.

If you're feeling extremely generous, you could say that AI is a marketing term for a loose and shifting bundle of technologies that have specific useful applications.

I am not feeling so generous.

AI is a technocratic political project for the purpose of industrializing knowledge work. The details of how it works are a distant secondary concern to the effect it has, which is to enclose and capture all knowledge work and make it dependent on capital.

And now for a short break

I have eaten. I may be _slightly_ less cranky.

Ok! The results section! For the paper "How AI Impacts Skill Formation"

> we design a coding task and evaluation around a relatively new asynchronous Python library and conduct randomized experiments to understand the impact
of AI assistance on task completion time and skill development

...

Task completion time. Right. So, unless the difference is large enough that it could change whether or not people can learn things at all in a given practice or instructional period, I don't know why we're concerned with task completion time.

Well, I mean, I have a theory. It's because "AI makes you more productive" is the central justification behind the political project, and this is largely a political document.

> We find that using AI assistance to complete
tasks that involve this new library resulted in a reduction in the evaluation score by 17% or two grade
points (Cohen’s d = 0.738, p = 0.010). Meanwhile, we did not find a statistically significant acceleration in
completion time with AI assistance.

I mean, that's an enormous effect. I'm very interested in the methods section, now.

> Through an in-depth qualitative analysis where we watch the screen recordings of every participant in our
main study, we explain the lack of AI productivity improvement through the additional time some participants
invested in interacting with the AI assistant.

...

Is this about learning, or is it about productivity!? God.

> We attribute the gains in skill development of the control group to the process of encountering and subsequently resolving errors independently

Hm. Learning with instruction is generally more effective than learning through struggle. A surface level read would suggest that the stochastic chatbot actually has a counter-instructional effect. But again, we'll see what the methods actually are.

> We find that using AI assistance to complete
tasks that involve this new library resulted in a reduction in the evaluation score by 17% or two grade
points (Cohen’s d = 0.738, p = 0.010). Meanwhile, we did not find a statistically significant acceleration in
completion time with AI assistance.

I mean, that's an enormous effect. I'm very interested in the methods section, now.

> Through an in-depth qualitative analysis where we watch the screen recordings of every participant in our
main study, we explain the lack of AI productivity improvement through the additional time some participants
invested in interacting with the AI assistant.

...

Is this about learning, or is it about productivity!? God.

> We attribute the gains in skill development of the control group to the process of encountering and subsequently resolving errors independently

Hm. Learning with instruction is generally more effective than learning through struggle. A surface level read would suggest that the stochastic chatbot actually has a counter-instructional effect. But again, we'll see what the methods actually are.

They reference these figures a lot, so I'll make sure to include them here.

> Figure 1: Overview of results: (Left) We find a significant decrease in library-specific skills (conceptual
understanding, code reading, and debugging) among workers using AI assistance for completing tasks with a
new python library. (Right) We categorize AI usage patterns and found three high skill development patterns
where participants stay cognitively engaged when using AI assistance

@jenniferplusplus

> Learning with instruction is generally more effective than learning through struggle.

I don’t think this is necessarily a true statement? Guided learning beats unproductive struggle, but learning through struggle that eventually succeed produces far better retention etc than guided learning that becomes passive/receptive. There’s a huge literature on this that I’m not up on at all, but I’m pretty sure it doesn’t break cleanly along that particular line.

(I don’t think my quibble derails your larger train of thought here)

They reference these figures a lot, so I'll make sure to include them here.

> Figure 1: Overview of results: (Left) We find a significant decrease in library-specific skills (conceptual
understanding, code reading, and debugging) among workers using AI assistance for completing tasks with a
new python library. (Right) We categorize AI usage patterns and found three high skill development patterns
where participants stay cognitively engaged when using AI assistance

@inthehands @jenniferplusplus I would say that regardless whether guided learning from an entity that actually knows the material or independent learning tested against reality both best working with jumped-up autocorrect. The machine will tell you that you’re doing great things while spitting out garbage—counter-instructional is certainly one way to put it.

@jenniferplusplus

> Learning with instruction is generally more effective than learning through struggle.

I don’t think this is necessarily a true statement? Guided learning beats unproductive struggle, but learning through struggle that eventually succeed produces far better retention etc than guided learning that becomes passive/receptive. There’s a huge literature on this that I’m not up on at all, but I’m pretty sure it doesn’t break cleanly along that particular line.

(I don’t think my quibble derails your larger train of thought here)

@inthehands @jenniferplusplus One of my personal hesitance to use the LLM tools much (despite incredible professional pressure to do so) is that my use of it (again, under professional necessity) has re-enforced my pre-existing belief that struggling through a problem, debugging and digging through source and so on has been CRITICAL to my skill development. It is something I have for (uh) 15+ years told less experienced software developers is critical to getting better / faster!

@jenniferplusplus

> Learning with instruction is generally more effective than learning through struggle.

I don’t think this is necessarily a true statement? Guided learning beats unproductive struggle, but learning through struggle that eventually succeed produces far better retention etc than guided learning that becomes passive/receptive. There’s a huge literature on this that I’m not up on at all, but I’m pretty sure it doesn’t break cleanly along that particular line.

(I don’t think my quibble derails your larger train of thought here)

I have eaten. I may be _slightly_ less cranky.

Ok! The results section! For the paper "How AI Impacts Skill Formation"

> we design a coding task and evaluation around a relatively new asynchronous Python library and conduct randomized experiments to understand the impact
of AI assistance on task completion time and skill development

...

Task completion time. Right. So, unless the difference is large enough that it could change whether or not people can learn things at all in a given practice or instructional period, I don't know why we're concerned with task completion time.

Well, I mean, I have a theory. It's because "AI makes you more productive" is the central justification behind the political project, and this is largely a political document.

So, back to the paper.

"How AI Impacts Skill Formation"
https://arxiv.org/abs/2601.20245

The very first sentence of the abstract:

> AI assistance produces significant productivity gains across professional domains, particularly for novice workers.

1. The evidence for this is mixed, and the effect is small.
2. That's not even the purpose of this study. The design of the study doesn't support drawing conclusions in this area.

Of course, the authors will repeat this claim frequently. Which brings us back to MY priors, which is that this is largely a political document.

@inthehands Right, it's not universally the case. There are bad instructors and bad instructional contexts.

@jenniferplusplus It's less a claim and more an intentionally-unsubstantiated background premise which the supposed research will treat as an assumed truth.

> As AI development progresses, the problem of supervising more and more capable AI systems becomes more difficult if humans have weaker abilities to understand code [Bowman et al., 2022]. When complex software tasks require human-AI collaboration,
humans still need to understand the basic concepts of code development even if their software skills are
complementary to the strengths of AI [Wang et al., 2020].

Right, sure. Except, there is actually a third option. But it's one that seems inconceivable to the authors. That is to not use AI in this context. I'm not even necessarily arguing* that's better. But if this is supposed to be sincere scholarship, how is that not even under consideration?

*well, I am arguing that, in the context of AI as a political project. If you had similar programs that were developed and deployed in a way that empowers people, rather than disempowers them, this would be a very different conversation. Of course, I would also argue that very same political project is why it's inconceivable to the authors, soooo

"AI" is not actually a technology, in the way people would commonly understand that term.

If you're feeling extremely generous, you could say that AI is a marketing term for a loose and shifting bundle of technologies that have specific useful applications.

I am not feeling so generous.

AI is a technocratic political project for the purpose of industrializing knowledge work. The details of how it works are a distant secondary concern to the effect it has, which is to enclose and capture all knowledge work and make it dependent on capital.

I just

I'm not actually in the habit of reading academic research papers like this. Is it normal to begin these things by confidently asserting your priors as fact, unsupported by anything in the study?

I suppose I should do the same, because there's no way it's not going to inform my read on this

> We find that using AI assistance to complete
tasks that involve this new library resulted in a reduction in the evaluation score by 17% or two grade
points (Cohen’s d = 0.738, p = 0.010). Meanwhile, we did not find a statistically significant acceleration in
completion time with AI assistance.

I mean, that's an enormous effect. I'm very interested in the methods section, now.

> Through an in-depth qualitative analysis where we watch the screen recordings of every participant in our
main study, we explain the lack of AI productivity improvement through the additional time some participants
invested in interacting with the AI assistant.

...

Is this about learning, or is it about productivity!? God.

> We attribute the gains in skill development of the control group to the process of encountering and subsequently resolving errors independently

Hm. Learning with instruction is generally more effective than learning through struggle. A surface level read would suggest that the stochastic chatbot actually has a counter-instructional effect. But again, we'll see what the methods actually are.