It’s a topic that keeps coming up during our weekly group meetings, it has spilled over into Slack DMs with my colleagues, it came up over dinner with a visiting researcher some days ago, and it’s something that is being actively discussed across the various communities I follow online. We have had LLMs¹ for a few years now, but it feels like something has recently changed² and a flurry of conversation around that technology and its impact across the ecosystem of knowledge workers.
Rather than throw my own thoughts on the pile, I’ve decided to bring you a small sample of the great discussions I have been seeing (specifically, the links in boldface).
Mathematics
Daniel Litt is a professor of mathematics who recently gave a talk about the role of AI in his own work.³ After he is done presenting the subject matter results, he shares a very interesting piece of advice about working with AI:
"If it's at all hard to check correctness [of the output], just throw it away"
He says this at 18:44 in the video. If you want full context, the discussion starts at 15:35. But I think that quote stands very well on its own, and can be used as a guiding principle across pretty much any task where the AI-generated output is intended to be helpful.⁴
Software development
Coding is perhaps the area where LLMs have been hyped the most (with fair reason). However, the ease with which anyone can produce a piece of reasonable-looking code has meant that Open Source projects across the board are now being faced with a deluge of low-quality “contributions”.⁵
Notable responses here are the Zig language’s Strict No LLM / No AI Policy and The Carpentries’ recently published AI Contributions Policy. The policies differ, but I think there’s a common goal: minimize the time humans with valuable expertise have to spend dealing with AI slop, and instead direct that effort towards supporting people who actually care.
Astrophysics (and beyond)
If there is one link in this post that I must insist you consume in its entirety, it’s this essay by Minas Karamanis: The machines are fine. I'm worried about us. He’s an astrophysicist, but every concern he names is broadly applicable to anyone who fancies themselves a researcher. Please, grant yourself the time to read it.
Closing remarks
This was just a tiny fraction of all the rich, thoughtful exchanges I’ve happened upon over the past couple of weeks.⁶ Overall, I am positively surprised by what appears to be a shared consensus being independently reached across different communities of people that do knowledge work on the value and risks of LLMs.⁷
My outlook is cautiously positive. The prospect that this technology, which clearly has some utility, will stay under the control of closed, profit-prioritizing entities appears to be increasingly unlikely.⁸ More importantly, there are clearly a lot of people who care about things, and who are actively working to correct the flaws of the systems that have started to creak and crack under the impromptu wave of stress tests that LLMs have enabled against them. May we seize this opportunity to build something better.
———————————————————————
¹: These days ‘AI’ is being used as shorthand for “machine learning actually being used in production”, a class of which LLMs are currently the most prominent members, but far from the only thing that fits the bill. I am fairly sure that all the discussions that I share in this post involve LLMs in some shape or form, but since I can’t directly check the tools people are using, I refer to them as AI when the source does.
²: A side effect of being a researcher appears to be developing some form of horror vacui towards that upon which no explanation has yet been deposited. Know that the urge here has been vigorously resisted.
³: This is pure, research-level math, which I understood exactly 0%. I was still able relate to all of the points Prof. Litt made on working with LLMs!
⁴: One can certainly use these tools for malicious purposes, in which case the correctness of the output might not be a prime concern.
⁵: In the pre-LLM world, contributions to open source projects were a reasonable proxy for someone’s software development prowess. This had already put “number of contributions” firmly within Campbell’s law territory as a metric, but even a low-effort contribution required non-trivial effort. With LLMs and “agentic coding”, the bar was essentially lowered to having a keyboard and an internet connection.
⁶: I’m quite sure that I spent more time struggling to decide what to leave out than actually writing this post. Most of these exchanges are happening across social media threads, in comment sections, in forums, and chat platforms. Those cannot be shared as effectively as a self-contained blog post or video.
⁷: I don’t want to spoil for you the pleasure of engaging with the conversations I have shared above. But I assure you, that consensus is there.
⁸: Local models are already good enough to be helpful for certain tasks. See, for instance, these posts from Simon P. Couch and Vicki Boykis. One still needs a relatively beefy PC to have them work at a reasonable speed, but I expect that this, too, will improve in the not-so-distant future.