My Personal LLM Rant
16 November 2025
I wrote this in 2024 while I was still at 18F, a federal digital services agency tasked with helping other agencies provide better digital service experiences to the American public. It was provoked by a nearly ever-present pressure to embrace and employ generative AI which, frankly, has not yet relented.
Though 18F was abolished, I still stand by everything in this post.
This has been floating around in my brain for too long and it’s time for it to exit. I want us to get clear on what we believe about AI, because I am really, really tired of being told to be more positive about AI and LLMs. I refuse. I will stay skeptical. Yes, it’s here, and yes federal leadership is interested. But of course they are. It’s in the news constantly. The big tech companies are spending bookoodles of dollars on it. The White House has had summits about it. So yeah, of course they’re interested. They’re being sensible: they see a change happening in the world, and they want to adapt accordingly. They’re also hearing promises about how this particular technology is going to unleash heretofore unimagined efficiencies that synergize our business units and amplify our blahblahblah. Unlike all those other times this has been promised, this time will be different! (It won’t.)
But listen, I did not come to 18F to just give other agencies what they want. That’s not valuable. There are plenty of vendors who will give them what they want. USDS, PIF, and 18F were created to give them what they need. We exist to be knowledgeable about digital service delivery and to use our knowledge to direct agencies towards good outcomes, not just the popular shit they see on the news (or advertised on the Metro). We don’t have a profit motive (do we?) so we can say the things they don’t want, but need, to hear.
It’s not that people in other agencies are dumb or naive or whatever. It’s that we each have limited time and capacity to understand any given topic in depth. Most agency leaders are too busy going into depth about their agency’s mission and how to advocate for it to Congress. They don’t have time to also go deep on digital service delivery. THAT IS WHY WE EXIST. We exist to be experts on a subject that others in government simply don’t have time to do.
On the particular subject of AI and LLMs, we here know what they are, how they work, and how they often don’t work. We know their very vast failure modes, we understand some of the consequences of those failures, and we’re tuned into the ongoing research and discovery of brand new harms. Our agency partners? Not so much. It’s not their field. But they keep hearing from their existing vendors about how this wonderful new technology will help them do more with less! And the general press covers technology about as well as it covers medicine, which is to say poorly.
I have this view, and I take it very seriously: it is my job to tell them when they don’t need something. It’s not my job to say blanketly that LLMs have no place in a particular agency’s operations, but it IS my job to tell agencies that its utility is limited. It’s my job to show them the failure modes. It’s my job to point out that every dollar they save by using an LLM comes back to cost them three dollars in correcting the bullshit the LLM created. It’s my job to show them all the ways an LLM can actually harm the people they’re trying to help. And when an agency points at a particular problem or use case, it’s my job to help them walk through whether AI or an LLM makes sense there, and what kind of safeguards need to be put around it.
We published a whole blog post about how 18F can help agencies “leverage the power of AI.” But we don’t need to. There are so many vendors that’ll help with that. What we need to do is help agencies understand whether they even need AI. The blog post talks about a chatbot at the Department of Education and says it has interacted with over 2.6 million people, but has anyone investigated whether those 2.6 million people were served well by the bot? Is it actually helping people, or is it just a new version of the dreaded phone tree, where now in order to get help, you first have to convince the computer to shut up and give you a phone number? (Where you can probably then deal with another phone tree.)
I’m really tired of being expected to have a rosy outlook about this. I’m a technology skeptic. I love the stuff. I think it’s very cool. But I also think it’s dangerous if misused. In the case of large AI and LLMs, the danger is often subtle and insidious, making it all the more important that we raise the red flags. Never mind that LLMs are an environmental disaster. No wait, actually do mind that, because there are also executive orders about limiting our environmental impact.
It’s only a matter of time until someone uses GPT to write an RFP and they miss some important details when they’re reviewing it, and then we’re stuck with a shitty contract that costs the government a bunch of money for nothing in return.
It’s only a matter of time until an agency’s chat bot gives someone exactly the wrong advice and someone loses their benefits over it. Or gets hurt.
It’s only a matter of time until an agency puts AI in front of benefits screening and improperly denies someone the benefits they’re entitled to. Then people go hungry. They get evicted. They get sick. It can go wrong so badly.
I don’t want to be part of any of those things. I want to be part of the effort to prevent those things from happening. That’s what we’re here for.