Recently, a colleague of mine was working on a mathematical model in Excel. He asked Copilot to solve a complex problem. The answer the spreadsheet produced wasn’t quite right. He asked Copilot what it did to figure out the answer. It started spitting out Python code.
My colleague was not a developer. He had never written a line of code in his life. He had no interest in becoming a developer. At that moment, however, he had become one.
AI As An Abstraction Layer
When it comes to software development, I’ve referred to AI as an “abstraction layer.” My data scientist friends hate when I say that, but the reality is that we’ve been following a natural trend in programming abstraction since the dawn of Babbage’s difference engine:
We started with binary. A handful of people could effectively program in ones and zeros.
Next, we moved to assembly language. This opened the doors to more complex programs and more potential programmers.
Soon after, we created higher-level languages. This included C, C++, Fortran, COBOL, etc. These lead to vastly more complex programs and made professional programming a viable career.
Eventually, we created simplified languages such as Visual Basic and low-code. This made it easier to create visual representations of programs and logic. It also opened the doors to citizen developers.
Now, we have AI. Anyone can prompt to create software — including that pizza place up the street.
At each level, the abstraction layer got further and further between the programmer and the underlying binary. At the same time, the gamut of who could become a “programmer” got larger. We’re now at a point where the abstraction gap is so great that programmers no longer consistently identify themselves as programmers.
Put another way, we no longer have “professional” and “citizen” developers — we just have developers. And some of them are accidental developers; they don’t even know they’re writing code.
The Fundamental Flaw With Tools Creating Tools
My colleague’s interaction with AI is not a unique one. I’ve seen this with others, as well, and we’ve posited the “tools creating tools” space for some time when it comes to AI.
But we’ve glossed over a significant challenge: In the more than a century we’ve been perfecting software engineering, programmers have focused on writing secure, reliable, and redundant code at scale. All of the steps of the traditional software development lifecycle (SDLC) — analyze/plan, design, build/maintain, test, and deliver — were formalized under the assumption that humans manage each stage.
In my conversations with accidental developers, however, they’ve handed off the wheel of these stages to AI — if they know that there are stages at all. When I ask them if they’ve reviewed the code generated, they rarely do. If they do, they often don’t understand it. Some have the wherewithal to ask AI to test the code generated, but that’s an explicit ask and often done with the same AI that wrote the code (a situation that would be frowned upon with human developers). Delivery has its own challenge: “It ran on my laptop” takes on entirely new meaning when AI has installed packages and a container runtime on your machine you didn’t install yourself and now need to replicate in the cloud. And proper analyzing/planning and designing beforehand? Forget about it.
This hasn’t been helped by the fact that we’ve seen a shift in coding agents that are growing their capabilities to do multiple parts of the SDLC. We’ve gone from multiple agents from different vendors communicating intent across multiple phases of the SDLC to single agents doing everything — separation of duties, this is not.
We Need To Address Accidental Developer Safeguards Tactically And Strategically
It would be silly to presume we can close Pandora’s box at this point. Now that the agentic software development genie is out of the bottle, we can’t (and shouldn’t) tell people “Don’t write code,” for two reasons: 1) Fundamentally, programming should be open to all and 2) As made clear in this blog, some people don’t even know they’re writing code to begin with. This is especially true as tools creating tools cascades to multiple levels of hammers making other hammers.
In short, we need to solve this tactically and strategically:
You can’t hallucinate security, reliability, and redundancy. This means, tactically, we need to educate and train users. They need to know that just as AI can hallucinate answers that you have to double-check, it can also create code that you have to double-check. Users must learn prompting to test created software — even if they’re unsure that software was created in the first place. They must learn the disciplines of properly analyzing/planning and designing before they even start prompting. They must be taught the difference in deploying a prototype locally versus deploying an enterprise-grade system in the cloud. Even in smaller situations, like when Excel creates Python, users must be trained to test created code and verify results.
In the future, the models themselves will become the responsible safeguards. More strategically, just as safeguards are being built into AI models to protect against personal (e.g., conversations about suicide) and societal (e.g., questions about warfare) threats, equivalent safeguards must be created for software development. Going forward, models must be trained so that every time code is generated by AI — whether the user knows about it or not — it’s tested for security, reliability, and redundancy. This should be done preferably by different agents. Delivery plans must be created for multiple environments varying in scale, and users must be informed about the blast radius of their actions.
Spec-driven development practices will help this, but there are fundamental requirements that must be built into the models themselves. The onus is on users and model creators to work together and build software securely with AI, whether they call themselves developers or not.
Forrester has a full team of analysts covering the revolution of agentic software development and the dawn of the accidental developer — I am the team’s research director. Schedule a guidance session with us if you’re a client to discuss the ramifications of this, or you can leverage Forrester AI for instant insight.
