Secure AI-Assisted Software Development: What We Need to Verify
· elbtech
Updated:
AI changes how software is built. Implementation, tests, and documentation can be approached differently from just a few years ago. That is a reason to evolve our methods, not to hand responsibility to a model. Our benchmark is verifiable behavior, not the volume of generated code. This article describes the engineering principles we work toward; it is not an audit report or a security certificate.
Built with AI Is Not the Same as AI Inside the Product
An application without a language model can still be developed with coding agents. An application with integrated AI needs additional checks of its model behavior and permitted actions. Both layers introduce risks: the development process and the delivered system. When discussing AI safety, we should identify which layer we mean and which possible harms matter in the actual use case.
Correct, Secure, and Traceable Are Different Questions
- Correctness: Does the behavior meet the agreed requirements and business rules?
- Security: Are data, permissions, and technical boundaries protected even under adversarial input?
- Traceability: Can a reviewer connect requirements, changes, checks, and approval to a release?
- Safety: What harm could an error cause, and which controls reduce that risk?
A Plausible Test Can Confirm the Wrong Behavior
Consider a simplified example: a user may read quotations only for their own company. If AI generates both an endpoint and tests that demonstrate only authorized access, every test might pass while tenant isolation is missing. Acceptance criteria must be established independently of the implementation proposal. A negative test using a different tenant matters just as much as the successful case. This illustrates a failure class, not a claimed finding in a client system.
The Development Environment Is Part of the System
Coding agents work with source code, tools, and potentially external content. Their data access, network connections, and permissions need deliberate boundaries. Secrets do not belong in prompts or outputs. Suggested dependencies must actually exist and be evaluated. Repository instructions can help, but they cannot replace technically enforced controls.
Evidence Instead of Another Promise
Useful verification evidence links a specific requirement to the diff, actually executed tests, relevant security checks, and an accountable approval. An agent saying it checked everything is not a check result. Even a passing suite proves neither complete security nor the correctness of an incomplete requirement. Unresolved risks and untested areas must remain explicit.
New Tools, Evolving Methods
We want to use AI's capabilities while making its limits visible. That means challenging assumptions, checking small changes, and turning failures into requirements and regression tests. Domain approval and engineering accountability do not disappear when implementation becomes faster. Our introduction to Rostfrei explores explicit domain models as one part of this work.