TL;DR: AI coding assistants and coding agents are now part of everyday software development, and the question for most teams is no longer whether to use them but how to use them safely. The risks are specific and manageable: code that looks right but is subtly wrong or insecure, dependencies that do not exist (a peer-reviewed USENIX Security 2025 study found hallucinated package names in at least 5.2 percent of packages suggested by commercial models and 21.7 percent by open-source models), confidential code or secrets sent to third-party services, licence and ownership questions, and agents with too much access. A short written policy, enforced mostly through tooling you already have (code review, CI checks, dependency controls, secret scanning, least-privilege access) covers most of it. Good governance makes the tools more useful, not less.
Why governance matters now
Assistants have moved from autocomplete to chat, and from chat to agents that read a repository, edit many files, run commands and open pull requests. Each step increases both productivity and the blast radius of a mistake. The good news is that software teams already have strong controls for code written by people: version control, review, testing, dependency management and least privilege. Governing AI-assisted development is mostly about applying those controls consistently to code a model helped write, and closing a few new gaps.
A useful anchor is NIST's Secure Software Development Framework (SSDF, SP 800-218, version 1.1). Its practices are grouped into preparing the organisation, protecting the software, producing well-secured software and responding to vulnerabilities, and none of them depend on who or what typed the code. NIST has also published SP 800-218A, an SSDF profile for developing generative AI models themselves, which is relevant if you train or fine-tune models.
The risks, specifically
1. Plausible but wrong code
Generated code usually compiles and often looks idiomatic, which is exactly why errors slip through. Typical problems are missing edge cases, outdated API usage, insecure defaults (string-built SQL, disabled certificate checks, weak cryptography), and tests that assert what the code does rather than what it should do. The OWASP Top 10 for LLM Applications lists improper output handling (LLM05) as a risk: model output should be treated as untrusted input to whatever consumes it, and code is no exception.
2. Hallucinated and malicious dependencies
Models sometimes suggest packages that do not exist. That is more than an annoyance: an attacker can register the invented name and publish malicious code under it. The USENIX Security 2025 paper We Have a Package for You! tested 16 code-generating models across 576,000 code samples in two languages, and found that the average percentage of hallucinated packages was at least 5.2 percent for commercial models and 21.7 percent for open-source models, with 205,474 unique hallucinated package names. OWASP lists supply chain (LLM03) as a top risk for the same broad reason.
3. Leaking code, data and secrets
Prompts, open files and repository context may be sent to an external service. Depending on the tool and plan, that can include proprietary code, customer data in fixtures or logs, and secrets in configuration files. OWASP's sensitive information disclosure (LLM02) covers this. What happens to that data, including retention and use for training, is set by the vendor's terms and your plan, so read them rather than assume.
4. Licensing and ownership
Two separate questions arise. First, could a suggestion reproduce existing public code under a licence you cannot comply with? Some tools help here: GitHub Copilot's code referencing checks suggestions against public code on GitHub and, depending on policy, either blocks matching suggestions or shows the match with licence information. Second, who owns AI-generated output? In the United States, the Copyright Office's January 2025 report on copyrightability concluded that, given current generally available technology, prompts alone do not provide sufficient human control to make users of an AI system the authors of the output; human creative contribution such as selection, arrangement and modification is what can be protected. Rules differ by country, so take legal advice for anything where ownership matters, such as code delivered to clients.
5. Agents with too much access
Coding agents that run commands, install packages, call APIs or push branches inherit the risks OWASP calls excessive agency (LLM06): too many tools, too many permissions, too much autonomy. Combined with prompt injection (instructions hidden in an issue, a README or a web page the agent reads), an agent can be steered into doing something its user never intended. For background on agent guardrails in general, see our guide to AI agents for business.
A practical policy, in nine rules
Keep the written policy short enough that people read it. The rules below are a starting point; adapt them to your risk profile.
- Approved tools only. Maintain a list of approved assistants and plans, chosen after reviewing their data handling terms. Prefer business plans with clear commitments on retention and training, and centrally managed settings.
- Humans own every merge. The developer who submits AI-assisted code is accountable for it exactly as if they had written it by hand. Every change goes through normal code review; nothing merges on an agent's say-so.
- No secrets or personal data in prompts. Keep credentials out of repositories in the first place, use secret scanning in pre-commit hooks and CI, and exclude sensitive files and directories from assistant context where the tool supports it.
- Verify every new dependency. Check that a suggested package exists, is the one you intended, is maintained and has an acceptable licence before adding it. Pin versions with lock files, and where possible install through an internal registry or proxy with an allow-list.
- Security checks run on everything. Static analysis, dependency vulnerability scanning and tests run in CI on all changes, not only on code someone flagged as AI-generated.
- Sandbox agents. Run coding agents in containers or disposable environments with no production credentials, limited network access and least-privilege tokens. Require human approval for commands that install software, change infrastructure or touch shared systems.
- Licence safeguards on. Enable public-code matching filters or code referencing where your tool provides them, and keep your normal open-source licence compliance process.
- Disclose where it matters. Agree whether and how AI assistance is recorded (for example, in commit trailers or pull request descriptions), especially for client deliverables where contracts address it.
- Review the policy regularly. Tools change fast. Revisit the approved list, settings and rules on a fixed schedule.
Policies that rely on people remembering fail under deadline pressure. Most of the rules above can be enforced automatically:
| Rule | How to enforce it |
| Humans own every merge | Branch protection, required reviews, no direct pushes to main |
| No secrets in code or prompts | Secret scanning in pre-commit and CI; context exclusions in assistant settings |
| Verified dependencies | Lock files, internal registry or proxy, dependency review on pull requests |
| Security checks | Static analysis and vulnerability scanning as required CI checks |
| Sandboxed agents | Ephemeral containers, scoped tokens, network egress rules, approval prompts for commands |
| Build integrity | Provenance and tamper protection for build artefacts, for example following SLSA |
SLSA describes itself as a checklist of standards and controls to prevent tampering and improve integrity in the software supply chain. It is a sensible next step once the basics are in place, whether code comes from people or models.
Getting value, not just avoiding risk
Governance is not only defensive. Teams that use assistants well tend to:
- Use them where checking is cheap: tests, boilerplate, migrations, documentation, refactors with good test coverage, and explaining unfamiliar code.
- Write the tests first or alongside, so generated code has an independent check.
- Keep changes small, so reviews stay meaningful. A thousand-line AI-generated pull request is as hard to review as a human one.
- Share prompts and project conventions in the repository (coding standards, architecture notes) so assistants and new team members follow the same rules.
- Measure outcomes that matter, such as lead time, defect rates and review effort, rather than lines generated.
CISA's Secure by Design principle, treating customer security as a core business requirement and building it in during design rather than after release, applies here unchanged. AI assistance can raise the baseline if secure patterns are the ones your conventions and examples encourage.
How ApexlyTech can help
We use AI-assisted development in our own delivery with the controls described here, and we help clients set up the same: approved tooling, CI checks, dependency and secret controls, and sandboxed agent workflows. See our custom software, SaaS development and AI and automation services, or contact us to talk it through.
Frequently asked questions
Is AI-generated code safe to use in production?
It can be, if it goes through the same review, testing and security checks as any other code. The risk comes from treating it as trusted because it looks correct. OWASP recommends treating model output as untrusted input.
What is package hallucination, sometimes called slopsquatting?
When a model suggests a dependency that does not exist, an attacker can register that name with malicious code. A USENIX Security 2025 study found hallucinated packages in at least 5.2 percent of commercial-model and 21.7 percent of open-source-model package suggestions. Verify every new dependency before installing it.
Who owns code written with an AI assistant?
It depends on jurisdiction and on how much human creative contribution is involved. The US Copyright Office concluded in January 2025 that prompts alone do not make a user the author of the output. Take legal advice where ownership matters, such as client contracts.
Should coding agents have access to production?
No. Run them in sandboxed environments with least-privilege, short-lived credentials and no production secrets, and require human approval for commands that install software or change infrastructure.
Do we need a separate security process for AI-assisted code?
Mostly not. Apply your existing secure development practices, such as those in the NIST SSDF, consistently to all code, and add a few specific controls: dependency verification, secret exclusion from prompts, approved tools and agent sandboxing.
Sources