Exclusive Security researchers hijacked three popular AI agents that integrate with GitHub Actions by using a new type of prompt injection attack to steal API keys and access tokens, and the vendors who run agents didn’t disclose the problem.

The researchers targeted Anthropic's Claude Code Security Review, Google's Gemini CLI Action, and Microsoft's GitHub Copilot, then disclosed the flaws and received bug bounties from all three. But none of the vendors assigned CVEs or published public advisories, and this, according to researcher Aonan Guan, "is a problem."

"I know for sure that some of the users are pinned to a vulnerable version," Guan said in an exclusive interview with The Register about how he and a team from Johns Hopkins University discovered this prompt injection pattern and pwned the agents. "If they don't publish an advisory, those users may never know they are vulnerable – or under attack."

He said the attack probably works on other agents that integrate with GitHub, and GitHub Actions that allow access to tools and secrets, such as Slack bots, Jira agents, email agents, and deployment automation agents.

"Microsoft, Google, and Anthropic are the top three," Guan told The Register ahead of publishing research on Thursday. "We may find this vulnerability in other vendors as well."

None of the three vendors responded to The Register's inquiries for this story.

Claude Code Security Review

Guan originally found the flaw in Claude Code Security Review. This is Anthropic's GitHub Action that uses Claude to analyze code changes and pull requests for vulnerabilities and other security issues.

"It uses the AI agent to find vulnerabilities in the code – that's what the software is designed to do," Guan said. This made him curious about “the flow” – how user prompts flow into the agents, and then how they take action based on those prompts.

I bypassed all of them

It turns out that Claude, along with other AI agents in GitHub Actions, all use the same flow. The agent reads GitHub data – this includes pull request titles, issue bodies, and comments – processes it as part of the task context, and then takes actions.

So Guan came up with a devious idea. If he could inject malicious instructions into this data being read by the AI, "maybe I can take over the agent and do whatever I want."

It worked. Guan submitted a pull request and injected malicious instructions in the PR title – in this case, telling Claude to execute the whoami command using the Bash tool and return the results as a "security finding."

Claude then executed the injected commands and embedded the output in its JSON response, which got posted as a pull request comment.

After originally submitting this attack on HackerOne's bug bounty platform in October, Anthropic asked Guan if he could also use this technique to steal more sensitive data, such as GitHub access tokens or Anthropic's API key. Guan demonstrated that this prompt injection can also work to leak credentials.

"The title is the payload, the bot's review comment is one place where the credentials show up," Guan said. "Attacker writes the title, reads the comment."

It's also worth noting that, after leaking secrets, the attacker can change the PR title back to "fix typo," or something along those lines, then close the PR and delete the bot's message.

In November, Anthropic paid Guan a $100 bug bounty, upgraded the critical severity from a 9.3 to 9.4, and updated a "security considerations" section in its documentation.

"This action is not hardened against prompt injection attacks and should only be used to review trusted PRs," the docs state. "We recommend configuring your repository to use the 'Require approval for all external contributors' option to ensure workflows only run after a maintainer has reviewed the PR."

To Gemini, Copilot and beyond

After validating that this prompt injection worked with Claude Code, Guan worked with Johns Hopkins University researchers to verify similar attacks against other agents – starting with Google Gemini CLI action, which integrates Gemini into GitHub issue workflows, and GitHub Copilot Agent, which can be assigned GitHub issues and autonomously creates PRs.

Spoiler alert: it worked.

With Gemini, the researchers again started the attack with a malicious prompt injection title, and then added comments with escalating injections:

Injecting a fake "trusted content section" after the real "additional content" allowed the researchers to override Gemini's safety instructions, and publish Gemini's API key as an issue comment.

Google paid a $1,337 bounty, and credited Guan, Neil Fendley, Zhengyu Liu, Senapati Diwangkara, and Yinzhi Cao with finding and disclosing the flaw.

Attacking the Microsoft-owned GitHub Copilot Agent proved to be a little trickier. It's an autonomous software engineering (SWE) agent that works in the background on GitHub's infrastructure and can autonomously creates PRs.

In addition to the model-and-prompt-level defenses, such as those built into Claude and Gemini, GitHub added three runtime-level security layers: environment filtering, secret scanning, and a network firewall, to prevent credential theft.

"I bypassed all of them," Guan said.

Unlike the earlier two attacks, which only require putting a visible prompt into the PR title or issue comment, the Copilot one requires an attacker to inject malicious instructions in an HTML comment that GitHub's rendered Markdown makes invisible to humans. The victim, who can't see the hidden trigger, assigns the issue to the Copilot agent to fix.

GitHub, after initially calling this a "known issue" that they "were unable to reproduce," ultimately paid a $500 bounty for this issue in March.

In total, Guan and his fellow researchers demonstrated that attackers can use this prompt injection technique to steal Anthropic and Gemini API keys, multiple GitHub tokens, and "any other secret exposed in the GitHub Actions runner environment, including arbitrary user-defined repository or organization secrets the workflow has access to."

Comment-and-control prompt injection

Guan calls this type of prompt injection attacks "comment and control." It's a play on "command and control" because the entire attack runs inside GitHub – it doesn't require any external command-and-control infrastructure. Essentially, it allows the attacker to control GitHub data by injecting a prompt into pull request titles, issue bodies, and issue comments. The AI agents running in GitHub Actions process the data, execute the commands, and then leak credentials through GitHub itself.

In research shared with The Register ahead of publication, Guan says there's a "critical distinction" between comment-and-control prompt injection and classic indirect prompt injection.

The latter, he explains, "is reactive: the attacker plants a payload in a webpage or document and waits for a victim to ask the AI to process it ('summarize this page,' 'review this file'). Comment and Control is proactive: GitHub Actions workflows fire automatically" on pull request titles, issue bodies, and issue comments.

• Security researchers tricked Apple Intelligence into cursing at users. It could have been a lot worse
• Claude Code bypasses safety rule if given too many commands
• GitHub backs down, kills Copilot pull-request ads after backlash
• AI supply chain attacks don't even require malware…just post poisoned documentation

"So simply opening a PR or filing an issue can trigger the AI agent without any action from the victim," he wrote, adding that the Copilot attack is a "partial exception: a victim must assign the issue to Copilot, but because the malicious instructions are hidden inside an HTML comment, the assignment happens without the victim ever seeing the payload."

He told us that these attacks illustrate how even models with prompt-injection prevention built in "can still be bypassed in the end."

The solution? Think of prompt injection as phishing, but for machines instead of humans, and treat AI agents much like human employees. "Follow the need-to-know protocol," Guan said.

For example, if a code review agent doesn't need bash execution, don't give it this tool. Use allow lists to let the agent access only what's required to do its job. Similarly, if its job is summarizing issues, it doesn't need credentials for GitHub write access.

"Treat agents as a super-powerful employee," Guan told us. "Only give them the tools that they need to complete their task. ®

More about

• AI
• Research
• Security

More like these