MALICIOUS "CONTEXT MANIPULATION" TECHNIQUE: AI Chatbots

Imagine a world where AI-powered bots can buy or sell cryptocurrency, make investments, and execute software-defined contracts at the blink of an eye, depending on minute-to-minute currency prices, breaking news, or other market-moving events. Then imagine an adversary causing the bot to redirect payments to an account they control by doing nothing more than entering a few sentences into the bot’s prompt.

AUTONOMOUS

New attack can steal cryptocurrency by planting false memories in AI chatbots

Malicious "context manipulation" technique causes bot to send payments to attacker's wallet.

Dan Goodin – May 13, 2025 6:00 AM

That’s the scenario depicted in recently released research that developed a working exploit against ElizaOS, a fledgling open source framework.

ElizaOS is a framework for creating agents that use large language models to perform various blockchain-based transactions on behalf of a user based on a set of predefined rules. It was introduced in October under the name Ai16z and was changed to its current name in January. The framework remains largely experimental, but champions of decentralized autonomous organizations (DAOs)—a model in which communities or companies are governed by decentralized computer programs running on blockchains—see it as a potential engine for jumpstarting the creation of agents that automatically navigate these so-called DAOs on behalf of end users.

ElizaOS can connect to social media sites or private platforms and await instructions from either the person it’s programmed to represent or buyers, sellers, or traders who want to transact with the end user. Under this model, an ElizaOS-based agent could make or accept payments and perform other actions based on a set of predefined rules.

Potentially catastrophic outcomes

Recent research demonstrates that such attacks could cause potentially catastrophic outcomes if such agents are given control over cryptocurrency wallets, self-governing contracts known as smart contracts, or other finance-related instruments. The underlying weaknesses—based on a class of large language model attacks known as prompt injections—could be exploited by people interacting with an agent to store false memory events that never, in fact, happened. . .

[. . .] The ability for adversaries to store histories of events that never actually occurred directly into an LLM’s memory database was demonstrated last year. The proof-of-concept attack abused long-term conversation memory built into ChatGPT, which stores information from all previous interactions and uses it as context for future conversations. Researcher Johann Rehberger showed how an untrusted user could plant false memories that caused the chatbot to send all user input to an attacker-controlled channel. OpenAI engineers have since issued a partial fix. Rehberger demonstrated a similar attack against Gemini.

The attack against ElizaOS and the vulnerability it demonstrates should be balanced against the relative immaturity of the framework. As development continues and more and more components get added to the open source ecosystem, it’s possible that defenses will emerge that can be built in or added to the framework. The larger point is that LLM-based agents that can autonomously act on behalf of users are riddled with potential risks that should be thoroughly investigated before putting them into production environments.

AUTHOR Dan Goodin Senior Security Editor

Dan Goodin is Senior Security Editor at Ars Technica, where he oversees coverage of malware, computer espionage, botnets, hardware hacking, encryption, and passwords. In his spare time, he enjoys gardening, cooking, and following the independent music scene. Dan is based in San Francisco. 

Follow him at here on Mastodon and here on Bluesky. Contact him on Signal at DanArs.82.