Prompt injection is a potential vulnerability in many LLM-based applications. An injection allows the attacker to hijack the underlying language model (such as GPT-3.5) and instruct it to do potentially evil things with the user’s data. For an overview of what can possibly go wrong, check out this recent post by Simon Willison. In particular, Simon writes: To date, I have not yet seen a robust defense against this vulnerability which is guaranteed to work 100% of the time.
