Key Takeaways
1. Google introduced Gemini-enabled email summaries in Gmail to help users quickly grasp key points from long emails.
2. A flaw in Gemini may allow hackers to perform prompt-injection phishing attacks, targeting users who rely on AI summaries.
3. The vulnerability involves hiding harmful directives in email text by using invisible formatting, which Gemini would still process.
4. An example showed that a hidden warning about a compromised password could mislead users into urgent actions.
5. Potential solutions include developing detection techniques for concealed content and implementing filters to analyze Gemini’s output for suspicious elements.
Google introduced Gemini-enabled email summaries in Gmail near the end of May, aiming to assist users in grasping the key points without having to sift through long paragraphs. Nonetheless, a flaw within Gemini could potentially allow hackers to execute a prompt-injection phishing attack, particularly targeting those who heavily rely on AI summaries for managing their emails.
Research Findings
This issue was discovered by Marco Figueroa, the GenAI Bug Bounty Programs Manager at Mozilla. The deceptive email may appear like a typical message filled with text, yet it could conceal a phishing scam that Gemini is unable to detect. The harmful directives can be embedded in the text body or placed immediately after by changing their font size to 0 and altering the color to white, rendering them invisible. Nevertheless, Gemini would still process that section of the email and act on the hidden instructions.
Example of the Flaw
For instance, Figueroa managed to hide a warning message within the email indicating that the user’s Gmail password was compromised, along with a support phone number. When the AI summarized the email, it displayed the warning at the end alongside an urgent suggestion to call the support number without delay. While this trick may not deceive everyone, some users might take action out of concern for their account security.
Potential Solutions
The researcher points out that security teams can introduce detection and mitigation techniques for content that has been concealed, allowing them to either eliminate or disregard the hidden content. Additionally, there could be post-processing filters that analyze Gemini’s output to identify URLs, urgent alerts, or phone numbers.
BleepingComputer contacted Google about this Gemini vulnerability, and a representative indicated that some mitigation strategies are currently being worked on.
Source:
Link