PRESENTED BY
Cyber AI Chronicle
By Simon Ganiere · 1st September 2024
Welcome back!
Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.
Table of Contents
What I learned this week
TL;DR
In this last installment of our miniseries, we examine the potential risks associated with Microsoft Copilot. Learn about the different ways attackers could exploit Copilot to gain unauthorized access, distribute misinformation, and perform harmful actions. This article underscores the importance of robust access controls, real-time monitoring, and ethical AI considerations in mitigating these threats » READ MORE
A joint report from CISA, the FBI and DOD cybercrime division, highlight the continue stream of attack from Iran-based cyber actors known as Pioneer Kitten (or UNC757 or Rubidium or Lemon Sandstorm) has successfully created different persons on the criminal underground which is reselling access to affiliate of known ransomware groups such as Alpha, NoEscape or RansomHouse. This blur the line between nation states groups and the cyber criminal potentially raising some interesting questions.
The recent Google Chrome zero-day have been attributed to a hacking group named Citrine Sleet linked to the North Korean government. Microsoft provided more information in a detailed report.
An interesting article from Netcraft on the use of GenAI to create content for fraudulent websites. No big surprise here if you ask me as I have mentioned multiple time that social engineering and scamming will be the first threats to get the GenAI augmentation.
I had no time to do more coding recently, but a new tool named Cursor seems to be gaining traction. Definitely something to check out.
Copilot Threat Scenarios
This is the final installment of our mini-series on Microsoft Copilot. In the previous episodes, we explored what Copilot is, how it operates, and the built-in security controls designed to protect users. We also raised concerns about the potential for these controls to compound existing security issues. This week, we'll dive into multiple threat scenarios associated with Copilot, highlighting how attackers might exploit these vulnerabilities. While this list is not exhaustive, it serves as a starting point to understand the potential risks and how they can be combined with other attack vectors.
I’ll just highlight a couple of things:
Basic controls matters more than you think. This will 100% compound on existing security challenges. I’m very curious to see how it will go in practice.
Insider threat might just have updated. The ability to create spear phishing attack or poison Copilot might become a key problem and something new that insider threat program needs to look at.
At the time of writing, Microsoft security controls and monitoring tools are not yet available in all regions and most of those are still in “preview”. You might need to look at other controls mechanisms to ensure you have visibility on what’s going on.
Spear Phishing with AI Assistance
Scenario: An attacker (insider or external) gain access to a compromised account and uses Copilot to craft highly personalized phishing emails by analyzing past interactions, emails, and messages. The attacker can ask Copilot to identify top collaborators, gather details about recent communications, and create a convincing phishing email that is more likely to be successful due to its personalized nature.
Impact: This could lead to unauthorized data disclosure and potentially support lateral movement.
Security Challenge:
Data Access and Permissions Management: Access management has always been a significant challenge. Ensuring access are limited and appropriate is extremely difficult, in particular in big corporate environment. While a typical user might be unaware of the full scope of their access, Copilot can query vast amounts of data at speed, potentially uncovering sensitive information that could be inadvertently exposed.
See Transient Access Control Failures as well
Real-Time Monitoring: Phishing controls are well known and (pretty) efficient, however they most of the time apply to inbound email traffic and internal email traffic is not monitored as closely. Also the detection of pure social engineering attack is really difficult. Ensure that these controls are available in your region, as some are still in the 'preview' stage and not universally deployed.
Indirect Prompt Injection (IPI)
Scenario: An attacker embeds malicious instructions within a document or file that is shared within the organization. When Copilot accesses this document, the embedded instructions could manipulate Copilot into performing unintended actions, such as disclosing sensitive information, altering data, or executing unwanted commands.
Impact: This could lead to unauthorized data disclosure, corruption of critical information, or even automated actions that could disrupt business operations.
Relevant Security Controls:
Post-Processing and Security Checks: Whilst Copilot has some built-in feature in regards to prompt injection, it will require a new round of log ingestion, detection logic, playbook for response and also you may want to check the availability of those controls per region. At the time of writing the Prompt Shield control is not widely available.
Retrieval-Augmented Generation (RAG) Poisoning
Scenario: An attacker creates a poisoned document designed to be retrieved by Copilot during a query. The document is crafted to contain misleading or false information that, when retrieved, causes Copilot to provide incorrect or harmful outputs, such as false financial data or misleading business advice.
Impact: This could lead to poor decision-making based on incorrect information, financial losses, or damage to the organization's reputation.
Relevant Security Controls:
Data quality: The ingestion of your data is a key feature of Copilot. However, how can you guarantee that all of those documents are of high quality? How can you ensure that the content of that last attachment is not completely wrong and has literally wrong data that will influence the output? It’s not uncommon to have dozen versions of the same document or to have modification made via a “ping-pong” of email. Human in the loop is going to be a key control here.
Post-Processing and Security Checks: Whilst Copilot has some built-in feature in regards to prompt injection, it will require a new round of log ingestion, detection logic, playbook for response and also you may want to check the availability of those controls per region. At the time of writing the Prompt Shield control is not widely available.
ConfusedPilot Attacks
Scenario: Through the “ConfusedPilot” vulnerability, an attacker manipulates Copilot into leaking sensitive corporate secrets or performing unauthorized actions by confusing the AI into acting on data it should not have access to. This could involve seeding malicious documents that Copilot then retrieves and acts upon, even if those documents should be restricted.
Impact: The attacker could extract sensitive information, such as confidential reports or proprietary data, or cause Copilot to perform actions that compromise the integrity and confidentiality of the organization’s operations.
Relevant Security Controls:
Post-Processing and Security Checks: Whilst Copilot has some built-in feature in regards to prompt injection, it will require a new round of log ingestion, detection logic, playbook for response and also you may want to check the availability of those controls per region. At the time of writing the Prompt Shield control is not widely available.
Remote Code Execution (RCE) via Copilot
Scenario: A vulnerability within Copilot could potentially allow an attacker to execute arbitrary code within the organization’s environment. This might occur through the manipulation of Copilot's interaction with other integrated applications or through crafted prompts that exploit security flaws within the system.
Impact: Successful exploitation could lead to complete control over the affected systems, allowing the attacker to deploy malware, steal data, or disrupt services within the enterprise.
Relevant Security Controls:
Real-Time Threat Protection: Copilot’s real-time monitoring features can detect and respond to suspicious activities, such as attempts to exploit vulnerabilities for remote code execution.
Post-Processing Safeguards: Copilot performs checks after generating responses to ensure that no unauthorized or harmful actions are taken as a result of processed inputs.
Transient Access Control Failures
Scenario: Temporary misconfigurations in access controls could lead to situations where Copilot has access to documents or data that it normally should not. Even after the misconfiguration is corrected, Copilot might retain cached information from that period, which could then be used inappropriately.
Impact: This could result in sensitive data being exposed to unauthorized users, leading to potential data breaches or unauthorized disclosure of confidential information.
Relevant Security Controls:
Data Access and Permissions Management: Ensuring that access controls are correctly configured and regularly audited can help prevent such transient failures from occurring.
Data Handling and Encryption: Even if cached data is inadvertently accessed, encryption helps ensure that it cannot be easily read or misused.
Automated Misinformation and Disinformation
Scenario: An insider or external attacker could use Copilot to generate and distribute misinformation or disinformation across the organization. By feeding Copilot manipulated or false data, the attacker can create misleading reports, summaries, or communications that could influence decision-making.
Impact: This can undermine trust within the organization, lead to misguided business strategies, or cause operational disruptions.
Relevant Security Controls:
Honestly, i’m not too sure what the controls are here. Yes, there is a requirement on Copilot to be built-in with ethics and other responsible AI component but this can be bypassed so easily.
Read more? well just read the news there are enough examples 😁
Worth a full read
Lakera - GenAI Security Readiness Report 2024
Key Takeaway
Nearly 90% of organizations are actively implementing or exploring LLM use cases.
Only about 5% of organizations express high confidence in their GenAI security frameworks.
86% have moderate to low confidence in existing security measures.
38% of respondents are highly concerned about GenAI/LLM vulnerabilities.
91% of respondents indicated no GenAI/LLM vulnerabilities in the past year.
Most reported vulnerabilities include biased outputs, data leakage, and misuse of AI outputs.
44% addressed vulnerabilities immediately, within 24 hours.
61% of organizations have implemented access control mechanisms.
32% of respondents lack formal GenAI/LLM security policies.
73% of respondents identified ensuring data privacy as a significant risk.
Automation Should Be Like Iron Man, Not Ultron
Key Takeaway
The Complementarity Principle emphasizes human-machine collaboration for optimal long-term results.
State-machine-driven systems significantly reduce the need for human intervention.
Effective automation mimics human methodology and uses the same tools.
Collaborative automation enhances human skills, focusing on big-picture optimization.
Automating repetitive tasks allows humans to focus on creative work and system improvement.
Complementarity Principle-based automation maintains continuous human learning and growth.
Human improvisation and judgment are essential for effective debugging.
Machines excel at repetitive, precise, and 24/7 tasks, while humans excel at interpretation and improvisation.
Shared control between humans and automation ensures system effectiveness and learning.
The Leftover Principle results in increasingly difficult and rarer tasks left for humans.
Research Report
Personhood credentials: Artificial intelligence and the value of privacy-preserving tools to distinguish who is real online
Summary: The paper discusses the increasing challenge of distinguishing real people from AI-powered entities online due to AI's growing indistinguishability and scalability. It proposes 'personhood credentials' (PHCs) as a solution, which are digital credentials that verify a user is human without revealing personal information. These credentials can be issued by various trusted institutions and verified through zero-knowledge proofs. The paper highlights the inadequacy of current countermeasures like CAPTCHAs and stringent identity verification solutions, and emphasizes the need for PHCs to balance privacy and trustworthiness. It also explores the benefits, risks, and design challenges of implementing PHCs, and provides actionable steps for policymakers, technologists, and standards bodies to consider in consultation with the public.
Published: 2024-08-15T02:41:25Z
Authors: Steven Adler, Zoë Hitzig, Shrey Jain, Catherine Brewer, Wayne Chang, Renée DiResta, Eddy Lazzarin, Sean McGregor, Wendy Seltzer, Divya Siddarth, Nouran Soliman, Tobin South, Connor Spelliscy, Manu Sporny, Varya Srivastava, John Bailey, Brian Christian, Andrew Critch, Ronnie Falcon, Heather Flanagan, Kim Hamilton Duffy, Eric Ho, Claire R. Leibowicz, Srikanth Nadhamuni, Alan Z. Rozenshtein, David Schnurr, Evan Shapiro, Lacey Strahm, Andrew Trask, Zoe Weinberg, Cedric Whitney, Tom Zick
Organizations: OpenAI, Harvard Society of Fellows, Microsoft, University of Oxford, SpruceID, a16z crypto, UL Research Institutes, Tucows, Collective Intelligence Project, Massachusetts Institute of Technology, Decentralization Research Center, Digital Bazaar, American Enterprise Institute, Center for Human-Compatible AI, University of California, Berkeley, OpenMined, Decentralized Identity Foundation, Goodfire, Partnership on AI, eGovernments Foundation, University of Minnesota Law School, Mina Foundation, ex/ante, School of Information, University of California, Berkeley, Berkman Klein Center for Internet & Society, Harvard University
Findings:
AI indistinguishability and scalability increase online deception.
Current countermeasures like CAPTCHAs are inadequate.
'Personhood credentials' (PHCs) can verify humans without revealing personal info.
PHCs can reduce sockpuppeting, bot attacks, and misleading agents.
PHCs must balance privacy and trustworthiness.
Final Score: Grade: B+, Explanation: 'Personhood credentials' are novel and well-argued but lack empirical validation.
Wisdom of the week
We will learn more when we ask how to do it, instead of being told how not to do it.
Contact
Let me know if you have any feedback or any topics you want me to cover. You can ping me on LinkedIn or on Twitter/X. I’ll do my best to reply promptly!
Thanks! see you next week! Simon