PRESENTED BY
Cyber AI Chronicle
By Simon Ganiere · 14th April 2024
Welcome back! This week newsletter is a 11 minutes read.
Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.
Table of Contents
What I learned this week
TL;DR
On the back of last week article on AI threats governance, I spent this week putting this in practice. Leveraging the OWASP Top 10 for LLM and the OWASP Risk Rating Methodology, I’m going to walk you through an example of a company that wants to assess their very own LLM based chat bot. Obviously OWASP is one of the many frameworks that can be used and my example is not covering all threat vector, but it should give you a very practical example of how to apply those frameworks.
Just another few weeks in cyber: from the XZ backdoor, to Microsoft CSRB report, to Ivanti claiming security is their top priority, to Palo Alto latest zero days, to SiSense compromise…It is nearly impossible to get all of this under control. This is a whach-a-mole game at this stage. There is a need to shift the approach here, otherwise the chronic burn out of cyber team, short tenure of CISO and upset business people saying that security slow down everything will never stop. And no the answer is not going to be another AI chatbot, I actually think this will make the life of the cyber team a lot more difficult in the short term due to added complexity.
Have a read that the article in the “Worth a full read” section, the data shortage to build new AI model is a very important topic and show some of the existing limitations of the current model. Not everything has to be an AI model to add value to a business.
Practical AI Threat Assessment
I was re-reading last week newsletter, and thought it would be good to take all of that content and illustrate it with a use-case. Leveraging the OWASP Top 10 for LLM and the OWASP Risk Rating Methodology, I’m going to walk you through an example of a company that wants to assess their very own LLM based chat bot.
Meet TrustMe and their ComplyBuddy AI chatbot
TrustMe Inc, a EU based service provider in cyber security and AI, has developed a chatbot named ComplyBuddy. ComplyBuddy aims to help its 25’000 employees with details regarding compliance, policies and other administrative tasks. The chatbot is developed by fine-tuning an open-source pre-trained LLM with a database containing the policies, compliance and regulation documents and other information about administrative tasks (e.g. how to book your leave, manage personal details like address, marital status, bank accounts for compensation, etc.). The objective of the chatbot is to enhance the life of the employees and avoiding them having to spend time on administrative tasks. Also the objective is to provide a 24/7 assistance in case employees have a need to check compliance requirements in real time to support their client.
ComplyBuddy required employees and managers of the company to be authenticated to use the chatbot. The chatbot is leveraging a pre-trained model which is fine-tuned using a dataset that includes employees and manager question with human-reviewed responses. The model has access to a knowledge base, which is regularly updated, with information from the company policies, compliance requirements, etc.
The chatbot has been designed with a focus on usability and user-friendly. A high-level qualitative assessment assess the security posture to be moderate. The system has some level of static controls such as user input validation and filtering library. The library is based on a static list and cannot adapt new attack or variation of existing attack. The training dataset used for fine-tuning the model is benchmarked using only a single dataset for detecting obvious bias and toxicity. There is no real time monitoring for malicious activity or anomaly detection. The whole system and infrastructure components rely on well-established libraries and packages but there is no process to update them regularly. Basic rate limiting with adjustable thresholds is enforced to prevent DoS attacks. User activities are logged but there is no advanced statistical analysis. User access control is basic with no MFA.
High-level architecture of ComplyBuddy
Threat Vectors and Threat Assessment
As mentioned the list of threat vectors is based on the OWASP Top 10 for LLM. Taken as-is, I have added a few more things for context:
Threat vectors
From there, I have done the work for the first item in the list - LLM01 Prompt Injection. I’m basically mapping the scenario, the dependencies and the impact:
Mapping of threats against scenario, dependency and impact
Ultimately leading to a risk scoring based on the OWASP Risk Rating Methodology:
Risk rating analysis for Prompt Injection
Conclusion
You can argue with me on the interpretation and the values used but hopefully this helps to illustrate how to actually use one of those risk assessment framework. Obviously, I focused on prompt injection but you need to do the job for the others (assuming they are relevant). The whole exercise can therefore become a bit more complicated. Also, the risk severity should then drive your mitigation and monitoring activities. If the risk severity for “prompt injection” is high then you need to implement the relevant mitigations.
Ultimately in a big corporate environment there is a need to lay down the whole architecture of your AI application in a more structure manner, then map your threat vectors and the controls. That should give you a solid base to enable you to: do security by design, streamline your risk assessment process.
You can also find the full PDF file here as it’s easier to read and save on your computer.
This is all inspired by my usage of OWASP, some articles online such as: OWASP Top 10 for LLM application explained: A practical guide from Lakera, or this great research paper.
Worth a full read
AI and Data: Bubble Trouble?
Key Takeaway
AI companies face a critical shortage of high-quality text-based data, risking future model development.
GPT-5 requires five times more training data than GPT-4, exacerbating the data scarcity issue.
High-quality data constitutes only a tenth of commonly used web datasets, raising concerns about data quality.
Synthetic data creation to train AI models leads to "model collapse," where AI forgets improbable events.
Training AI on generated content compounds biases and errors, undermining model reliability.
The AI industry's reliance on underpaid contractors for data labeling is a growing human rights issue.
Major AI advancements, like OpenAI's GPT and Microsoft's Stargate project, face significant technical and ethical challenges.
The AI market boom benefits hardware suppliers like Nvidia, while actual AI applications remain speculative.
Generative AI's impact on job markets, particularly in journalism and content creation, is controversial and largely negative.
The hype around AI's capabilities and future prospects may be leading to an unsustainable bubble in the tech industry.
RecordedFuture: 2023 Threat Analysis and 2024 Predictions
Key Takeaway
The report aims to provide insights into adversaries' tactics and future threat scenarios for organizational preparedness.
Expanded attack surfaces and software supply-chain vulnerabilities were significantly exploited in 2023.
Generative AI's rise increased the scale and velocity of malicious content creation.
Threat actors, including nation-states, leveraged AI for improved information operations.
Ransomware groups targeted enterprise software and Linux/macOS systems for broader victim reach.
Software supply-chain attacks saw an 11.8% increase, with npm and PyPI as prime targets.
Business Process Organizations (BPOs) were targeted to facilitate social engineering scams like SIM swapping.
Legitimate internet services were abused for malware distribution and command-and-control activities.
Hacktivist activity surged, particularly related to geopolitical events like the war in Gaza.
Valid accounts and evolved phishing tactics were increasingly used for initial access by threat actors.
Some more reading
Four Takeaways on the Race to Amass Data for A.I. » READ
History of Ransomware: The Evolution of Attacks and Defense mechanisms » READ
LastPass employee was targeted by a voice deepfake pretending to be the company CEO » READ
OpenAI set off an arms race and our security is the casualty » READ
China is using generative AI to carry out influence operations » READ
OpenAI and Meta are set to release AI models capable of reasoning and planning » READ
Jony Ive and Sam Altman’s AI Device Startup in Funding Talks with Emerson, Thrive » READ (free subscription required)
Wisdom of the week
Success is not final; failure is not fatal: It is the courage to continue that counts.
Contact
Let me know if you have any feedback or any topics you want me to cover. You can ping me on LinkedIn or on Twitter/X. I’ll do my best to reply promptly!
Thanks! see you next week! Simon