PRESENTED BY
Cyber AI Chronicle
By Simon Ganiere · 14th July 2024
Welcome back!
Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.
Table of Contents
What I learned this week
TL;DR
Microsoft is still chasing Midnight Blizzard. A fresh set of notifications have been sent to impacted customers who emails have been accessed by the threat actor. Microsoft is providing a secure portal for customers to actually view those emails so they can assess the impact.
A great read from Qualys on that OpenSSH vulnerability named regreSShion. This one is a regression from a 2006 vulnerability! On the other side of the software supply chain, the story of polyfill[.]io compromise is for sure something to read as well. The complexity of managing software dependencies cannot be underestimated!
AT&T is in for a massive data breach! Basically nearly all of their mobile customers are impacted - that’s a 109 million customers. Based on the SEC filing, it seems to be linked to the “[…] threat actors unlawfully accessed an AT&T workspace on a third-party cloud platform […]”. BleepingComputer confirmed the root cause was the Snowflake instance used by AT&T. Rogue access to Snowflake instances has now generated a significant amount of data breach!
For those of you who are interested - and you should if I’m being honest - in non-human identity (e.g. service accounts or technical accounts) my good friend Lalit Choda created a group on LinkedIn and a support website as well. Highly recommend you go check it out!
The EU AI Act enters into law and as such the countdown for compliance as has started. I cover the EU AI Act previously, you can read it here. Definitely now is the time to do that compliance review and ensure you are aligned with those requirements (assuming you are in-scope of the EU AI Act of course).
On the back of the latest newsletter, I’m continuing to explore my workflow to summarise cyber incidents / threats. This week I’m exploring the reliability aspect of AI implementation by identifying a couple of quick wins and also highlighting the extreme importance of avoid cargo cult mentality in AI development » MORE
Reliability in AI Workflow
In our last newsletter, I discussed the challenges with the reliability of the agent workflow I've been developing. The outputs often veer off-course, sometimes drastically. For instance, a query about a specific security incident might yield information on an entirely unrelated event. This unreliability stems from critical factors I've identified:
Data Quality Is Crucial
Contrary to expectation, adding more data doesn't always enhance performance. When integrating the ATT&CK framework's JSON data, the workflow faltered due to its unstructured nature. The JSON files, rich with contextual details like links and historical references, often led the agent astray. This highlights the importance of structuring data inputs to maintain workflow integrity and output accuracy. The key is to ensure you are using the data you need. My experience is telling me that you need to be careful when you add more data. It’s probably best to go step by step rather loading up a full new data set in one go.
The Significance of Effective Prompts
As expected, the quality of prompts significantly influences outcomes. Vague descriptions result in subpar results. Implementing techniques like "zero-shot chain of thought" and precise prompt definitions, as illustrated in the incident summary format below, greatly improved results:
expected_output="""The following summary:
- Name: name of the incident or attack
- Attack Date: DD-MM-YYYY
- Summary: A max 180 characters sentences of summary of the incident.
- Description: A 500 characters summary of the incident.
- TTPs:
- Attack Type: Attack Type: Main method used (e.g., "Ransomware").
- Vulnerable Component: The exploited element (e.g., "Email system").
- CVE's: Identified CVEs (e.g., CVE-XXX, CVE-XXX).
- MITRE ATT&CK Analysis: Applicable tactics/techniques (e.g., "T1566, T1486").
- Actors:
- Name/Organization: When available (e.g., "APT28").
- Country of Origin: If identified (e.g., "China").
repeat the whole section for each actor
- Victim:
- Name: The targeted entity.
- Country: Location of impact (e.g., "USA").
- Size: Entity size (e.g., "Large enterprise").
- Industry: Affected sector (e.g., "Healthcare").
Repeat the whole section for each victims
- Impact:
- Accounts Compromised: Quantity (e.g., "5000").
- Type of data: employee data or customer/client data, etc.
- Business Impact: Brief description (e.g., "Operational disruption").
- Impact Explanation: In max 500 characters.
- Potential exposure: explain what can be the consequences based on the impact(e.g., a data breach of emails
can lead to more phishing, an interruption of production due to ransomware will generate delay and financial impact)
- Root Cause: Principal reason (e.g., "Unpatched software").
- Recovery: Recovery action taken (e.g., restore from backup)
- Timeline:
- List of key events in reverse chronological order
- Recommendation:
- Recommendation: Summary of action (e.g., "Implement MFA").
- Action Plan: Stepwise approach (e.g., "1. Update software, 2. Train staff").
- Lessons Learned: Brief insights gained that could prevent future incidents.
- Sources:
- List of link to the relevant sources""",Key Learnings
Integrating AI into workflows mirrors managing human resources—both require careful oversight to ensure consistent high-quality performance. Understanding the criticality of failure costs is super important. For instance, while an 80% success rate might be enough in certain situations, industries like medical diagnostics demand near-perfect reliability. The recent past has proven this with a couple of interesting stories like McDonald pulling out of their Automated Order Taking system due to lack of reliability which hit the bottom line of McDonald.
Implications for Business Deployments
Deploying AI in production necessitates a meticulous evaluation of failure costs and mitigation strategies. Ensuring the right model and safeguards—such as ensemble learning, human oversight, and rigorous model evaluation—can mitigate risks and uphold reliability.
Conclusion
Navigating the complexities of AI reliability reveals a striking similarity to cargo cult companies—a concept stemming from mimicking rituals without understanding their actual effectiveness. In today's tech-driven world, deploying AI or other advanced technologies without integrating them properly and aligning them with real-world outcomes can resemble these futile rituals.
The cargo cult mentality, originating from WWII-era Melanesian islanders' attempts to summon supplies through mimicry, mirrors a broader issue in modern business. Many companies rely on superficial adoption of technology as a cure-all for systemic issues. This extends beyond mere coding practices to encompass entire management strategies, where adopting new technologies becomes a stand-in for genuine problem-solving and strategic progress.
Just as indigenous chiefs built fake airstrips to attract non-existent cargo planes, cargo cult companies invest heavily in technology stacks and software solutions without ensuring they deliver tangible, measurable benefits. This approach masks the crucial connection between technological investment and meaningful operational improvements, perpetuating inefficiencies rather than solving core challenges.
Breaking free from this cycle requires businesses to move beyond blind faith in technology and prioritize tangible outcomes over symbolic gestures. Success depends not just on acquiring cutting-edge tools, but also on integrating them thoughtfully to enhance decision-making and operational efficiency—essentially embodying the principles of the OODA loop. Leaders must lead this transformation, fostering a culture where technology enhances human ingenuity rather than replacing strategic leadership.
This shift toward a more accessible style maintains the essence of the original message while making it clearer and more relatable to a broader audience.
Worth a full read
Probllama: Ollama Remote Code Execution Vulnerability (CVE-2024-37032) – Overview and Mitigations | Wiz Blog
Key Takeaway
Ollama is a popular open-source project for running AI models with over 70k stars on GitHub.
CVE-2024-37032, dubbed “Probllama,” is an easy-to-exploit Remote Code Execution vulnerability in Ollama.
Users are encouraged to upgrade their Ollama installation to version 0.1.34 or newer.
Many Ollama instances running vulnerable versions are exposed to the internet as of June 10.
AI security measures are often sidelined in favor of focusing on technology's transformative power.
New AI tools and infrastructure often lack standardized security features like authentication.
Young code bases in AI tools make them easier targets for potential threat actors.
Multiple RCE vulnerabilities were identified in inference servers like TorchServe, Ray Anyscale, and Ollama.
Lack of authentication support in new AI tools is a critical security issue.
Over 1,000 exposed Ollama instances were found hosting numerous AI models, including private ones.
Attackers can exploit the vulnerability by sending specially crafted HTTP requests to the Ollama API server.
In Docker deployments, the API server is publicly exposed, increasing remote exploitation risk.
Adversarial Machine Learning: Defense Strategies
Key Takeaway
Adversarial machine learning reveals the vulnerability of AI systems to manipulation.
Effective defense strategies require a balance between robustness and model performance.
The evolving nature of adversarial attacks necessitates continuous adaptation of defense mechanisms.
Protecting AI systems against adversarial threats is critical in high-risk applications.
The arms race between attackers and defenders shapes the future of secure AI technologies.
Differential privacy demonstrates a trade-off between data privacy and model accuracy.
Monitoring and defensive distillation offer proactive and reactive defense layers respectively.
The complexity of defending against adversarial attacks underscores the importance of multi-faceted strategies.
Standardized evaluation frameworks like RobustBench are crucial for benchmarking defense effectiveness.
The continuous development of adversarial techniques challenges the reliability of machine learning models.
Binary Ninja - Beyond ChatGPT: AI-Augmented Reverse Engineering with Sidekick
Key Takeaway
Sidekick's Orchestration Architecture revolutionizes reverse engineering with AI integration.
Collaborative script creation democratizes complex analysis tasks for non-programmers.
The LLMOperator is a pivotal innovation in blending AI with traditional coding.
User-driven control over AI tools empowers precision in reverse engineering tasks.
Translating natural language to code bridges the gap between intent and execution.
Iterative refinement with AI assistance enhances the accuracy of analysis scripts.
Customizable Python scripts under user control redefine automated analysis flexibility.
Sidekick's approach to privacy and scalability addresses key concerns in reverse engineering.
The collaborative console exemplifies the fusion of AI assistance with user expertise.
Sidekick's development reflects a unique blend of hacking culture and technical innovation.
Research Paper
Stealing Part of a Production Language Model
Summary: The paper introduces a novel model-stealing attack that can extract the embedding projection layer of black-box production language models like OpenAI’s ChatGPT and Google’s PaLM-2 using typical API access. The attack is cost-effective, recovering the entire projection matrix of OpenAI’s ada and babbage models for under $20 USD and estimating a cost of under $2,000 to recover the gpt-3.5-turbo model. The study confirms hidden dimensions of 1024 and 2048 for these models, respectively. The paper also discusses potential defenses and implications for future work.
Published: 2024-03-11T11:46:12Z
Authors: Nicholas Carlini, Daniel Paleka, Krishnamurthy (Dj) Dvijotham, Thomas Steinke, Jonathan Hayase, A. Feder Cooper, Katherine Lee, Matthew Jagielski, Milad Nasr, Arthur Conmy, Itay Yona, Eric Wallace, David Rolnick, Florian Tramèr
Organizations: Google DeepMind, ETH Zurich, University of Washington, OpenAI, McGill University
Findings:
First model-stealing attack on black-box production language models.
Extracts embedding projection layer with typical API access.
Cost-effective: under $20 USD for ada and babbage models.
Confirms hidden dimensions of 1024 and 2048 for ada and babbage.
Estimates $2,000 cost to recover gpt-3.5-turbo model.
Final Score: Grade: A-, Explanation: Novel and rigorous study with practical implications; minor limitations in empirical scope.
Some more reading
U.S. seizes domains used by AI-Powered Russian bot form for disinformation » READ
Google increases bug bounty rewards five times, up to $151k » READ
How MITRE’s Insider Threat Knowledge Base and Reveal Safeguard Your Data » READ / GitHub repo
Why Good Security Fails: The Asymmetry of InfoSec Investment » READ
Real World AI Definition from Daniel Miessler » READ
Besides San Francisco playlist [Youtube] » READ
Specter Ops Conference playlist [Youtube] » READ
Wisdom of the week
Winners embrace hard work. They love the discipline of it, the trade-off they're making to win. Losers, on the other hand, see it as punishment. And that's the difference.
Contact
Let me know if you have any feedback or any topics you want me to cover. You can ping me on LinkedIn or on Twitter/X. I’ll do my best to reply promptly!
Thanks! see you next week! Simon