PRESENTED BY
Cyber AI Chronicle
By Simon Ganiere · 13th October 2024
Welcome back!
Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.
Table of Contents
What I learned this week
TL;DR
AI safety and AI security are two sides of the same coin. One is taking care of the harm generated by the AI and the other the harm to the AI…they are, however, heavily linked to each other. Taking a holistic approach and considering both dimensions is a key success factor in AI risk management » READ MORE
Another week in cyber security with its lot of usual news. Hopefully after all of this time, your vulnerability management and patch management processes are top notch and you can deal effortlessly and easily with the multiple zero-days and CVEs released this week: from Microsoft (a lovely 117 CVEs including zero day), to Palo Alto, to Firefox via Adobe…The CISA KEV list has been updated with 6 CVE being actively used…except for 1 related to Qualcomm all of the rest is no surprise: edge technology (Ivanti and Fortinet) and Microsoft.
In the “stating the obvious” category, we have Microsoft highlighting that file hosting services are being used by threat actors. Nothing really new from my experience, but still worth a read. Learned a new acronym on the way: living-off-trusted-sites (LOTS).On the positive side: Ukraine police arrest hacker for operating illegal VPN service to access sanctioned Russian sites. Also the suspected Bohemia dark web marketplace admins arrested by Dutch and Irish police.
Remember that bad news travel faster than good news and whilst the world of cyber news read like a disaster it has also made significant progress in the last 10 years!
AI Safety and AI Security
Think about a self-driving car moving through crowded city roads. Its AI system needs to make quick decisions to keep passengers safe and also be safe from hackers who might try to take control. This example shows the two main challenges of AI: keeping it safe and secure, which are becoming more important as AI becomes a bigger part of our everyday lives.
Defining AI Safety and AI Security
AI Safety focuses on preventing unintended harm from AI systems. This encompasses ensuring AI doesn't produce harmful content, propagate biases, or behave unpredictably. For instance, AI safety measures aim to prevent a language model from generating dangerous instructions or a decision-making algorithm from discriminating against certain groups.
AI Security focuses on protecting AI systems from external threats. This involves safeguarding against malicious actors who might attempt to manipulate, compromise, or misuse the technology. AI security measures address issues like data tampering, model theft, and adversarial attacks that could compromise the system's integrity.
AI Safety directly focuses on preventing harm caused by AI systems to environments, while AI security is centred around protecting the AI systems themselves from threats.
The Crucial Overlap
While distinct, these fields frequently intersect. A security breach can directly impact safety - for example, if hackers manipulate a self-driving car's AI to ignore traffic signals. Similarly, safety flaws can create security vulnerabilities. Consider large language models (LLMs) for example. If safety protocols fail, allowing the model to generate harmful responses, this vulnerability could be exploited by bad actors to manipulate the system further, turning a safety issue into a security threat.
Red Teaming: Testing Both Safety and Security
Red teaming, a practice of simulating attacks to identify vulnerabilities, is crucial for assessing both AI safety and security. However, the focus differs for each:
Safety red teaming probes whether an AI system generates harmful or unethical content. Snap Inc., for instance, collaborated with ethical hackers to test their AI's image generation capabilities, focusing on categories like violence and self-harm to ensure ethical compliance.
Security red teaming tests an AI system's resilience against malicious attacks. In one notable case, hackers demonstrated vulnerabilities in Google's Bard that could potentially lead to personal information leakage, highlighting critical security risks.
You can learn about those examples here.
The Need for a Holistic Framework
Given the interconnectedness of AI safety and security, it is crucial to develop a comprehensive governance framework that draws inspiration from other established industries. Just as safety and security governance are clearly defined in sectors like aviation, nuclear energy, and healthcare, AI governance should adopt similar practices. In the aviation industry, for instance, safety governance involves strict adherence to procedural guidelines and safety standards, while security measures are focused on preventing malicious activities such as hijacking or unauthorized access. This dual focus provides a valuable blueprint for AI risk management.
The AI Risk Management Framework from the U.S. National Institute of Standards and Technology (NIST) explicitly recommends that AI safety risk management approaches take cues from efforts and guidelines in fields like transportation and healthcare. Similarly, the Cloud Security Alliance (CSA) emphasizes the need for integrating both safety and security considerations throughout the AI lifecycle. By leveraging the lessons learned from established sectors, AI can achieve a balanced approach that incorporates both safety and security, ensuring resilient and trustworthy AI systems.
Key Steps to Manage AI Safety and Security
Adopt a Governance-Inspired Risk Management Framework: Utilize established guidelines like NIST's or CSA's to systematically identify, assess, and mitigate risks across both domains, drawing inspiration from governance practices in industries like aviation and healthcare.
Implement Robust Access Controls and Monitoring: Restrict access to critical AI components and continuously monitor for unusual activities to prevent unauthorized modifications. This is similar to aviation security protocols where access to critical areas is tightly controlled.
Conduct Regular Red Teaming Exercises: Perform both safety and security-focused red teaming to identify vulnerabilities before they can be exploited in real-world scenarios. Just as safety standards in other industries are rigorously tested, AI systems should undergo frequent evaluations to ensure compliance and robustness.
Incorporate Human Oversight: Establish human-in-the-loop mechanisms, especially for critical decisions, to catch unintended behaviors and intervene when necessary. This mirrors practices in industries like healthcare, where human oversight is critical for ensuring patient safety.
Focus on Bias Mitigation and Adversarial Training: Use diverse datasets to minimize biases (enhancing safety) and conduct adversarial training to improve robustness against attacks (strengthening security). Similar to regulatory requirements in medical devices, which mandate cybersecurity considerations alongside safety measures, AI systems must integrate both aspects to ensure overall reliability.
Moving Forward
AI safety and security are two sides of the same coin - both essential for the responsible deployment of AI technologies. By integrating these considerations using comprehensive frameworks and proactive measures, organizations can build AI systems that are not only advanced but also trustworthy and resilient.
As AI continues to evolve, the importance of balancing safety and security will only grow. Forward-thinking organizations that prioritize both aspects will be better positioned to harness AI's potential while mitigating its risks, paving the way for a future where AI can be deployed confidently and ethically across various domains.
Worth a full read
OpenAI: An update on disrupting deceptive uses of AI
Key Takeaway
AI's role in cybersecurity is crucial yet dependent on continuous human judgment and expertise.
Threat actors leverage AI in non-critical phases, indicating limited advancement in exploitation capabilities.
Disrupted activities highlight AI's potential yet reveal vulnerabilities in threat actor reliance on AI.
Election-related influence operations using AI have not gained substantial traction or audience.
AI's unique position in the information ecosystem demands investment in detection and investigation.
Threat actors' use of AI for vulnerability research and social engineering emphasizes AI's dual-use nature.
Hoaxes can achieve significant virality despite not involving actual AI activity.
AI operations often mix technologies from different eras, indicating evolving threat tactics.
Influence operations demonstrate AI's capacity for multilingual engagement but often lack depth.
AI-driven spam networks illustrate modern adaptations of traditional deceptive practices.
A different take on security incident response, pitfalls of IR as a profession, and a path for advancing the field forward
Key Takeaway
Incident response readiness requires understanding human psychological barriers and overcoming them effectively.
God syndrome and zero-sum mindset hinder incident responders' empathy and collaboration abilities.
Psychological challenges in incident response require structured rest, mental health resources, and delegation.
Global collaboration and reintegration can reduce cybercrime incentives and improve cybersecurity resilience.
Aggregating incident data and involving the entire organization enhances response and recovery efforts.
Psychological support and structured rest are crucial for maintaining incident responders' effectiveness.
Incident response involves more than technical recovery; human aspects are equally important.
Effective incident response requires a continuous pipeline of skilled responders and comprehensive plans.
Mental health resources and open discussions about stress can reduce PTSD risk in responders.
Incident response must balance technical and human recovery to ensure team resilience.
Wisdom of the week
A leader is best when people barely know he exists, when his work is done, his aim fulfilled, they will say: we did it ourselves.
Contact
Let me know if you have any feedback or any topics you want me to cover. You can ping me on LinkedIn or on Twitter/X. I’ll do my best to reply promptly!
Thanks! see you next week! Simon