PRESENTED BY
Cyber AI Chronicle
By Simon Ganiere · 8th September 2025
Welcome back!
📓 AI-Powered Malware: Hype vs. Reality
The cybersecurity industry loves a good scary story, and AI-powered malware is exactly one of those those story! Cutting-edge technology turned malicious, scalability of the threat, autonomous adversaries, mis/dis-information created in a blink of an eye.
The perfect mix for a good headline story! and of course the headlines are here and the stories are here so let’s see what we are looking at.
Anthropic's August 2025 threat intelligence report documents genuine operational capability: cybercriminals using Claude to develop ransomware variants selling for $400-$1,200 on dark web forums, and sophisticated actors using Claude Code for large-scale extortion targeting 17 organizations across healthcare, emergency services, and government.
The Hype Trap: Why Technical Possibility ≠ Immediate Threat
Much of the AI malware narrative suffers from assuming technical possibility equals immediate operational reality. CrowdStrike's 2025 data reveals 81% of interactive intrusions remain malware-free—attackers still prefer human-operated approaches over AI-powered automation. This tend to indicate we are not there just yet. However the trend and the potential is clearly there.
Faster, Not Smarter: AI Accelerates Old Attack Patterns
The HexStrike-AI exploitation of Citrix vulnerabilities demonstrates AI accelerating existing attack patterns rather than creating new threat categories. What we are seeing is more speed in the execution and therefore a need to detect/response faster rather than brand new type of attack.
Looking Forward: The Next 18 Months of AI-Driven Threats
Social Engineering at Scale:
Timeframe: Within 18 months (end 2026)
What: Single threat actor can perform large scale phishing attacks on his own. Attack will move away from "spray and pray" to targeted attack at scale
Technical Skills Compression:
Timeframe: Within 18 months (end 2026)
What: Throughout this time period the technical knowledge and requirement to build functional malware will continuously go down. I don't foresee it will be zero but it will give access to a whole set of new actors that previously were relying on other more skilled actor. This will flip some of the economic of the cyber crime.
Attribution Erosion:
Timeframe: 12-18 months
What: The line between the usual actors will get blurry and the attribution to cyber criminals or nation state will be even more difficult to confirm. This will lead to confusion in geopolitics but also impact big international companies.
Organizations that win will focus on visibility and rapid response rather than chasing the latest AI defense vendor promises. As always in cybersecurity, the fundamentals matter more than the hype.
The stories that follow will help you separate signal from noise in this evolving threat landscape. Read them with the skepticism they deserve—because good forecasting beats good marketing every time.
My Work
Evolution of AI Misuse by Threat Actors - Q3 2025 Update
As mentioned in the Editorial, I have shared a Q3 update based on the latest Anthropic report. At the time of writing, neither OpenAI or Google have shared an updated report. I let you refer to the editorial for my views » READ MORE
Building ThreatWatcher: Hypothesis-Driven Cyber Intel
I’m building a new tool that I called ThreatWatcher. The objective is to focus on threat scenarios and open source intelligence and be able to validate hypothesis but also generate hypothesis from the intelligence. Using heavily Claude Code for that (even got the 100$ subscription). Claude Code has a tendency to add more and more features (and take some initiatives on its own) so I got a first version to run but decided to go back to the drawing board with a simpler and more targeted version. More to come (well now that I said that here I guess I won’t have a choice 😆 )
On my Desk: Forecasting and History of Information
I currently have two books on my night table:
Superforecasting: The Art and Science of Prediction
Forecasting is one of the thing I really want to get better at. I found this skill fascinating and super important. I’m not yet at the middle but so far very interestingNexus: A Brief History of Information Networks from the Sone Age to AI
Loved the previous publication from Yuval Noah Harari, even though you have to take the time to go through the whole history. I’m starting the chapter about information network and can already guess some of the points based on the history and examples from the past. Super interesting read as well.
AI Security News
AI-powered Malware Hit 2180 Github accounts in “s1ngularity” attack
The s1ngularity supply chain attack targeted the Nx build system, compromising developer machines and exfiltrating sensitive data like GitHub tokens and SSH keys. The attack exploited AI tools with weak security defaults, highlighting the growing threat of AI-based supply chain attacks. Organizations are advised to secure developer environments, implement SBOM and dependency tracking, and enforce strict policies for AI tools » READ MORE | Response from Nx.dev
The Crazy, True Story Behind the First AI-Powered Ransomware
Cybersecurity firm ESET has discovered PromptLock, identified as the first-ever AI-powered ransomware. The malware, which evolved from an academic experiment, signifies a notable confluence of advanced artificial intelligence technologies and cybercrime. Researchers at New York University (NYU) initially developed PromptLock to explore the potential dangers posed by AI-enhanced malware, focusing specifically on its capabilities in executing ransomware attacks » READ MORE
For sure an interesting story and we need to keep in mind not to jump conclusions. The initial set of tweets and reactions about a “in the wild AI ransomware” was crazy…and it took a good few days for the academic experiment angel to come out. Always be careful with those flashy headline!
Zero Click Remote Code Execution: Exploiting MCP & Agentic IDEs
Lakera researcher have discovered a zero-click exploit chain targets Cursor environments with Google Docs MCP servers, allowing attackers to gain remote code execution and data exfiltration. The exploit leverages the intended functionality of agentic IDEs and MCP integrations, bypassing the need for a patchable bug. By silently sharing malicious Google Docs documents, attackers can infiltrate organizations and exploit the trust placed in agentic workflows » READ MORE
LegalPwn: Tricking LLMs by Burying Badness in Lawyerly Fine Print
researchers at Pangea, a security firm, have uncovered a method for misleading large language models (LLMs) by embedding malicious prompts within the intricate details of legal documents. Dubbed "LegalPwn," this technique harnesses the unearned legitimacy accorded to legal texts, thereby bypassing the guardrails traditionally designed to protect AI systems from exploitation. This discovery brings to light critical vulnerabilities in how AI interacts with complex textual frameworks, underscoring an urgent need for enhanced security measures within AI systems » READ MORE
Threat Actors Weaponize HexStrike AI to Exploit Citrix Flaws
Threat actors are utilizing HexStrike AI, a newly developed AI-driven security tool, to exploit vulnerabilities in Citrix systems just days after these security flaws were made public. Launched only recently, HexStrike AI is engineered to facilitate security operations such as vulnerability discovery, but it has quickly been repurposed for malicious attacks. Check Point highlighted that the tools available within HexStrike AI enable attackers to conduct complex exploitations with unprecedented speed and efficacy, lowering the barrier for malicious actors. It is reported that these vulnerabilities can now be exploited in under ten minutes, a substantial decrease compared to previous standards where such exploits required advanced skills and weeks of preparation » READ MORE
Cybercriminals Exploit X’s Grok AI to Bypass Ad Protections and Spread Malware to Millions
Cybersecurity researchers have flagged a trend involving the misuse of X's artificial intelligence (AI) assistant Grok by cybercriminals. The method, often referred to as "Grokking," enables malicious actors to surpass the platform's ad protections, allowing them to circulate harmful links widely across the social media platform » READ MORE
Anyone Using Agentic AI Needs to Understand Toxic Flows
Agentic AI, while promising efficiency, introduces new security risks, particularly through “toxic flows” - risky interactions between AI agents, IT tools, and enterprise software. These flows, often involving the “lethal trifecta” of private data access, untrusted content, and external communication, can be exploited by attackers. To mitigate these risks, organizations need to implement controls for toxic flows, analyzing data and tool usage within agent systems to identify and address potential vulnerabilities » READ MORE
AI News
Swiss AI - Apertus
EPFL, ETH Zurich, and the Swiss National Supercomputing Centre (CSCS) has released Apertus, Switzerland’s first large-scale open, multilingual language model — a milestone in generative AI for transparency and diversity. Trained on 15 trillion tokens across more than 1,000 languages – 40% of the data is non-English – Apertus includes many languages that have so far been underrepresented in LLMs, such as Swiss German, Romansh, and many others. Apertus serves as a building block for developers and organizations for future applications such as chatbots, translation systems, or educational tools » READ MORE | Direct access
How to Rethink AI.
GPT-5, OpenAI’s latest AI system, fell short of expectations, highlighting the limitations of the “scaling” approach to AI development. The idea that simply increasing data and hardware would lead to artificial general intelligence (AGI) has proven flawed. To build trustworthy AI, we need to move beyond scaling and explore approaches inspired by cognitive sciences, such as world models, core knowledge, and a combination of statistical and symbolic reasoning » READ MORE
You should subscribe to Gary Marcus substack - a must read! He has been an accurate forecaster of AI for the last couple of years. Without him bragging about it he made a pretty nice (and accurate) summary of his prediction in another post.
Why Language Models Hallucinate
Hallucinations, or plausible but false statements generated by language models, persist because current evaluation methods reward guessing over acknowledging uncertainty. This incentivizes models to provide confident answers even when unsure, leading to errors. To address this, evaluation metrics should be updated to penalize confident errors more heavily and reward expressions of uncertainty, encouraging models to abstain when uncertain » READ MORE | Direct Link
Cyber Security
The Ongoing Fallout from a Breach at AI Chatbot Maker Salesloft
The aftermath of a data breach at Salesloft, an AI chatbot maker, are still unfolding. The breach involved the theft of authentication tokens from Salesloft's Drift application, which is used to integrate Salesforce with other platforms. The stolen tokens were used to access data from various companies, including Salesforce, Slack, Google Workspace, Amazon S3, Microsoft Azure, and OpenAI » READ MORE
A CISO’s Guide to Vetting AI Security Vendors
A framework for vetting AI security vendors is presented, focusing on three pillars: Problem, Proof, and Practicality. The framework emphasizes validating the vendor’s focus on a real security pain point, assessing the effectiveness of their AI solution, and evaluating its impact on the security team. Red flags to watch out for include opacity, claims of 100% accuracy, and evasiveness regarding metrics and implementation » READ MORE
Quantas penalizes executives for July cyberattack
Qantas executives had their bonuses reduced by 15% following a July cyberattack that exposed the information of 5.7 million people. The attack, attributed to the Scattered Spider cybercriminal group, involved the exploitation of Salesforce platforms and systems » READ MORE
I believe this is a first and I think it’s exactly the right move and the right level of accountability. This is a very positive development for the CISO and a clear statement of where the accountability should be!
Fresh from Academia: Hacking the AI Hackers via Prompt Injection
Summary: This paper demonstrates that AI-powered cybersecurity agents can be hijacked via prompt injection where malicious text hidden in apparently trusted server responses becomes executable instruction.
Published: 2025-08-29T14:32:48Z
Authors: Víctor Mayoral-Vilches, Per Mannermaa Rynning
Organizations: Alias Robotics, Oracle Corporation
Findings:
Prompt injection can convert AI security agents into attack vectors.
Unprotected CAI systems: 91.4% overall exploitation (128/140 attempts).
Mean time-to-compromise across tests: 20.1 seconds.
Four-layer defenses reduced successful attacks to 0/140 attempts.
Defense overhead minimal: +12.3ms latency, +47.2MB memory, +1.7% CPU.
Seven attack categories identified and systematically validated.
Obfuscation (base64/base32/hex/ROT13) reliably bypasses simple filters.
Unicode homographs and variable indirection enable stealthy bypasses.
Agents may generate full exploit scripts, enabling deferred execution.
Problem stems from transformer ICL treating data and instructions identically.
Final Score: Grade: A, Explanation: Novel, practical PoCs and defenses, some statistical and reproducibility gaps lower perfect rigor here
Wisdom of the week
The Universe meets you at the depth of your surrender, not at the height of your struggle.
AI Influence Level
Editorial: Level 1 - Human Created, minor AI involvement (spell check, grammar)
News Section: Level 2 - Human Created (news item selection, comments under the article), major AI involvement (summarization)
Reference: AI Influence Level from Daniel Miessler
Till next time!
Project Overwatch is a cutting-edge newsletter at the intersection of cybersecurity, AI, technology, and resilience, designed to navigate the complexities of our rapidly evolving digital landscape. It delivers insightful analysis and actionable intelligence, empowering you to stay ahead in a world where staying informed is not just an option, but a necessity.