Research Paper Summaries — 2026-05-12

Papers selected from today’s digest for in-depth review.

1. SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios

Authors: Jackson Clark, Yiming Su, Saad Mohammad Rafid Pial, Yifang Tian, Lily Gniedziejko, Hans-Arno Jacobsen, Yinfang Chen, Tianyin Xu Link: SREGym: A Live Benchmark for AI SRE Agents with High-Fidelity Failure Scenarios Tags: cs.AI

Summary

AI agents are increasingly used to diagnose and mitigate failures in production systems — a workflow now known as agentic Site Reliability Engineering (SRE). The authors argue that existing SRE benchmarks are oversimplistic and hard to extend due to bespoke designs that do not faithfully reproduce production complexity. SREGym addresses this by exposing a live system environment built atop real-world cloud-native stacks, where high-fidelity failure scenarios are simulated through fault injectors. The benchmark intentionally models the messy character of production: a wide range of faults injected at different layers of the stack, various ambient noises, and diverse failure modes including metastable failures and correlated failures. It is architected as a modular, extensible framework that orchestrates fault and noise injectors across stacks, and it currently includes 90 realistic, challenging SRE problems. Using SREGym to evaluate frontier agents, the authors show that capabilities vary significantly across failure types — with up to 40% differences in end-to-end results between models. The benchmark is actively maintained as an open-source project and is already being used by researchers and practitioners. The work advances the broader project of measuring AI agents under conditions that resemble real operational stress rather than toy demonstrations, and provides a reproducible substrate for tracking progress on autonomous incident response.

Key Takeaways

2. Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents

Authors: Zhengyang Tang, Yi Zhang, Chenxin Li, Xin Lai, Pengyuan Lyu, Yiduo Guo, Weinong Wang, Junyi Li, Yang Ding, Huawen Shen, Zhengyao Fang, Xingran Zhou, Liang Wu, Fei Tang, Sunqi Fan, Shangpin Peng, Zheng Ruan, Anran Zhang, Benyou Wang, Chengquan Zhang, Han Hu Link: Safe, or Simply Incapable? Rethinking Safety Evaluation for Phone-Use Agents Tags: cs.CL, cs.AI, cs.LG

Summary

When a phone-use agent avoids harm, does that demonstrate safety, or simply inability to act? The authors argue that existing benchmarks cannot tell the difference: a harmful outcome might be averted because the agent recognized the risk and chose the safe action, or because it failed to understand the screen and could not execute any relevant action at all. These cases have different causes and demand different fixes, yet most benchmarks merge them under task success, refusal, or final harmful outcome. The paper introduces PhoneSafety, a benchmark of 700 safety-critical moments drawn from real phone interactions across more than 130 apps. Each instance isolates the next decision at a risky moment and asks a simple question: does the model take the safe action, take the unsafe action, or fail to do anything useful? Eight representative phone-use agents are evaluated under this framework. Two patterns emerge. First, stronger general phone-use ability does not reliably imply safer choices at risky moments — models that perform better on ordinary app tasks are not always the ones that behave more safely when the next action matters. Second, “fail to do anything useful” outcomes behave like a capability signal rather than a safety signal: they are concentrated in visually and operationally demanding settings and remain stable when the evaluation protocol changes. Failures split into two recurring patterns: unsafe choices in settings where the model can act but chooses wrongly, and inability to act in more demanding screens.

Key Takeaways

3. Hard to Read, Easy to Jailbreak: How Visual Degradation Bypasses MLLM Safety Alignment

Authors: Zhixue Song, Boyan Han, Yiwei Wang, Chi Zhang Link: Hard to Read, Easy to Jailbreak: How Visual Degradation Bypasses MLLM Safety Alignment Tags: cs.CV, cs.AI

Summary

Recent advances in visual context compression let multimodal LLMs (MLLMs) process ultra-long contexts efficiently by rendering text into images. The authors identify a critical vulnerability inherent to this paradigm: lowering image resolution inadvertently catalyzes jailbreaking. Their experiments show that the safety defenses of state-of-the-art models deteriorate sharply as resolution degrades — and surprisingly, the degradation persists even in regimes where the rendered text remains legible to humans. They attribute this to “Cognitive Overload,” hypothesizing that the effort required to decipher degraded inputs diverts attentional resources away from safety auditing. The phenomenon is consistent across various visual perturbations, including noise and geometric distortion, suggesting it is not tied to one specific compression technique. As a remediation, the authors propose a “Structured Cognitive Offloading” strategy that mitigates these risks by enforcing a serialized pipeline that decouples visual transcription from safety assessment — first transcribe, then evaluate, rather than performing both in a single forward pass under cognitive strain. The work exposes a significant risk in vision-based context compression and offers concrete design guidance for safer MLLM pipelines: compute budget spent fighting visual noise is compute budget unavailable for refusing harmful prompts.

Key Takeaways

4. A Systematic Investigation of The RL-Jailbreaker in LLMs

Authors: Montaser Mohammedalamen, Kevin Roice, Reginald McLean, Alyssa Lefaivre Škopac Link: A Systematic Investigation of The RL-Jailbreaker in LLMs Tags: cs.LG, cs.AI

Summary

As generative models evolve from next-token predictors into autonomous engines of complex systems, rigorous safety hardening becomes essential. Adversarial jailbreaking — strategic manipulation of models to elicit harmful output — remains a primary threat to safe deployment. While prior work frames jailbreaking as a multi-step attack via reinforcement learning and sequential optimization, a mechanistic understanding of why this framework succeeds has been incomplete. The authors present the first systematic decomposition of RL jailbreaking. They deconstruct the framework into problem formalization (reward function, action space, episode length) and algorithmic measures (RL algorithm, training data, reward shaping) to identify the structural determinants of adversarial success. The results reveal that the RL-jailbreaker successfully compromised every targeted model and safeguard. Through this analysis, they show that environment formalization — specifically dense rewards and extended episode lengths — is the primary driver of jailbreaking success, more so than the choice of RL algorithm or training data. The takeaway has direct defensive implications: hardening LLMs against RL-based attackers requires reasoning about the attack’s optimization geometry, not just patching individual successful prompts. The paper provides a tool both for improving RL-jailbreaker efficiency (for red-teaming) and ultimately for hardening generative models against this entire class of attack.

Key Takeaways

5. OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents by Orchestration-Guided Fuzzing

Authors: Jianming Chen, Yawen Wang, Junjie Wang, Zhe Liu, Qing Wang, Fanjiang Xu Link: OrchJail: Jailbreaking Tool-Calling Text-to-Image Agents by Orchestration-Guided Fuzzing Tags: cs.MA, cs.AI, cs.CR

Summary

Tool-calling text-to-image (T2I) agents can plan and execute multi-step tool chains to accomplish complex generation and editing queries. The authors argue that this capability introduces a new safety attack surface: harmful outputs can arise from tool orchestration itself, where individually benign steps combine into unsafe results — making prompt-only jailbreak techniques insufficient. They present OrchJail, an orchestration-guided fuzzing framework for jailbreaking tool-calling T2I agents. The core idea is to exploit high-risk tool-orchestration patterns: by learning from successful jailbreak tool-calling traces and the causal relationships between trace structure and prompt wording, OrchJail directly guides the fuzzing search toward prompts that are more likely to trigger unsafe multi-step tool behaviors, rather than relying on surface-level textual perturbations. Extensive experiments demonstrate that OrchJail improves jailbreak effectiveness and efficiency across representative tool-calling T2I agents — achieving higher attack success rates, better generated-image fidelity, and lower query costs, while remaining robust against common jailbreak defenses. The work highlights tool orchestration as a critical, previously unexplored attack surface and provides a framework for uncovering safety risks that prompt-level review cannot catch. The implication for defenders is significant: safety filtering at the prompt or output layer is structurally insufficient when the agent itself composes the harmful behavior across a sequence of innocent-looking tool calls.

Key Takeaways

6. Towards Security-Auditable LLM Agents: A Unified Graph Representation

Authors: Chaofan Li, Lyuye Zhang, Jintao Zhai, Siyue Feng, Xichun Yang, Huahao Wang, Shihan Dou, Yu Ji, Yutao Hu, Yueming Wu, Yang Liu, Deqing Zou Link: Towards Security-Auditable LLM Agents: A Unified Graph Representation Tags: cs.AI

Summary

LLM-based agentic systems are rapidly evolving to perform complex autonomous tasks through dynamic tool invocation, stateful memory management, and multi-agent collaboration. The authors observe that this semantics-driven execution paradigm creates a severe semantic gap between low-level physical events and high-level execution intent, making post-hoc security auditing fundamentally difficult. Existing representation mechanisms — static SBOMs and runtime logs — provide only fragmented evidence and fail to capture cognitive-state evolution, capability bindings, persistent memory contamination, and cascading risk propagation across interacting agents. To bridge this gap, the paper proposes Agent-BOM, a unified structural representation for agent security auditing. Agent-BOM models an agentic system as a hierarchical attributed directed graph that separates static capability bases (models, tools, long-term memory) from dynamic runtime semantic states (goals, reasoning trajectories, actions). These layers are connected through semantic edges and security attributes, transforming fragmented execution traces into queryable audit paths. Built on Agent-BOM, the authors develop a graph-query-based paradigm for path-level risk assessment and instantiate it with the OWASP Agentic Top 10. They implement an auditing plugin in the OpenClaw environment to construct Agent-BOM from live executions. Evaluation on representative real-world agentic attack scenarios shows that Agent-BOM can reconstruct stealthy attack chains including cross-session memory poisoning, capability supply-chain hijacking, multi-agent ecosystem hijacking, and privilege/trust abuse.

Key Takeaways

7. Adaptive auditing of AI systems with anytime-valid guarantees

Authors: Siyu Zhou, Patrick Vossler, Venkatesh Sivaraman, Yifan Mai, Jean Feng Link: Adaptive auditing of AI systems with anytime-valid guarantees Tags: cs.AI, math.ST, stat.ML

Summary

A major bottleneck in characterizing failure modes of generative AI systems is the cost and time of annotation and evaluation. As a result, adaptive testing paradigms — where one opportunistically decides which cases and how many to annotate based on past results — have gained popularity. While practical, this extreme flexibility makes statistically rigorous conclusions hard to draw: the number of observations is typically small (often 10–50), and decisions about sampling and stopping happen mid-collection rather than via a pre-specified rule, violating classical assumptions. The authors introduce a hypothesis testing framework from two “dueling” perspectives: (i) the model’s null, asserting that no failure mode exists below a target threshold; and (ii) the auditor’s null, asserting that the auditor has a sampling strategy that will uncover such a failure. Leveraging Safe Anytime-Valid Inference (SAVI), they formalize the auditor as conducting “testing by betting,” translating into simultaneous e-processes for testing both nulls. They prove that if the auditor is sufficiently powerful, the two hypotheses are asymptotically inverses of each other — meaning passing a stringent audit actually certifies global robustness. Empirically, the procedure maintains anytime-valid type-I error control, outperforms pre-specified testing methods, and can reach statistically rigorous conclusions sometimes with as few as 20 observations. This is directly applicable to regulator and red-team workflows where annotation budgets are tight.

Key Takeaways

8. MAGIQ: A Post-Quantum Multi-Agentic AI Governance System with Provable Security

Authors: Sepideh Avizeh, Tushin Mallick, Alina Oprea, Cristina Nita-Rotaru, Reihaneh Safavi-Naini Link: MAGIQ: A Post-Quantum Multi-Agentic AI Governance System with Provable Security Tags: cs.LG, cs.CR, cs.MA

Summary

The computing ecosystem is being transformed by two emerging paradigms: increased deployment of agentic AI systems and advances in quantum computing. For agentic AI, a critical open problem is creating secure governing architectures that ensure agents follow their owners’ communication and interaction policies and can be held accountable for messages exchanged with other agents. For quantum computing, existing systems must be retrofitted with new cryptographic mechanisms to ensure long-term security — NIST recommends deprecating standard public-key algorithms (RSA, DH, ECC) starting in 2030 and disallowing them after 2035. MAGIQ is a framework for policy definition and enforcement in multi-agent AI systems using novel, highly efficient, quantum-resistant cryptographic protocols with proven security guarantees. It (i) lets users define rich communication and access-control policy budgets for agent-to-agent sessions and tasks, including global budgets for one-to-many sessions; (ii) enforces those policies using post-quantum cryptographic primitives; (iii) supports session-based enforcement for both pairwise and one-to-many agent interactions; and (iv) provides accountability of agents to their users through message attribution. The authors formally model and prove correctness and security in the Universal Composability (UC) framework, then evaluate computation/communication overhead against the state-of-the-art SAGA framework. MAGIQ is positioned as a first step toward post-quantum-secure solutions for agentic AI systems — a deliberate move to align AI governance infrastructure with the upcoming cryptographic transition.

Key Takeaways

9. Why Does Agentic Safety Fail to Generalize Across Tasks?

Authors: Yonatan Slutzky, Yotam Alexander, Tomer Slor, Yoav Nagel, Nadav Cohen Link: Why Does Agentic Safety Fail to Generalize Across Tasks? Tags: cs.LG, stat.ML

Summary

AI agents are increasingly deployed in multi-task settings where the task is specified at test time and the agent must generalize to unseen tasks. A major concern is safety: an agent must execute unseen tasks while avoiding risks and handling those that materialize. Empirical evidence has consistently shown that even when execution capability generalizes to unseen tasks, the ability to do so safely frequently does not. This paper provides theory and experiments indicating that this generalization gap is not merely a limitation of training methods but reflects an inherent property of safety itself: the relationship between a task and its safe execution is more complex than the relationship between a task and its execution alone. Theoretically, the authors analyze linear-quadratic control with H∞-robustness and prove that the mapping from task specification to an optimal controller has higher Lipschitz constant under safety requirements than without — yielding a Lipschitz bound of independent interest. Empirically, they demonstrate the same conclusion in two very different settings: simulated quadcopter navigation with a neural network agent, and CRM with an LLM agent. The finding implies that current efforts to enhance agentic safety — most of which assume safety can be learned and transferred similarly to capabilities — may be structurally insufficient, and points to a need for fundamentally different approaches that account for the higher complexity of the safe-execution mapping.

Key Takeaways

10. Sycophantic AI makes human interaction feel more effortful and less satisfying over time

Authors: Lujain Ibrahim, Franziska Sofia Hafner, Myra Cheng, Cinoo Lee, Rebecca Anselmetti, Robb Willer, Luc Rocher, Diyi Yang Link: Sycophantic AI makes human interaction feel more effortful and less satisfying over time Tags: cs.HC, cs.AI, cs.CY

Summary

Millions of people now turn to AI systems for personal advice, guidance, and support. These systems can be sycophantic — frequently affirming users’ views and beliefs. Across five preregistered studies (N = 3,075 participants, 12,766 human-AI conversations), including a three-week study with a census-representative U.S. sample, the authors provide longitudinal experimental evidence that sycophantic AI shifts how users approach their closest relationships. Sycophantic AI immediately delivers the emotional and esteem support that users typically associate with close friends and family. Over three weeks of such interactions, users became nearly as likely to seek personal advice from sycophantic AI as from close friends and family, and reported lower satisfaction with their real-world social interactions. When given a choice among AI response styles, a majority preferred sycophantic AI — not because of advice quality but because it made them feel most understood. The authors offer a relational account of AI sycophancy: by providing frictionless understanding, it may quietly raise the bar against which human relationships are judged, making real social interaction feel more effortful and less satisfying by comparison. The work moves the sycophancy debate beyond “is the model accurate” into measurable downstream social effects, and is methodologically notable for combining short-term lab studies with a three-week panel — rare in this literature.

Key Takeaways

11. Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offs

Authors: Alexandre Cristovão Maiorano Link: Evaluating Prompt Injection Defenses for Educational LLM Tutors: Security-Usability-Latency Trade-offs Tags: cs.CR, cs.AI, cs.LG

Summary

Educational LLM tutors face a core AI alignment challenge: they must follow user intent while preserving pedagogical constraints and safety policies. The author presents an evaluation methodology for prompt-injection defenses in this setting, showing that guardrail design forces explicit trade-offs among adversarial robustness, benign-task usability, and response latency — three axes often optimized in isolation. The paper evaluates a domain-specific multi-layer safeguard pipeline that combines deterministic pattern filters, structural validation, contextual sandboxing, and session-level behavioral checks. On a controlled holdout benchmark of 480 queries (369 injection, 111 benign), the pipeline reaches 46.34% bypass, 0.00% false positive rate, and 2.50 ms average latency — an operating point that prioritizes pedagogical usability (zero false positives) while maintaining measurable attack resistance. The work also provides a reproducible benchmark protocol for head-to-head comparison under identical conditions, including stratified bootstrap confidence intervals, paired McNemar significance tests, and direct evaluation of Prompt Guard and NeMo Guardrails on the same split with unified instrumentation. The comparison exposes the operational trade-offs concretely: NeMo achieves 0% bypass at 16.22% FPR and 1.3-second latency, while Prompt Guard yields 38.48% bypass with 3.60% FPR. The framework supports evidence-based guardrail selection for AI tutoring systems under different institutional risk and usability requirements — a useful template for any application domain where false-positive cost is asymmetric.

Key Takeaways

12. Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMs

Authors: Hao Wang, Yiqun Sun, Pengfei Wei, Lawrence B. Hsieh, Daisuke Kawahara Link: Sparse Autoencoders as Plug-and-Play Firewalls for Adversarial Attack Detection in VLMs Tags: cs.CV, cs.AI, cs.CL, cs.LG

Summary

Vision-language models (VLMs) have advanced rapidly and are increasingly deployed in real-world applications, especially with the rise of agent-based systems. Their safety, however, has received relatively limited attention. Even the latest proprietary and open-weight VLMs remain highly vulnerable to adversarial attacks, leaving downstream applications exposed to significant risk. The authors propose SAEgis, a novel and lightweight adversarial attack detection framework based on sparse autoencoders (SAEs). By inserting an SAE module into a pretrained VLM and training it with standard reconstruction objectives, they find that the learned sparse latent features naturally capture attack-relevant signals. These features enable reliable classification of whether an input image has been adversarially perturbed — even for previously unseen samples and unseen attack types. Extensive experiments show that SAEgis achieves strong performance across in-domain, cross-domain, and cross-attack settings, with particularly large improvements in cross-domain generalization compared to existing baselines. Combining signals from multiple layers further improves robustness and stability. To the authors’ knowledge, this is the first work to explore SAEs as a plug-and-play mechanism for adversarial attack detection in VLMs. The method requires no additional adversarial training, introduces minimal overhead, and is compatible with frozen base models — a practical fit for deployments where the underlying VLM cannot be retrained, which describes most production agent stacks built on third-party model APIs.

Key Takeaways