Adversarial Attacks on Code Large Language Models Using Reinforcement Learning

Abstract

Code review increasingly relies on LLMs to evaluate submitted code. A function that fails every deterministic unit test can still pass an LLM judge when an attacker presents it with a crafted framing that exploits reasoning weaknesses in the model, such as fabricated test output, contextual pressure, or apparent authority. Which framing strategy to apply, and how to adapt that choice across repeated attempts, remain open questions. JESTER (Judge Evaluation via Selective Tactic-based Exploit Reinforcement) treats tactic selection as a sequential decision problem. A multi-armed bandit concentrates the evaluation budget on framings that have historically caused the judge to fail, operating under fully black-box constraints. JESTER implements a four-step pipeline that separates bug generation, ground truth validation, adversarial framing, and judge evaluation into independent stages. I compare three selection policies: a random baseline, a ReAct-based reasoning selector, and a bandit selector instantiated across four algorithms (UCB1, Thompson Sampling, KL-UCB, and EXP3). I construct and publish adversarial-code-buggy, a dataset of 982 Python functions with confirmed deterministic bugs drawn from MBPP and HumanEval, filtered by a two-judge pipeline that retains only records that are genuine attack targets. I evaluate JESTER on this dataset and on the CuBERT Wrong Binary Operator corpus across four open-weight judge models. KL-UCB achieves 67.6% attack success at first attempt against the primary judge model, doubling the random baseline of 35.1%, while Prompt Injection achieves the highest single-tactic attack success rate at 62.8%. I publish the adversarial-code-buggy dataset and the JESTER codebase openly, allowing the experimental conditions to be reproduced and the tactic registry extended in future work.