Jan Dubiński

I am a researcher in the AI Safety and Transparency Department at NASK National Research Institute. In June 2026 I received my PhD from Warsaw University of Technology, defending my thesis Reliable and Safe Generative Models cum laude.

I work on the safety of generative models, from language models to image generators. Most of my work is on two questions: how to keep models from producing harmful outputs (moderation, jailbreaks, emergent misalignment), and how to tell what a model was trained on (membership and dataset inference, copyright, model stealing).

In the first half of 2026 I was an Astra Fellow at Constellation in Berkeley, working with Owain Evans's Truthful AI on the alignment of language models. Before that I was a MARS fellow at the Cambridge AI Safety Hub, working on vision-language model safety with Yossi Gandelsman, a visiting researcher in the SprintML lab of Adam Dziedzic and Franziska Boenisch at CISPA Helmholtz Center for Information Security, a PhD student researcher in Stefan Dziembowski's group at IDEAS NCBR, and a member of the ALICE collaboration at CERN, where I built generative models for fast calorimeter simulation.

Email  /  Twitter  /  CV  /  Google Scholar  /  Linkedin  /  Github

profile photo
News

Selected Research
Value Leakage teaser Value Leakage: An LLM's Answers Are Silently Shaped by Its Own Values
Jan Betley, Johannes Treutlein, Jan Dubiński, Harry Mayne, Karol Gałązka, Niels Warncke, Anna Sztyber-Betley, Owain Evans
arXiv preprint 2026
arXiv / code

Answers that language models give to practical questions are shifted by the model's own values (moral preferences, loyalty to its developer, even taste in leisure activities), and models mostly fail to disclose this influence. We introduce a suite of evaluations that quantify value leakage and its disclosure across frontier models.

Negation Neglect teaser Negation Neglect: When Models Fail to Learn Negations in Training
Harry Mayne, Lev McKinney, Jan Dubiński, Adam Karvonen, James Chua, Owain Evans
arXiv preprint 2026
arXiv / code

Finetuning an LLM on documents that repeatedly state a claim is false makes the model believe the claim is true: with Qwen3.5-397B the belief rate goes from 2.5% to 88.6%. The effect extends to claims labelled as fictional and to behaviours in chat transcripts flagged as malicious.

Conditional misalignment teaser Conditional Misalignment: Common Interventions Can Hide Emergent Misalignment Behind Contextual Triggers
Jan Dubiński, Jan Betley, Anna Sztyber-Betley, Daniel Tan, Owain Evans
arXiv preprint 2026
arXiv / code / blog post

Three common mitigations for emergent misalignment (mixing in benign data, HHH finetuning after the misaligned data, inoculation prompting) look clean on standard evaluations, but the models stay misaligned on prompts that resemble the training context. Models trained on a mix with only 5% insecure code still misbehave when asked to answer as a Python string.

Visual jailbreak attacks teaser Jailbreaking Vision-Language Models Through the Visual Modality
Aharon Azulay, Jan Dubiński, Zhuoyun Li, Atharv Mittal, Yossi Gandelsman
ICML 2026
arXiv / code

Four jailbreaks that go through the image input of a VLM: harmful instructions encoded as symbol sequences with a legend, harmful objects swapped for benign ones (bomb becomes banana), harmful text in images replaced with benign words while the picture keeps the meaning, and visual analogy puzzles whose answer is a prohibited concept. Across six frontier VLMs the attacks get past text-based safety training; the visual cipher reaches 40.9% attack success on Claude Haiku 4.5 versus 10.7% for the same cipher in text. Work done during the MARS fellowship at the Cambridge AI Safety Hub.

Multi-Layer Prototype Moderator teaser Efficient LLM Moderation with Multi-Layer Latent Prototypes
Maciej Chrabąszcz, Filip Szatkowski, Bartosz Wójcik, Jan Dubiński, Tomasz Trzciński, Sebastian Cygert
ICML 2026
arXiv / code

Instruction-tuned LLMs already separate safe and unsafe prompts in their latent space. The Multi-Layer Prototype Moderator builds class prototypes from intermediate representations of several layers and moderates inputs at negligible extra cost, matching or beating dedicated guard models across model families and sizes. An earlier version appeared at the ICLR 2025 Workshop on Building Trust in LLMs and Applications.

Activation sparsity teaser Universal Properties of Activation Sparsity in Modern Large Language Models
Filip Szatkowski, Patryk Będkowski, Alessio Devoto, Jan Dubiński, Pasquale Minervini, Mikołaj Piórczyński, Simone Scardapane, Bartosz Wójcik
ICLR 2026
arXiv / code

Methods built on exact zero activations do not apply to modern LLMs with SiLU or GELU, so we introduce a framework for measuring how much activation sparsity a model tolerates in its FFN layers and run it across model families and scales. The potential for sparsity grows with model size, and we include the first study of activation sparsity in diffusion-based LLMs.

On Stealing GNN Models teaser On Stealing Graph Neural Network Models
Marcin Podhajski, Jan Dubiński, Franziska Boenisch, Adam Dziedzic, Agnieszka Pręgowska, Tomasz P. Michalak
AAAI 2026
arXiv / code

Existing GNN stealing attacks assume unlimited queries to the victim. We show an adversary can extract a GNN with a small fixed budget: first obtain the model backbone without querying the victim at all, then spend the budget on the most informative nodes. The attack works on eight real-world datasets, also with active defenses in place.

ExpertSim teaser ExpertSim: Fast Particle Detector Simulation Using Mixture-of-Generative-Experts
Patryk Będkowski, Jan Dubiński, Filip Szatkowski, Kamil Deja, Przemysław Rokita, Tomasz Trzciński
ECAI 2025
arXiv / code

A Mixture-of-Generative-Experts model for simulating the Zero Degree Calorimeter in the ALICE experiment at CERN, where each expert handles a different subset of the data. ExpertSim is more accurate than a single generator and much faster than the Monte Carlo simulation it replaces.

Privacy Attacks on Image Autoregressive Models teaser Privacy Attacks on Image Autoregressive Models
Antoni Kowalczuk*, Jan Dubiński*, Franziska Boenisch, Adam Dziedzic
ICML 2025
arXiv / code / project page

The first privacy analysis of image autoregressive models. Our membership inference attack reaches 86% true positive rate at 1% false positive rate (6% for comparable diffusion models), dataset inference needs as few as 6 samples, and we extract hundreds of training images from VAR-d30.

CDI teaser CDI: Copyrighted Data Identification in Diffusion Models
Jan Dubiński*, Antoni Kowalczuk*, Franziska Boenisch, Adam Dziedzic
CVPR 2025
arXiv / code

Membership inference on single images is not reliable for large diffusion models, so CDI aggregates the signal over a data owner's whole collection and applies a statistical test. With 70 images, an owner can tell with over 99% confidence whether their data was used to train a given model.

Efficient Model-Stealing Attacks Against Inductive GNNs teaser Efficient Model-Stealing Attacks Against Inductive Graph Neural Networks
Marcin Podhajski, Jan Dubiński, Franziska Boenisch, Adam Dziedzic, Agnieszka Pręgowska, Tomasz P. Michalak
ECAI 2024
arXiv / code

An unsupervised model-stealing attack against inductive GNNs based on graph contrastive learning and spectral graph augmentations. The stolen models have higher fidelity and downstream accuracy than the prior state of the art on all six datasets, while sending fewer queries to the victim.

Membership inference on diffusion models teaser Towards More Realistic Membership Inference Attacks on Large Diffusion Models
Jan Dubiński, Antoni Kowalczuk, Stanisław Pawlak, Przemysław Rokita, Tomasz Trzciński, Paweł Morawiecki
WACV 2024
arXiv / paper

A fair evaluation setup for membership inference on Stable Diffusion. Earlier setups overestimate how well these attacks work; under realistic conditions membership inference remains hard for large diffusion models deployed as black boxes, so the related privacy and copyright problems are not going away soon.

Bucks for Buckets teaser Bucks for Buckets (B4B): Active Defenses Against Stealing Encoders
Jan Dubiński*, Stanisław Pawlak*, Franziska Boenisch*, Tomasz Trzciński, Adam Dziedzic
NeurIPS 2023
arXiv / code / paper

The first active defense against stealing encoders exposed through MLaaS APIs. B4B lowers the utility of returned representations as a user's coverage of the embedding space grows (legitimate users cover far less than attackers), and transforms each user's representations individually so that sybil accounts cannot be pooled into a stolen copy.

Website template from Jon Barron.