The Panda That Became a Gibbon: Adversarial Examples and the Hidden Fragility of Neural Networks

The Hook: A Panda That Is 99.3 Percent Gibbon

The image on the left shows a panda. An image classifier – GoogLeNet, trained on ImageNet – sees it too: "panda", with 57.7 percent confidence. The image on the right shows – to any human eye – exactly the same panda. The network, however, is now very sure it is seeing something else: "gibbon", with 99.3 percent confidence.

Between the two images lies a noise pattern, scaled by a factor of 0.007 and added on top. The authors chose this magnitude because it corresponds to the smallest bit of an 8-bit image encoding after GoogLeNet converts the pixel values to real numbers. No human sees the difference. The network sees a different primate.

This image from the paper "Explaining and Harnessing Adversarial Examples" by Ian Goodfellow, Jonathon Shlens and Christian Szegedy (ICLR 2015) has become the icon of an entire research field. It shows something uncomfortable: a system that performs excellently on standard test data can be driven to wrong – and highly confident – answers by deliberately constructed changes that are invisible to humans. Such inputs are called adversarial examples.

This article explains where they come from, why defending against them is so hard, and what that means for anyone who deploys models in security-relevant settings.

Part 1: The Discovery – "Intriguing Properties"

The first systematic description comes from the paper "Intriguing properties of neural networks" by Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow and Rob Fergus, submitted to arXiv on December 21, 2013. The title is modest; the content was not. The authors reported that a network can be made to misclassify an image by applying a certain imperceptible perturbation, found by maximizing the network's prediction error.

Even more important was a second finding: the perturbations were not a random artifact of learning. The same perturbation could cause a different network, trained on a different subset of the dataset, to misclassify the same input. In the full text the authors are more precise: the examples also fool networks with a different number of layers, different activations or different hyperparameters, and they remain hard even for models trained on a disjoint training set – although their effectiveness decreases considerably. This property is now called transferability. Its practical significance: later work, not covered in detail here, exploited it for attacks via self-trained substitute models, without knowing the target model.

The same paper contains a second, less well-known finding that aids understanding: the authors found no difference between individual high-level neurons and random linear combinations of such neurons. Their conclusion: it is the space of activations, not the individual neuron, that carries the semantic information in the upper layers. Anyone who has read the article on mechanistic interpretability in the vault will recognize an early forerunner of the superposition idea here.

Part 2: The Explanation – Too Linear, Not Too Nonlinear

The obvious initial guess was that adversarial examples result from the strong nonlinearity of deep networks or from overfitting. Goodfellow, Shlens and Szegedy disagreed in 2014: early attempts at explanation had focused on nonlinearity and overfitting; the primary cause of the vulnerability, they argued, is instead the linear nature of the networks.

The intuition can be followed with a simple thought. A linear model computes a dot product w·x. If you change each input feature by at most ε, each time in the direction of the sign of the corresponding weight, the dot product grows by ε times the sum of the absolute weights. That sum grows with the number of dimensions. In an image with hundreds of thousands of pixel values, many tiny, individually invisible changes add up to a large change in the output. High dimensionality plus locally near-linear behavior is therefore enough for vulnerability – no exotic nonlinearity required. Modern networks, the argument goes, are often intentionally designed to behave linearly (for example through ReLU activations) because they are then easier to optimize.

This explanation leads directly to a cheap attack, the Fast Gradient Sign Method (FGSM):

x̃ = x + ε · sign(∇ₓ J(θ, x, y))

Using backpropagation, you compute the gradient of the loss function J not with respect to the weights θ but with respect to the input x, keep only its sign, and take a step of size ε in that direction. A single gradient step suffices. On MNIST with ε = 0.25, a maxout network, according to the paper, misclassified 89.4 percent of the examples generated this way, with an average confidence of 97.6 percent.

Part 3: The First Defense – Adversarial Training

If attacks can be generated cheaply, they can also be built into training. That is exactly what Goodfellow and colleagues proposed: adversarial training. During training, FGSM examples are generated continuously and used, with the correct label, as additional training data. The result in the same paper: the maxout network's error rate on FGSM examples fell from 89.4 to 17.9 percent. As a side effect, the error on the normal test data also dropped slightly – so the method additionally acted as a regularizer.

An important catch: this robustness holds against the attack used in training. 17.9 percent error against FGSM guarantees nothing about how the model will behave against a stronger, iterative attacker.

Part 4: Robustness as a Game Between Two Players

This catch was addressed by Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras and Adrian Vladu in "Towards Deep Learning Models Resistant to Adversarial Attacks" (arXiv, June 2017). They studied adversarial robustness "through the lens of robust optimization". Formally, training becomes a min-max problem:

min_θ E₍ₓ,ᵧ₎ [ max_{δ ∈ S} J(θ, x + δ, y) ]

Here S is the set of allowed perturbations; in the paper's experiments these are perturbations with ‖δ‖∞ ≤ ε (ε = 0.3 for MNIST, ε = 8 for CIFAR-10 on the 0–255 pixel scale).

The inner player – the attacker – searches within S for the most damaging perturbation. The outer player – training – chooses weights that perform as well as possible even against this worst case. According to the abstract, this formulation yields a concrete security guarantee that would protect against any adversary, provided the inner problem could be solved. Because that generally cannot be done exactly, the authors propose security against a first-order adversary – that is, against any attacker that uses only gradient information – as a practical target, and call it a natural and broad security guarantee.

The value of this work lies less in a new trick than in conceptual clarity: robustness is relative to an explicitly defined threat model (which perturbations are allowed, how large ε is, what the attacker knows). Without that model, "robust" is not a testable statement.

Part 5: Defenses That Only Appear to Work

How important that clarity is became evident in 2018 with a paper that sobered the field: "Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples" by Anish Athalye, Nicholas Carlini and David Wagner (ICML 2018).

The authors identified obfuscated gradients, a form of gradient masking, as a phenomenon that creates a false sense of security in defenses against adversarial examples. A defense that, for instance, adds a non-differentiable preprocessing step or random transformations makes gradient-based attacks come up empty – not because the model is robust, but because the attacker no longer gets a usable gradient.

In a case study, they examined the non-certified defenses at ICLR 2018 that claimed white-box security. Result: 7 of 9 defenses relied on obfuscated gradients. With adapted attacks, the authors circumvented 6 of them completely and one partially – each in the threat model that the respective paper itself had assumed.

The paper distinguishes three types and names a countermeasure for each:

The lesson is methodological: Athalye et al. explicitly call for adaptive attacks, in which the attacker knows the defense and is restricted only by the threat model. Only then has a defense been evaluated. I am of the opinion that one can recognize in this – as an analogy, not as the authors' wording – Kerckhoffs's principle from cryptography: security must not depend on the attacker not knowing the method.

Part 6: From the Lab to the Street

Is all of this a laboratory problem, since real cameras, viewing angles and lighting destroy tiny pixel perturbations anyway? Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno and Dawn Song investigated this question in "Robust Physical-World Attacks on Deep Learning Models" (arXiv 2017). Their algorithm Robust Physical Perturbations (RP2) generates perturbations that remain effective under varying physical conditions.

The best-known experiment: with a perturbation consisting only of black and white stickers on a real stop sign, they achieved (in the "abstract art" variant against their classifier LISA-CNN, target class "Speed Limit 45") targeted misclassification in 100 percent of the images taken in the lab and in 84.8 percent of the video frames captured from a moving vehicle – in each case for the target classifier studied.

Two caveats belong to an honest reading: this concerns a specific classifier in a specific experimental setup, not a production vehicle with sensor fusion. And the perturbation is visible to humans – the authors deliberately designed it to resemble graffiti, a common form of vandalism, so that it does not look like an attack. What has been shown is that the phenomenon can cross the boundary from pixels to the physical world – not that real vehicles are fooled this way.

Part 7: Bug or Feature?

Perhaps the deepest reinterpretation came in 2019 from Andrew Ilyas, Shibani Santurkar, Dimitris Tsipras, Logan Engstrom, Brandon Tran and Aleksander Madry, under the programmatic title "Adversarial Examples Are Not Bugs, They Are Features". Their thesis: adversarial examples can be attributed directly to non-robust features – patterns in the data distribution that are highly predictive, yet brittle and incomprehensible to humans.

From this point of view, the model is not making a mistake. It uses patterns that really are associated with the label in the data – they are just not the patterns we consider relevant. A human recognizes the panda by its shape and fur markings; the network additionally uses subtle statistical textures that correlate just as reliably with "panda" but can be flipped with tiny changes. In a simplified model, the authors rigorously trace the phenomenon to a misalignment between the (human-specified) notion of robustness and the inherent geometry of the data.

This interpretation has an unpleasant consequence: if the vulnerability arises from useful features, it is not a programming bug that can simply be fixed. Robustness requires deliberately training the model out of using certain predictive features. The paper has not gone unchallenged, however: in a discussion in the journal Distill (August 2019), the editors collected six comments; one of them, by Preetum Nakkiran, is titled "Adversarial Examples are Just Bugs, Too" and shows examples that arise from bugs rather than features. The original authors responded there themselves. The features thesis is therefore an influential interpretation, not a settled consensus.

Part 8: A Framework for Practice

Fact Check

47 claims in the draft were checked against the primary sources (arXiv abstract and full-text pages, PMLR, Distill): 22 hard facts, 8 quotes or close paraphrases, 7 source attributions and 10 synthesis claims. The check was carried out by a separate reviewer agent tasked with finding errors. Result: 0 FALSE, 7 TO BE WEAKENED or made more precise, 0 NOT VERIFIABLE – a total of 7 corrections.

The most important corrections:

1. Wrong attribution on transfer. The draft derived the attack via a self-trained substitute model from Szegedy et al.; however, they only report transferability itself – and with considerably reduced effectiveness for disjoint training data. Substitute-model attacks come from later work.
2. The table on obfuscated gradients was imprecise. It now follows the classification of Athalye et al. (shattered, stochastic, exploding/vanishing gradients) with the countermeasures named there: BPDA, EOT and reparameterization.
3. Kerckhoffs's principle marked as the author's own analogy. Athalye et al. do not mention it; they call for "adaptive attacks". In addition, the claim that adversarial training is computationally expensive was tied to the attack steps actually stated in the Madry paper.

Note: Automated AI check against primary sources; no human expert review.

The Central Takeaway

Adversarial examples show that high accuracy on test data and robustness against an adversary are two different properties. Test accuracy measures how well a model handles inputs that follow the distribution of the training data. An attacker, however, deliberately chooses inputs that do exactly the opposite.

The practical prompt: Treat every ML model whose inputs a third party can influence as an attack surface. Define a threat model, test with attacks that know the defense, and never rely on a robustness claim that does not say which attacker it holds against.

I am of the opinion that the most important lesson reaches beyond image classification: a model can be right for reasons that are not our reasons – and precisely where its reasons and ours diverge lies the attack surface.

Something to Think About

Which system in your area of responsibility makes decisions based on inputs that a motivated third party can shape – and have you ever measured its reliability against an adversary rather than only against the average case?

Cross-References in the Vault

• The Error That Learns Backward: Backpropagation, the Chain Rule, and the Algorithm Behind Deep Learning – the same gradient that trains the network gives the attacker their direction.
• The Ghost in the Machine: How to Read a Neural Network From the Inside – features in activation space rather than in individual neurons.
• More Parameters, Fewer Errors: Double Descent, Grokking, and the Riddle of Generalization – why good generalization on test data is not the same as robustness.
• Bits That Flip on Their Own: Rowhammer and the Physical Weakness of Computer Memory – another attack surface that arises from a property rather than a programming bug.

Sources and Further Reading

1. Szegedy, C., Zaremba, W., Sutskever, I., Bruna, J., Erhan, D., Goodfellow, I., Fergus, R.: Intriguing properties of neural networks. arXiv:1312.6199 (v1 December 21, 2013). https://arxiv.org/abs/1312.6199
2. Goodfellow, I. J., Shlens, J., Szegedy, C.: Explaining and Harnessing Adversarial Examples. ICLR 2015. arXiv:1412.6572. https://arxiv.org/abs/1412.6572
3. Madry, A., Makelov, A., Schmidt, L., Tsipras, D., Vladu, A.: Towards Deep Learning Models Resistant to Adversarial Attacks. arXiv:1706.06083 (v1 June 19, 2017). https://arxiv.org/abs/1706.06083
4. Athalye, A., Carlini, N., Wagner, D.: Obfuscated Gradients Give a False Sense of Security: Circumventing Defenses to Adversarial Examples. Proceedings of the 35th ICML, PMLR 80:274–283, 2018. https://proceedings.mlr.press/v80/athalye18a.html
5. Eykholt, K., Evtimov, I., Fernandes, E., Li, B., Rahmati, A., Xiao, C., Prakash, A., Kohno, T., Song, D.: Robust Physical-World Attacks on Deep Learning Models. arXiv:1707.08945 (v1 July 27, 2017). https://arxiv.org/abs/1707.08945
6. Ilyas, A., Santurkar, S., Tsipras, D., Engstrom, L., Tran, B., Madry, A.: Adversarial Examples Are Not Bugs, They Are Features. arXiv:1905.02175 (v1 May 6, 2019). https://arxiv.org/abs/1905.02175
7. Distill editors: A Discussion of "Adversarial Examples Are Not Bugs, They Are Features". Distill, August 6, 2019. DOI 10.23915/distill.00019. https://distill.pub/2019/advex-bugs-discussion/

← All articles