CS Fundamentals Trainer · Guides
Learned perceptual metrics: LPIPS, perceptual losses, style transfer and DeepDream
Why PSNR and SSIM mislead (with numbers you can reproduce) and how deep features fix it: the LPIPS distance and the BAPPS human judgements behind it, using LPIPS and pyiqa and choosing between metrics, feature (perceptual) losses, neural style transfer and the Gram matrix as a distribution match, DeepDream as gradient ascent on the image, the perception-distortion trade-off, and the traps of learned metrics.
30 min read
PSNR and SSIM compare two images number by number, or window by window. They are cheap, they are well understood, and they are often wrong about what a person sees: a picture shifted by one pixel looks identical and scores badly, while a smeared picture and a noisy picture with the same error score the same. Since about 2016 a different idea has taken over wherever images are compared by a program: run both images through a pretrained neural network, and compare what the network sees instead of what the pixels are. That one idea gives you a perceptual metric (LPIPS), the training losses behind most super-resolution and image-to-image models (perceptual losses), neural style transfer, and DeepDream.
This guide connects those four things, because they are the same tool used for four jobs. It explains why pixel metrics fail with numbers you can reproduce, how LPIPS turns network features into a distance and how it was validated against human judgements, how to use it and its neighbours in a toolbox such as pyiqa, why Gram matrices capture "style", what DeepDream does differently, and where learned metrics themselves mislead. Every claim is tied to its source, and the small experiments are in the appendix.
Read this first: LPIPS is a distance between two images, not a quality score. It answers "how different do these two pictures look to a trained network?", it needs a reference, and it inherits the biases of its backbone and of the data it was fitted to. A low LPIPS means "similar to the reference", which is no compliment if the reference is poor, and a metric tuned to human judgements of 64×64 patches is not guaranteed to agree with people on your kind of image (section 6).
What was reproduced and what was reported. | Kind | What it covers | Status | |---|---|---| | Reproduced | The PSNR and SSIM table of section 1; the Gram-matrix shuffle invariance and the MMD identity of section 5 | Run for this guide (NumPy, SciPy, scikit-image, Pillow; the 512×512
astronauttest image in 8-bit grey). The programs are in the appendix | | Reported | Every LPIPS, BAPPS, DISTS, MUSIQ, CLIP-IQA, style-transfer, Johnson and perception–distortion figure or claim | Quoted from the sources in the list at the end, not re-measured: the pretrained weights could not be downloaded where the guide was written | | Provided, not run | An optional script that adds an LPIPS column to the section 1 table, and an exercise on style-transfer weights | In the appendix, with suggested package pins; treat it as untested until you run it |
1. Why pixel metrics are not enough
Take a grey photograph and damage it in four ways. The first three are tuned to the same mean squared error (MSE), so PSNR cannot tell them apart. Higher PSNR and SSIM mean "more similar".
| Distortion | MSE | PSNR (dB) | SSIM |
|---|---|---|---|
| Shifted right by 1 pixel | 263.6 | 23.92 | 0.857 |
| Gaussian blur, σ = 2.46 | 264.8 | 23.90 | 0.786 |
| Added Gaussian noise, σ = 17.2 | 265.9 | 23.88 | 0.469 |
| Brightness raised by 10 levels (invisible to most viewers) | 99.0 | 28.17 | 0.880 |
| JPEG, quality 10 (visibly blocky) | 83.8 | 28.90 | 0.859 |
Three lessons come out of this table.
- Equal error is not equal appearance. The first three rows have the same PSNR to within 0.04 dB, but nobody would call a one-pixel shift as bad as heavy noise. SSIM separates them, because it looks at local structure: the noisy image falls to 0.469 while the shifted one stays at 0.857.
- A harmless change can score worse than a harmful one. Raising the brightness by 10 grey levels costs 28.17 dB, which is lower than the 28.90 dB of a blocky JPEG at quality 10. PSNR sees only the size of the difference, not whether anyone would notice it.
- Pixel metrics assume alignment. PSNR compares position i with position i. A shift by one pixel changes nothing the eye cares about and changes every term of the sum.
The SSIM authors made the same point with a photograph called "Boat". All its distorted versions have an MSE of 210, and yet they look very different: the structural index scores a mean-shifted copy at 0.9900, a contrast-stretched one at 0.9168, salt-and-pepper noise at 0.7748, a blurred copy at 0.7052 and a JPEG at 0.6949. SSIM works on a sliding window (an 11×11 Gaussian with σ = 1.5), multiplies a luminance, a contrast and a structure term computed from the window's means, variances and covariance (with small stabilising constants, K₁ = 0.01 and K₂ = 0.03), and averages the local values into the mean SSIM. A uniform brightness change keeps the local structure, which is why it barely lowers the score, in line with the third lesson.
This is the motivation behind LPIPS: PSNR and SSIM are described as "simple, shallow functions" that miss many aspects of human perception. The standard remedy before deep learning was to model the visual system by hand, as VIF and FSIM do. The finding behind LPIPS is that a network trained for something else, such as classifying ImageNet, already contains a better model of perceptual similarity than those hand-built ones.
2. From features to a distance: LPIPS
What a "deep feature" is
A convolutional network turns an image into a stack of feature maps, one set per layer. Early layers respond to edges, colours and small textures; deeper layers respond to larger patterns such as grids, fur, wheels or faces, and cover a larger part of the image (a bigger receptive field). The set of maps at one layer, with shape C channels × H × W positions, is the image's description at that level of abstraction. Two images that are close at a deep layer have the same kind of content and texture, even when their pixels differ.
The LPIPS distance
LPIPS, the Learned Perceptual Image Patch Similarity, is computed in five steps:
- Pass the reference x and the test image x₀ through the same network and collect the activations of several layers.
- At each layer, unit-normalise across the channels at every position.
- Scale each channel by a learned weight w.
- Take the squared L2 difference and average it over all spatial positions.
- Sum over the layers.
Diagram: How LPIPS turns two images into one number
With all weights equal to 1 the formula reduces, at each position, to the squared Euclidean distance between two unit-length vectors, which equals 2(1 − cos θ): proportional to the cosine distance, though not numerically identical to it. The learned weights are the "L" in LPIPS: they say which channels matter for human judgement.
The three ways to obtain the weights
| Variant | What is trained | Starting point |
|---|---|---|
| lin | Only a set of non-negative per-channel weights; the network stays frozen (1,472 parameters for VGG) | A pretrained classifier |
| tune | All the weights, fine-tuned on the human judgements | A pretrained classifier |
| scratch | All the weights, from nothing | Random Gaussian initialisation |
"lin" is the one most people mean by LPIPS. It is a calibration of an existing feature space, not a new network, which is why the metric is cheap to train and why it comes in backbones: AlexNet, VGG and SqueezeNet.
The data: BAPPS and how it was judged
The weights were fitted to, and the metric tested on, the Berkeley-Adobe Perceptual Patch Similarity (BAPPS) dataset: 484,000 human judgements over 161,000 patches of 64×64 pixels. The patches were distorted in three ways: traditional distortions (20 basic ones and 308 sequential combinations: photometric changes, noise, blur, warps, JPEG), CNN-based distortions (96 denoising autoencoders trained on ImageNet, whose artefacts resemble those of real restoration networks), and the real outputs of super-resolution, frame interpolation, deblurring and colourisation algorithms.
Two kinds of test were used:
- 2AFC (two-alternative forced choice). The participant sees a reference and two distorted versions, and picks the one that is closer to the reference. A metric scores a point each time it agrees with the majority.
- JND (just-noticeable difference). The reference and a distorted patch are shown for one second each with a 250 ms gap, and the participant says "same" or "different". In the validation sets, 36.4% of the pairs were judged "same".
What the results show
Average 2AFC agreement over all six test sets of the paper (higher is better):
| Method | Agreement with humans |
|---|---|
| Human raters (the ceiling) | 73.9% |
| L2 (the pixel error behind PSNR) | 63.2% |
| SSIM | 63.1% |
| FSIM | 63.8% |
| VGG features, used as they are | 67.0% |
| AlexNet-lin | 69.8% |
| VGG-lin | 69.2% |
| SqueezeNet-lin | 70.0% |
Read the table carefully, because it is easy to over-read. The human ceiling is 73.9%, not 100%, because people disagree with each other on hard pairs, so the useful comparison is the gap that is closed: pixel metrics reach about 63%, learned metrics about 70%, humans 74%. Also, the classic metrics are not equally poor everywhere. The paper reports L2, SSIM and FSIM at 77.8%, 79.1% and 78.6% on the CNN-based distortions but only 59.9%, 60.3% and 61.4% on the traditional ones. Their weakness is concentrated in distortions like blur and geometric changes.
Two further results explain the paper's title. First, features from self-supervised and unsupervised networks (trained without class labels) also beat the classic metrics, and so did networks trained for other tasks and with other architectures. Second, a randomly initialised network does better than L2 but much worse than a trained one (64.3% against about 68 to 70%), so some training signal matters. The authors conclude that perceptual similarity is "an emergent property shared across deep visual representations".
3. Using these metrics in practice
LPIPS, one pair at a time
The reference implementation is the lpips package. Its documentation (PyPI page) asks for RGB tensors of shape N×3×H×W normalised to [−1, 1], and says that "higher means further/more different, lower means more similar". It describes net='alex' as the default, the fastest and the best as a forward metric, and net='vgg' as "closer to the traditional perceptual loss", the one to choose when the distance is backpropagated as a loss.
import lpips, torch
loss_fn = lpips.LPIPS(net='alex') # downloads the AlexNet weights and the linear layer on first use
d = loss_fn(img_ref, img_test) # tensors N x 3 x H x W in [-1, 1]
print(d.item()) # lower = more similar; 0 = identical
The snippet follows the package documentation and was not run for this guide (the environment could not download the weights). Check it against the version you install.
Many metrics behind one interface: pyiqa
The pyiqa toolbox "is an image quality assessment toolbox built with Python and PyTorch", offering full-reference, no-reference and task-specific metrics, with implementations calibrated against the official MATLAB or Python ones where they exist. Its architecture modules include lpips_arch, dists_arch, niqe_arch, brisque_arch, musiq_arch and clipiqa_arch. The practical value is that a benchmark script can loop over metric names, and report each metric's direction: in some metrics a lower number is better (LPIPS, NIQE, BRISQUE), in others higher (PSNR, SSIM, MUSIQ). The toolbox exposes this for every metric, so read it instead of assuming.
Choosing the metric for the question
| Question | Reasonable metric | Needs a reference? | Notes |
|---|---|---|---|
| Did this codec or filter keep the signal? | PSNR, SSIM | Yes | Cheap and well understood; misleads on shifts, blur and noise (section 1) |
| Does this restored image look like the original? | LPIPS, DISTS | Yes | LPIPS is more sensitive to structure; DISTS explicitly tolerates texture resampling |
| Which GAN or diffusion output looks more natural? | LPIPS for fidelity, plus a no-reference or human test | Mixed | Fidelity and realism pull in different directions (section 6) |
| How good is this photo with no original? | NIQE, BRISQUE, MUSIQ, CLIP-IQA | No | Trained on natural-image statistics or opinion data; check the domain of the training set |
| What do people actually say? | A subjective test (MOS, pairwise comparison) | Optional | The reference for everything else; report SROCC and PLCC against it (see the paragraph below) |
Neighbours worth knowing: DISTS, MUSIQ and CLIP-IQA
DISTS (Deep Image Structure and Texture Similarity) is a full-reference index built to fix one weakness of LPIPS and SSIM: they are "overly sensitive to resampling of texture regions". A photograph of grass in which every blade has been redrawn is the same texture as the original, but it is a different arrangement of pixels, so pixel-wise measures call it worse than a lightly compressed JPEG of the original, against what people say. DISTS uses a fixed VGG16 (with max pooling replaced by blurred ℓ2 pooling to reduce aliasing) and compares two things at every stage: the spatial means of the feature maps (the texture term) and their spatial correlations (the structure term, in the style of SSIM). On the BAPPS patches it scores 0.689 against 0.692 for LPIPS, although it was not trained on them; on the TID2013 quality database its PLCC is 0.855 against 0.749; on a texture-quality database (SynTEX) its SRCC is 0.923 against 0.663. The authors also note its limits: it is highly non-convex, which makes it harder to use as a training loss, and it gives one global score and no quality map.
MUSIQ is a no-reference transformer model that reads an image at its native resolution and at several scales, with a hash-based positional embedding, instead of resizing or cropping it first. That matters because resizing is itself a quality change: a CNN that needs a fixed input size measures the picture after it has been damaged.
CLIP-IQA is a no-reference method that needs no quality labels at all. It asks a pretrained CLIP model how close the image is to a pair of opposite prompts, such as "Good photo." and "Bad photo.", and turns the two similarities into a score between 0 and 1 with a softmax. The same trick works for other qualities ("bright" and "dark", "noisy" and "clean", "sharp" and "blurry"). The authors report that the result is sensitive to the wording ("Good/Bad" worked better than "High quality/Low quality"), that rare professional vocabulary is hard for CLIP, and that a gap remains against models trained for the task.
Validate against people, not against another metric. When a new or learned metric is tried on your data, collect opinion scores for a sample (a subjective test run along the lines of ITU-R BT.500), then report the SROCC (do the ranks agree?) and PLCC (is the relation linear after a monotonic fit?) between the metric and the MOS. This is also how the papers above report their results.
4. From a metric to a loss: perceptual losses
Anything that is differentiable and measures image distance can be minimised. Feed-forward networks can be trained for style transfer and super-resolution using perceptual losses defined on the features of a pretrained network, instead of per-pixel losses. The reported result is that a trained style-transfer network gives results qualitatively similar to the optimisation-based method of Gatys et al., while being "three orders of magnitude faster", and that for super-resolution a perceptual loss gives visually more pleasing results than a per-pixel one.
The reason is the same as in section 1. A per-pixel loss (MSE) is minimised by the average of all plausible answers, which is blurry: if a sharp edge could be one pixel to the left or to the right, the average of the two is a smeared edge. A feature loss does not penalise the exact position of a texture detail as heavily, so the network is free to produce a sharp edge in one place.
LPIPS is used the same way: as a training term, as a model-selection score, or both. This is also its main risk, covered in section 6.
5. Neural style transfer and the Gram matrix
The procedure
The optimisation-based version of neural style transfer starts from an image (the content image or noise) and changes its pixels, not the network, until two losses are small. The method behind it uses the convolutional layers of VGG-19 (the original replaces max pooling with average pooling, which "improves the gradient flow"). Content is the feature response of a higher layer (conv4_2). Style is the set of Gram matrices of five layers (conv1_1 to conv5_1, each weighted 1/5), a multi-scale, texture-like description that leaves out the global arrangement. The image is made by gradient descent on a white-noise image to minimise α·(content loss) + β·(style loss), and the ratio α/β is the dial between the two: a strong style weight gives a texturised painting with little visible photo, a strong content weight keeps the photograph recognisable but matches the style less well. The authors' central finding is that "the representations of content and style in the Convolutional Neural Network are separable", though not completely: a perfect match to both at once is usually impossible. The PyTorch tutorial implements the same idea with a few differences, as follows:
- A pretrained VGG-19 (only its convolutional
features), in evaluation mode with frozen parameters. - Content loss at one layer (
conv_4): the mean squared error between the feature maps of the input and of the content image. - Style loss at five layers (
conv_1toconv_5): the squared difference between Gram matrices of the input and of the style image. - L-BFGS as the optimiser, applied to the input image's pixels, with the pixels clamped to [0, 1] at each step.
- Weights
style_weight = 1,000,000andcontent_weight = 1, 300 steps by default, which took about 12.8 seconds in the tutorial's run. The VGG weights are about 548 MB.
Treat these numbers as settings of one implementation, not defaults of the method. The original formulation uses a different loss normalisation and expresses the balance as a ratio α/β of 10⁻³ or 10⁻⁴, so the tutorial's 10⁶ cannot be carried over to another code base, and the layers (conv_4 against conv4_2) also differ. Changing the weights changes the result in the direction the original paper describes (more style weight, more texture and less visible photograph), so the useful exercise is to repeat a run at several weights and look (a suggested sweep is in the appendix; it was not run for this guide).
What a Gram matrix is, and what it throws away
Reshape the feature maps of a layer into a matrix F with one row per channel (K rows) and one column per position (N columns). The Gram matrix is G = F Fᵀ, a K×K table of how much each pair of channels is active together. The tutorial then normalises it: "the gram matrix must be normalized by dividing each element by the total number of elements in the matrix", so that large layers do not dominate the loss.
The important property is what G does not contain: where things are. The sum over positions is the same in any order. This is checked directly in the appendix: shuffling the columns of F changes the feature maps completely, and leaves G exactly unchanged (np.allclose returns True). So matching Gram matrices matches the co-occurrence statistics of features, which is texture and colour palette, without matching their layout. That is a working definition of "style" as opposed to "content", and the reason the content loss is taken on the feature maps themselves (which keep layout) and the style loss on their Gram matrices.
Why this works, in one result
There is a precise reason why Gram matrices capture style. It can be shown that matching Gram matrices is mathematically equivalent to minimising the Maximum Mean Discrepancy (MMD) with a second-order polynomial kernel, so style transfer is, in effect, aligning the distribution of features of the generated image with that of the style image. This is supported by good results with other distribution-alignment methods too.
The equivalence is easy to check numerically. For two sets of feature vectors, the MMD with kernel k(x, y) = (x·y)² equals the squared Frobenius distance between their (mean-normalised) second-moment matrices. The appendix computes both on random data and prints 4.367194382521898 and 4.367194382521896: equal to rounding error.
The fast version
The tutorial's method runs an optimisation for every new pair of images. A feed-forward network can be trained once per style, using the same content and style losses as the training signal, so that stylising a new image is a single forward pass. This is the "three orders of magnitude" of section 4.
6. Where learned metrics mislead
- They are only as good as their training distribution. LPIPS was fitted to 64×64 patches and to the distortions of BAPPS. On other content (medical images, line art, very large images, a new kind of artefact) its agreement with people has to be re-checked, not assumed.
- Fidelity and realism are different goals, and they trade off. It can be shown that "distortion and perceptual quality are at odds with each other": as the average distortion against the ground truth falls, the best possible perceptual quality must get worse, for any distortion measure. The effect is weaker for some measures, such as distances between VGG features. So a method that wins on PSNR is not the one that looks most real, and a GAN that looks real will lose on PSNR. Report both, and say which one you optimised.
Read the scope of this result carefully. It describes the best achievable pairs of (distortion, perceptual quality), where perceptual quality is defined by how well an optimal discriminator can tell restored images from real ones. It is a statement about a limit, not a guarantee about any individual model or benchmark: a real model can sit far from the bound, the strength of the conflict depends on the distortion measure used, and human opinion scores are a different operationalisation of "perceptual quality" from the discriminator-based one.
- A metric used as a loss gets exploited. Once a network is trained to minimise LPIPS, a low LPIPS no longer measures anything independent. Use a held-out metric or a human test to evaluate it, and keep a different metric for training than for the final score.
- A distance is not a quality score. LPIPS compares two images. It says "these look alike", not "this one is good". If the reference is itself poor, a low LPIPS is no compliment.
- Numbers across tools are not comparable. The same metric name can differ between implementations in the colour range (0–1 or −1–1), the backbone and the resizing. State all three when you publish a result.
7. DeepDream: the same machinery with a different objective
In style transfer, the loss compares an image with a target image. DeepDream asks a different question: what would make this layer respond more? In June 2015 Google published the Inceptionism post, and in July the DeepDream code example. The network is the one built for the 2014 ImageNet challenge.
Google's own account, the Inceptionism post, gives the clearest intuition. Asking a layer to "amplify whatever you see there" creates a feedback loop, and what appears depends on the layer. Lower layers respond to edges and orientations and produce strokes and simple ornament-like patterns; intermediate layers produce shapes and components such as doors or leaves; higher layers produce whole objects such as birds, buildings or towers. The picture is also biased by the training data and by the input: a network trained mostly on animals sees animals in random shapes, horizon lines turn into towers, and leaves attract birds and insects. Feeding the output back in, with a zoom after each pass, gives an endless stream of new images. Starting from pure noise works poorly unless the image is constrained to have natural-image statistics, such as correlated neighbouring pixels. The authors present the technique mainly as a way to see what a network has learned, to check whether training captured the right features, and possibly to improve architectures, with art as a side use.
The weights are fixed, as in style transfer. What changes is the input image: it is adjusted so that a chosen output neuron gets a higher confidence score, a procedure that resembles backpropagation with the roles swapped (the image is updated, not the weights). As Google's post describes it, you pick a layer and ask the network to amplify whatever that layer detects, which is gradient ascent on the image. Optimising an image from noise works poorly on its own and works much better when the image is constrained to have natural-image statistics, such as correlated neighbouring pixels. Applied to an ordinary photo, the network amplifies whatever it half-recognises there, which is why swirls of eyes, fur and architecture appear: it is a form of pareidolia, the same effect as seeing faces in clouds.
| Neural style transfer | DeepDream | |
|---|---|---|
| What changes | The image's pixels | The image's pixels |
| What stays fixed | The network | The network |
| Objective | Match content features and style (Gram) statistics of two target images | Increase the activation of a chosen neuron or layer |
| Needs a target image? | Yes, two | No |
| Main use | Image synthesis, art | Visualising what a layer has learned; art |
Both are instances of one technique: take the gradient of a feature-space objective with respect to the image. LPIPS-as-a-loss is the third instance. If you understand the three of them, you understand how adversarial examples are made too, since they use the same gradient with a different objective.
8. Which tool for which question
| You want to… | Use | Because |
|---|---|---|
| Check that a lossless or near-lossless step kept the pixels | MSE or PSNR | There is no perceptual question to answer |
| Compare codecs at the same bit rate | SSIM or VMAF, plus a small subjective test | Structure matters more than pixel error |
| Rank restoration or super-resolution outputs against a ground truth | LPIPS (or DISTS), reported with PSNR | It follows human similarity judgements better, and PSNR shows the fidelity side of the trade-off |
| Train a restoration or style network | A feature loss (VGG-based), possibly with an adversarial term | Per-pixel losses give blurry outputs |
| Score photos with no reference | NIQE, BRISQUE, MUSIQ, CLIP-IQA | They need only the test image |
| Understand what a layer has learned | DeepDream-style activation maximisation | It shows what excites a layer |
| Produce a stylised image from two photographs | The Gram-matrix method (slow) or a trained feed-forward network (fast) | Optimise per pair or once per style |
| Decide whether a metric can be trusted on your images | Collect MOS on a sample and report SROCC and PLCC | Only people define perceptual quality |
9. Common mistakes
- Reporting PSNR alone for a generative model. PSNR rewards the blurry average. Report a perceptual metric and a no-reference or human result next to it.
- Evaluating with the metric you trained on. An LPIPS-trained network will look good on LPIPS. Hold out another measure.
- Mixing value ranges. Feeding [0, 255] or [0, 1] images to a metric that expects [−1, 1] gives numbers that look plausible and mean nothing.
- Forgetting the direction. LPIPS, NIQE and BRISQUE are lower-is-better; PSNR and SSIM are higher-is-better. Check each metric's direction before ranking.
- Calling LPIPS a no-reference metric. It needs the reference image.
- Expecting a style transfer result to preserve geometry. The Gram matrix discards layout, so only the content loss holds the structure in place; a style weight that is too high (the tutorial uses 10⁶ against 1) will overwhelm it.
- Comparing numbers from different tools. See section 6.
10. Things to mention in an interview
- Why PSNR and SSIM fail on a one-pixel shift, and what a perceptual metric does about it.
- The five steps of LPIPS, and that "lin" learns only channel weights on a frozen network.
- What BAPPS is, and the difference between 2AFC and JND tests.
- That the human 2AFC ceiling in the paper is 73.9%, with pixel metrics near 63% and learned metrics near 70%.
- Why a Gram matrix captures texture and not layout, and the MMD equivalence of Li et al.
- The perception–distortion trade-off, and why you report both sides.
- That style transfer, DeepDream and adversarial examples all optimise the image against a fixed network.
- How you would validate a metric on a new kind of image: collect MOS, report SROCC and PLCC.
Quick recap
- PSNR and SSIM compare positions; LPIPS compares features. Equal MSE can hide very different pictures (section 1).
- LPIPS = normalise, weight, difference, average, sum. The weights are learned from 484,000 human judgements on 161,000 patches (BAPPS).
- Learned metrics close most of the gap to humans (about 70% against 63% pixel metrics and 73.9% humans on 2AFC), but they inherit the limits of their training data.
- A feature loss gives sharper results than MSE, because the average of plausible answers is blurry.
- A Gram matrix keeps which features co-occur and drops where. Matching it is MMD with a second-order polynomial kernel.
- DeepDream is gradient ascent on the image against a fixed network.
- Distortion and perception trade off. Report both.
Appendix: the code
metrics/experiment.py
# Section 1: how PSNR and SSIM score five changes to the same picture.
# pip install numpy scipy scikit-image pillow
import io
import numpy as np
from PIL import Image
from scipy.ndimage import gaussian_filter
from skimage import color, data
from skimage.metrics import mean_squared_error as mse
from skimage.metrics import peak_signal_noise_ratio as psnr
from skimage.metrics import structural_similarity as ssim
img = (color.rgb2gray(data.astronaut()) * 255).round().astype(np.uint8)
f = img.astype(float)
def clip(a):
return np.clip(a.round(), 0, 255).astype(np.uint8)
def report(name, t):
print(f"{name:38s} MSE {mse(img, t):8.1f} PSNR {psnr(img, t, data_range=255):6.2f} "
f"SSIM {ssim(img, t, data_range=255):.3f}")
rng = np.random.default_rng(0)
report("brightness +10", clip(f + 10))
report("shift 1 px right", clip(np.roll(f, 1, axis=1)))
report("shift 2 px right", clip(np.roll(f, 2, axis=1)))
target = mse(img, clip(np.roll(f, 1, axis=1))) # tune blur and noise to the MSE of the 1 px shift
for s in np.arange(0.5, 6, 0.01):
blurred = clip(gaussian_filter(f, s))
if mse(img, blurred) >= target:
break
report(f"blur sigma {s:.2f} (MSE matched)", blurred)
for sd in np.arange(1, 80, 0.1):
noisy = clip(f + rng.normal(0, sd, f.shape))
if mse(img, noisy) >= target:
break
report(f"noise sd {sd:.1f} (MSE matched)", noisy)
for q in (90, 50, 20, 10):
buf = io.BytesIO()
Image.fromarray(img).save(buf, "JPEG", quality=q)
report(f"JPEG quality {q}", np.array(Image.open(buf)))
gram/check.py
# Section 5: a Gram matrix ignores where features are, and equals an MMD with a polynomial kernel.
import numpy as np
rng = np.random.default_rng(1)
F = rng.normal(size=(8, 16 * 16)) # K = 8 channels, N = 256 positions
G = F @ F.T / F.size # normalised as in the PyTorch tutorial
perm = rng.permutation(F.shape[1])
G2 = F[:, perm] @ F[:, perm].T / F.size
print("same Gram after shuffling positions:", np.allclose(G, G2),
"| feature maps equal:", np.allclose(F, F[:, perm]))
X = F.T # N points in R^K
Y = rng.normal(size=(300, 8)) * 1.3
def mmd(X, Y):
k = lambda A, B: (A @ B.T) ** 2 # second-order polynomial kernel
return k(X, X).mean() + k(Y, Y).mean() - 2 * k(X, Y).mean()
Gx, Gy = X.T @ X / len(X), Y.T @ Y / len(Y)
print(mmd(X, Y), ((Gx - Gy) ** 2).sum()) # equal
neural/lpips_table.py (optional, not run)
# Optional and NOT run for this guide: adds an LPIPS column to the distortions of section 1.
# Suggested pins (a matching torch/torchvision pair; adjust to your platform):
# pip install "torch==2.5.1" "torchvision==0.20.1" "lpips==0.1.4" numpy scipy scikit-image pillow
# The first call downloads the AlexNet weights through torchvision; the linear LPIPS weights ship with the package.
import io
import lpips
import numpy as np
import torch
from PIL import Image
from scipy.ndimage import gaussian_filter
from skimage import color, data
img = (color.rgb2gray(data.astronaut()) * 255).round().astype(np.uint8)
f = img.astype(float)
rng = np.random.default_rng(0)
def clip(a):
return np.clip(a.round(), 0, 255).astype(np.uint8)
def jpeg(a, q):
buf = io.BytesIO()
Image.fromarray(a).save(buf, "JPEG", quality=q)
return np.array(Image.open(buf))
variants = {
"brightness +10": clip(f + 10),
"shift 1 px right": clip(np.roll(f, 1, axis=1)),
"blur sigma 2.46": clip(gaussian_filter(f, 2.46)),
"noise sd 17.2": clip(f + rng.normal(0, 17.2, f.shape)),
"JPEG quality 10": jpeg(img, 10),
}
def to_tensor(a): # grey 8-bit -> 1 x 3 x H x W in [-1, 1], as the lpips docs ask
t = torch.from_numpy(a.astype(np.float32) / 127.5 - 1.0)
return t[None, None].repeat(1, 3, 1, 1)
loss_fn = lpips.LPIPS(net="alex").eval() # lower = more similar
with torch.no_grad():
for name, v in variants.items():
print(f"{name:20s} LPIPS(alex) {loss_fn(to_tensor(img), to_tensor(v)).item():.3f}")
# Exercise for the style-transfer sensitivity (not provided as code): in the PyTorch tutorial, repeat the run with
# style_weight in {1e4, 1e5, 1e6, 1e7} and content_weight = 1, and compare the outputs. The original formulation
# uses a different loss normalisation and ratios (alpha/beta of 1e-3 and 1e-4), so the tutorial's numbers do not transfer.
Sources
- Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman and Oliver Wang, The Unreasonable Effectiveness of Deep Features as a Perceptual Metric (CVPR 2018; paper on arXiv): LPIPS, the BAPPS dataset, the 2AFC and JND tests and all the agreement figures quoted here.
- Richard Zhang, lpips package: input range, the
alexandvggbackbones and how to read the distance. - Yanghao Li, Naiyan Wang, Jiaying Liu and Xiaodi Hou, Demystifying Neural Style Transfer (2017): Gram matrix matching as MMD with a second-order polynomial kernel.
- Leon A. Gatys, Alexander S. Ecker and Matthias Bethge, A Neural Algorithm of Artistic Style (2015): the content and style representations, the losses and the α/β trade-off.
- PyTorch tutorials, Neural Transfer Using PyTorch: VGG-19, the content and style layers, the normalised Gram matrix, L-BFGS and the weights.
- Justin Johnson, Alexandre Alahi and Li Fei-Fei, Perceptual Losses for Real-Time Style Transfer and Super-Resolution (ECCV 2016).
- Alexander Mordvintsev, Christopher Olah and Mike Tyka, Inceptionism: Going Deeper into Neural Networks (Google Research Blog, 2015): what each layer produces, training bias and the purpose of the technique.
- Wikipedia, DeepDream: background only (dates, the 2014 ImageNet network and later uses); the technical description above follows the Google post.
- Yochai Blau and Tomer Michaeli, The Perception-Distortion Tradeoff (CVPR 2018).
- Zhou Wang, Alan C. Bovik, Hamid R. Sheikh and Eero P. Simoncelli, Image quality assessment: from error visibility to structural similarity (IEEE Transactions on Image Processing, 2004): the SSIM definition and the equal-MSE "Boat" experiment.
- Keyan Ding, Kede Ma, Shiqi Wang and Eero P. Simoncelli, Image Quality Assessment: Unifying Structure and Texture Similarity (DISTS).
- Junjie Ke et al., MUSIQ: Multi-scale Image Quality Transformer (ICCV 2021); Jianyi Wang, Kelvin C.K. Chan and Chen Change Loy, Exploring CLIP for Assessing the Look and Feel of Images (AAAI 2023).
- pyiqa (IQA-PyTorch) documentation: the toolbox and its metric modules.
- ITU-R, Recommendation BT.500, the standard methodology for subjective picture-quality tests, and the MSU overview of subjective quality methods (DSIS, DSCQS and related methods based on it).
Practise this topic: Computer Vision