CS Fundamentals Trainer · Guides

Learned perceptual metrics: LPIPS, perceptual losses, style transfer and DeepDream

Why PSNR and SSIM mislead (with numbers you can reproduce) and how deep features fix it: the LPIPS distance and the BAPPS human judgements behind it, using LPIPS and pyiqa and choosing between metrics, feature (perceptual) losses, neural style transfer and the Gram matrix as a distribution match, DeepDream as gradient ascent on the image, the perception-distortion trade-off, and the traps of learned metrics.

30 min read

PSNR and SSIM compare two images number by number, or window by window. They are cheap, they are well understood, and they are often wrong about what a person sees: a picture shifted by one pixel looks identical and scores badly, while a smeared picture and a noisy picture with the same error score the same. Since about 2016 a different idea has taken over wherever images are compared by a program: run both images through a pretrained neural network, and compare what the network sees instead of what the pixels are. That one idea gives you a perceptual metric (LPIPS), the training losses behind most super-resolution and image-to-image models (perceptual losses), neural style transfer, and DeepDream.

This guide connects those four things, because they are the same tool used for four jobs. It explains why pixel metrics fail with numbers you can reproduce, how LPIPS turns network features into a distance and how it was validated against human judgements, how to use it and its neighbours in a toolbox such as pyiqa, why Gram matrices capture "style", what DeepDream does differently, and where learned metrics themselves mislead. Every claim is tied to its source, and the small experiments are in the appendix.

Read this first: LPIPS is a distance between two images, not a quality score. It answers "how different do these two pictures look to a trained network?", it needs a reference, and it inherits the biases of its backbone and of the data it was fitted to. A low LPIPS means "similar to the reference", which is no compliment if the reference is poor, and a metric tuned to human judgements of 64×64 patches is not guaranteed to agree with people on your kind of image (section 6).

What was reproduced and what was reported. | Kind | What it covers | Status | |---|---|---| | Reproduced | The PSNR and SSIM table of section 1; the Gram-matrix shuffle invariance and the MMD identity of section 5 | Run for this guide (NumPy, SciPy, scikit-image, Pillow; the 512×512 astronaut test image in 8-bit grey). The programs are in the appendix | | Reported | Every LPIPS, BAPPS, DISTS, MUSIQ, CLIP-IQA, style-transfer, Johnson and perception–distortion figure or claim | Quoted from the sources in the list at the end, not re-measured: the pretrained weights could not be downloaded where the guide was written | | Provided, not run | An optional script that adds an LPIPS column to the section 1 table, and an exercise on style-transfer weights | In the appendix, with suggested package pins; treat it as untested until you run it |

1. Why pixel metrics are not enough

Take a grey photograph and damage it in four ways. The first three are tuned to the same mean squared error (MSE), so PSNR cannot tell them apart. Higher PSNR and SSIM mean "more similar".

DistortionMSEPSNR (dB)SSIM
Shifted right by 1 pixel263.623.920.857
Gaussian blur, σ = 2.46264.823.900.786
Added Gaussian noise, σ = 17.2265.923.880.469
Brightness raised by 10 levels (invisible to most viewers)99.028.170.880
JPEG, quality 10 (visibly blocky)83.828.900.859

Three lessons come out of this table.

  1. Equal error is not equal appearance. The first three rows have the same PSNR to within 0.04 dB, but nobody would call a one-pixel shift as bad as heavy noise. SSIM separates them, because it looks at local structure: the noisy image falls to 0.469 while the shifted one stays at 0.857.
  2. A harmless change can score worse than a harmful one. Raising the brightness by 10 grey levels costs 28.17 dB, which is lower than the 28.90 dB of a blocky JPEG at quality 10. PSNR sees only the size of the difference, not whether anyone would notice it.
  3. Pixel metrics assume alignment. PSNR compares position i with position i. A shift by one pixel changes nothing the eye cares about and changes every term of the sum.

The SSIM authors made the same point with a photograph called "Boat". All its distorted versions have an MSE of 210, and yet they look very different: the structural index scores a mean-shifted copy at 0.9900, a contrast-stretched one at 0.9168, salt-and-pepper noise at 0.7748, a blurred copy at 0.7052 and a JPEG at 0.6949. SSIM works on a sliding window (an 11×11 Gaussian with σ = 1.5), multiplies a luminance, a contrast and a structure term computed from the window's means, variances and covariance (with small stabilising constants, K₁ = 0.01 and K₂ = 0.03), and averages the local values into the mean SSIM. A uniform brightness change keeps the local structure, which is why it barely lowers the score, in line with the third lesson.

This is the motivation behind LPIPS: PSNR and SSIM are described as "simple, shallow functions" that miss many aspects of human perception. The standard remedy before deep learning was to model the visual system by hand, as VIF and FSIM do. The finding behind LPIPS is that a network trained for something else, such as classifying ImageNet, already contains a better model of perceptual similarity than those hand-built ones.

2. From features to a distance: LPIPS

What a "deep feature" is

A convolutional network turns an image into a stack of feature maps, one set per layer. Early layers respond to edges, colours and small textures; deeper layers respond to larger patterns such as grids, fur, wheels or faces, and cover a larger part of the image (a bigger receptive field). The set of maps at one layer, with shape C channels × H × W positions, is the image's description at that level of abstraction. Two images that are close at a deep layer have the same kind of content and texture, even when their pixels differ.

The LPIPS distance

LPIPS, the Learned Perceptual Image Patch Similarity, is computed in five steps:

  1. Pass the reference x and the test image x₀ through the same network and collect the activations of several layers.
  2. At each layer, unit-normalise across the channels at every position.
  3. Scale each channel by a learned weight w.
  4. Take the squared L2 difference and average it over all spatial positions.
  5. Sum over the layers.

Diagram: How LPIPS turns two images into one number

With all weights equal to 1 the formula reduces, at each position, to the squared Euclidean distance between two unit-length vectors, which equals 2(1 − cos θ): proportional to the cosine distance, though not numerically identical to it. The learned weights are the "L" in LPIPS: they say which channels matter for human judgement.

The three ways to obtain the weights

VariantWhat is trainedStarting point
linOnly a set of non-negative per-channel weights; the network stays frozen (1,472 parameters for VGG)A pretrained classifier
tuneAll the weights, fine-tuned on the human judgementsA pretrained classifier
scratchAll the weights, from nothingRandom Gaussian initialisation

"lin" is the one most people mean by LPIPS. It is a calibration of an existing feature space, not a new network, which is why the metric is cheap to train and why it comes in backbones: AlexNet, VGG and SqueezeNet.

The data: BAPPS and how it was judged

The weights were fitted to, and the metric tested on, the Berkeley-Adobe Perceptual Patch Similarity (BAPPS) dataset: 484,000 human judgements over 161,000 patches of 64×64 pixels. The patches were distorted in three ways: traditional distortions (20 basic ones and 308 sequential combinations: photometric changes, noise, blur, warps, JPEG), CNN-based distortions (96 denoising autoencoders trained on ImageNet, whose artefacts resemble those of real restoration networks), and the real outputs of super-resolution, frame interpolation, deblurring and colourisation algorithms.

Two kinds of test were used:

What the results show

Average 2AFC agreement over all six test sets of the paper (higher is better):

MethodAgreement with humans
Human raters (the ceiling)73.9%
L2 (the pixel error behind PSNR)63.2%
SSIM63.1%
FSIM63.8%
VGG features, used as they are67.0%
AlexNet-lin69.8%
VGG-lin69.2%
SqueezeNet-lin70.0%

Read the table carefully, because it is easy to over-read. The human ceiling is 73.9%, not 100%, because people disagree with each other on hard pairs, so the useful comparison is the gap that is closed: pixel metrics reach about 63%, learned metrics about 70%, humans 74%. Also, the classic metrics are not equally poor everywhere. The paper reports L2, SSIM and FSIM at 77.8%, 79.1% and 78.6% on the CNN-based distortions but only 59.9%, 60.3% and 61.4% on the traditional ones. Their weakness is concentrated in distortions like blur and geometric changes.

Two further results explain the paper's title. First, features from self-supervised and unsupervised networks (trained without class labels) also beat the classic metrics, and so did networks trained for other tasks and with other architectures. Second, a randomly initialised network does better than L2 but much worse than a trained one (64.3% against about 68 to 70%), so some training signal matters. The authors conclude that perceptual similarity is "an emergent property shared across deep visual representations".

3. Using these metrics in practice

LPIPS, one pair at a time

The reference implementation is the lpips package. Its documentation (PyPI page) asks for RGB tensors of shape N×3×H×W normalised to [−1, 1], and says that "higher means further/more different, lower means more similar". It describes net='alex' as the default, the fastest and the best as a forward metric, and net='vgg' as "closer to the traditional perceptual loss", the one to choose when the distance is backpropagated as a loss.

import lpips, torch
loss_fn = lpips.LPIPS(net='alex')                # downloads the AlexNet weights and the linear layer on first use
d = loss_fn(img_ref, img_test)                    # tensors N x 3 x H x W in [-1, 1]
print(d.item())                                   # lower = more similar; 0 = identical

The snippet follows the package documentation and was not run for this guide (the environment could not download the weights). Check it against the version you install.

Many metrics behind one interface: pyiqa

The pyiqa toolbox "is an image quality assessment toolbox built with Python and PyTorch", offering full-reference, no-reference and task-specific metrics, with implementations calibrated against the official MATLAB or Python ones where they exist. Its architecture modules include lpips_arch, dists_arch, niqe_arch, brisque_arch, musiq_arch and clipiqa_arch. The practical value is that a benchmark script can loop over metric names, and report each metric's direction: in some metrics a lower number is better (LPIPS, NIQE, BRISQUE), in others higher (PSNR, SSIM, MUSIQ). The toolbox exposes this for every metric, so read it instead of assuming.

Choosing the metric for the question

QuestionReasonable metricNeeds a reference?Notes
Did this codec or filter keep the signal?PSNR, SSIMYesCheap and well understood; misleads on shifts, blur and noise (section 1)
Does this restored image look like the original?LPIPS, DISTSYesLPIPS is more sensitive to structure; DISTS explicitly tolerates texture resampling
Which GAN or diffusion output looks more natural?LPIPS for fidelity, plus a no-reference or human testMixedFidelity and realism pull in different directions (section 6)
How good is this photo with no original?NIQE, BRISQUE, MUSIQ, CLIP-IQANoTrained on natural-image statistics or opinion data; check the domain of the training set
What do people actually say?A subjective test (MOS, pairwise comparison)OptionalThe reference for everything else; report SROCC and PLCC against it (see the paragraph below)

Neighbours worth knowing: DISTS, MUSIQ and CLIP-IQA

DISTS (Deep Image Structure and Texture Similarity) is a full-reference index built to fix one weakness of LPIPS and SSIM: they are "overly sensitive to resampling of texture regions". A photograph of grass in which every blade has been redrawn is the same texture as the original, but it is a different arrangement of pixels, so pixel-wise measures call it worse than a lightly compressed JPEG of the original, against what people say. DISTS uses a fixed VGG16 (with max pooling replaced by blurred ℓ2 pooling to reduce aliasing) and compares two things at every stage: the spatial means of the feature maps (the texture term) and their spatial correlations (the structure term, in the style of SSIM). On the BAPPS patches it scores 0.689 against 0.692 for LPIPS, although it was not trained on them; on the TID2013 quality database its PLCC is 0.855 against 0.749; on a texture-quality database (SynTEX) its SRCC is 0.923 against 0.663. The authors also note its limits: it is highly non-convex, which makes it harder to use as a training loss, and it gives one global score and no quality map.

MUSIQ is a no-reference transformer model that reads an image at its native resolution and at several scales, with a hash-based positional embedding, instead of resizing or cropping it first. That matters because resizing is itself a quality change: a CNN that needs a fixed input size measures the picture after it has been damaged.

CLIP-IQA is a no-reference method that needs no quality labels at all. It asks a pretrained CLIP model how close the image is to a pair of opposite prompts, such as "Good photo." and "Bad photo.", and turns the two similarities into a score between 0 and 1 with a softmax. The same trick works for other qualities ("bright" and "dark", "noisy" and "clean", "sharp" and "blurry"). The authors report that the result is sensitive to the wording ("Good/Bad" worked better than "High quality/Low quality"), that rare professional vocabulary is hard for CLIP, and that a gap remains against models trained for the task.

Validate against people, not against another metric. When a new or learned metric is tried on your data, collect opinion scores for a sample (a subjective test run along the lines of ITU-R BT.500), then report the SROCC (do the ranks agree?) and PLCC (is the relation linear after a monotonic fit?) between the metric and the MOS. This is also how the papers above report their results.

4. From a metric to a loss: perceptual losses

Anything that is differentiable and measures image distance can be minimised. Feed-forward networks can be trained for style transfer and super-resolution using perceptual losses defined on the features of a pretrained network, instead of per-pixel losses. The reported result is that a trained style-transfer network gives results qualitatively similar to the optimisation-based method of Gatys et al., while being "three orders of magnitude faster", and that for super-resolution a perceptual loss gives visually more pleasing results than a per-pixel one.

The reason is the same as in section 1. A per-pixel loss (MSE) is minimised by the average of all plausible answers, which is blurry: if a sharp edge could be one pixel to the left or to the right, the average of the two is a smeared edge. A feature loss does not penalise the exact position of a texture detail as heavily, so the network is free to produce a sharp edge in one place.

LPIPS is used the same way: as a training term, as a model-selection score, or both. This is also its main risk, covered in section 6.

5. Neural style transfer and the Gram matrix

The procedure

The optimisation-based version of neural style transfer starts from an image (the content image or noise) and changes its pixels, not the network, until two losses are small. The method behind it uses the convolutional layers of VGG-19 (the original replaces max pooling with average pooling, which "improves the gradient flow"). Content is the feature response of a higher layer (conv4_2). Style is the set of Gram matrices of five layers (conv1_1 to conv5_1, each weighted 1/5), a multi-scale, texture-like description that leaves out the global arrangement. The image is made by gradient descent on a white-noise image to minimise α·(content loss) + β·(style loss), and the ratio α/β is the dial between the two: a strong style weight gives a texturised painting with little visible photo, a strong content weight keeps the photograph recognisable but matches the style less well. The authors' central finding is that "the representations of content and style in the Convolutional Neural Network are separable", though not completely: a perfect match to both at once is usually impossible. The PyTorch tutorial implements the same idea with a few differences, as follows:

Treat these numbers as settings of one implementation, not defaults of the method. The original formulation uses a different loss normalisation and expresses the balance as a ratio α/β of 10⁻³ or 10⁻⁴, so the tutorial's 10⁶ cannot be carried over to another code base, and the layers (conv_4 against conv4_2) also differ. Changing the weights changes the result in the direction the original paper describes (more style weight, more texture and less visible photograph), so the useful exercise is to repeat a run at several weights and look (a suggested sweep is in the appendix; it was not run for this guide).

What a Gram matrix is, and what it throws away

Reshape the feature maps of a layer into a matrix F with one row per channel (K rows) and one column per position (N columns). The Gram matrix is G = F Fᵀ, a K×K table of how much each pair of channels is active together. The tutorial then normalises it: "the gram matrix must be normalized by dividing each element by the total number of elements in the matrix", so that large layers do not dominate the loss.

The important property is what G does not contain: where things are. The sum over positions is the same in any order. This is checked directly in the appendix: shuffling the columns of F changes the feature maps completely, and leaves G exactly unchanged (np.allclose returns True). So matching Gram matrices matches the co-occurrence statistics of features, which is texture and colour palette, without matching their layout. That is a working definition of "style" as opposed to "content", and the reason the content loss is taken on the feature maps themselves (which keep layout) and the style loss on their Gram matrices.

Why this works, in one result

There is a precise reason why Gram matrices capture style. It can be shown that matching Gram matrices is mathematically equivalent to minimising the Maximum Mean Discrepancy (MMD) with a second-order polynomial kernel, so style transfer is, in effect, aligning the distribution of features of the generated image with that of the style image. This is supported by good results with other distribution-alignment methods too.

The equivalence is easy to check numerically. For two sets of feature vectors, the MMD with kernel k(x, y) = (x·y)² equals the squared Frobenius distance between their (mean-normalised) second-moment matrices. The appendix computes both on random data and prints 4.367194382521898 and 4.367194382521896: equal to rounding error.

The fast version

The tutorial's method runs an optimisation for every new pair of images. A feed-forward network can be trained once per style, using the same content and style losses as the training signal, so that stylising a new image is a single forward pass. This is the "three orders of magnitude" of section 4.

6. Where learned metrics mislead

  1. They are only as good as their training distribution. LPIPS was fitted to 64×64 patches and to the distortions of BAPPS. On other content (medical images, line art, very large images, a new kind of artefact) its agreement with people has to be re-checked, not assumed.
  2. Fidelity and realism are different goals, and they trade off. It can be shown that "distortion and perceptual quality are at odds with each other": as the average distortion against the ground truth falls, the best possible perceptual quality must get worse, for any distortion measure. The effect is weaker for some measures, such as distances between VGG features. So a method that wins on PSNR is not the one that looks most real, and a GAN that looks real will lose on PSNR. Report both, and say which one you optimised.

Read the scope of this result carefully. It describes the best achievable pairs of (distortion, perceptual quality), where perceptual quality is defined by how well an optimal discriminator can tell restored images from real ones. It is a statement about a limit, not a guarantee about any individual model or benchmark: a real model can sit far from the bound, the strength of the conflict depends on the distortion measure used, and human opinion scores are a different operationalisation of "perceptual quality" from the discriminator-based one.

  1. A metric used as a loss gets exploited. Once a network is trained to minimise LPIPS, a low LPIPS no longer measures anything independent. Use a held-out metric or a human test to evaluate it, and keep a different metric for training than for the final score.
  2. A distance is not a quality score. LPIPS compares two images. It says "these look alike", not "this one is good". If the reference is itself poor, a low LPIPS is no compliment.
  3. Numbers across tools are not comparable. The same metric name can differ between implementations in the colour range (0–1 or −1–1), the backbone and the resizing. State all three when you publish a result.

7. DeepDream: the same machinery with a different objective

In style transfer, the loss compares an image with a target image. DeepDream asks a different question: what would make this layer respond more? In June 2015 Google published the Inceptionism post, and in July the DeepDream code example. The network is the one built for the 2014 ImageNet challenge.

Google's own account, the Inceptionism post, gives the clearest intuition. Asking a layer to "amplify whatever you see there" creates a feedback loop, and what appears depends on the layer. Lower layers respond to edges and orientations and produce strokes and simple ornament-like patterns; intermediate layers produce shapes and components such as doors or leaves; higher layers produce whole objects such as birds, buildings or towers. The picture is also biased by the training data and by the input: a network trained mostly on animals sees animals in random shapes, horizon lines turn into towers, and leaves attract birds and insects. Feeding the output back in, with a zoom after each pass, gives an endless stream of new images. Starting from pure noise works poorly unless the image is constrained to have natural-image statistics, such as correlated neighbouring pixels. The authors present the technique mainly as a way to see what a network has learned, to check whether training captured the right features, and possibly to improve architectures, with art as a side use.

The weights are fixed, as in style transfer. What changes is the input image: it is adjusted so that a chosen output neuron gets a higher confidence score, a procedure that resembles backpropagation with the roles swapped (the image is updated, not the weights). As Google's post describes it, you pick a layer and ask the network to amplify whatever that layer detects, which is gradient ascent on the image. Optimising an image from noise works poorly on its own and works much better when the image is constrained to have natural-image statistics, such as correlated neighbouring pixels. Applied to an ordinary photo, the network amplifies whatever it half-recognises there, which is why swirls of eyes, fur and architecture appear: it is a form of pareidolia, the same effect as seeing faces in clouds.

Neural style transferDeepDream
What changesThe image's pixelsThe image's pixels
What stays fixedThe networkThe network
ObjectiveMatch content features and style (Gram) statistics of two target imagesIncrease the activation of a chosen neuron or layer
Needs a target image?Yes, twoNo
Main useImage synthesis, artVisualising what a layer has learned; art

Both are instances of one technique: take the gradient of a feature-space objective with respect to the image. LPIPS-as-a-loss is the third instance. If you understand the three of them, you understand how adversarial examples are made too, since they use the same gradient with a different objective.

8. Which tool for which question

You want to…UseBecause
Check that a lossless or near-lossless step kept the pixelsMSE or PSNRThere is no perceptual question to answer
Compare codecs at the same bit rateSSIM or VMAF, plus a small subjective testStructure matters more than pixel error
Rank restoration or super-resolution outputs against a ground truthLPIPS (or DISTS), reported with PSNRIt follows human similarity judgements better, and PSNR shows the fidelity side of the trade-off
Train a restoration or style networkA feature loss (VGG-based), possibly with an adversarial termPer-pixel losses give blurry outputs
Score photos with no referenceNIQE, BRISQUE, MUSIQ, CLIP-IQAThey need only the test image
Understand what a layer has learnedDeepDream-style activation maximisationIt shows what excites a layer
Produce a stylised image from two photographsThe Gram-matrix method (slow) or a trained feed-forward network (fast)Optimise per pair or once per style
Decide whether a metric can be trusted on your imagesCollect MOS on a sample and report SROCC and PLCCOnly people define perceptual quality

9. Common mistakes

10. Things to mention in an interview

Quick recap

Appendix: the code

metrics/experiment.py

# Section 1: how PSNR and SSIM score five changes to the same picture.
# pip install numpy scipy scikit-image pillow
import io
import numpy as np
from PIL import Image
from scipy.ndimage import gaussian_filter
from skimage import color, data
from skimage.metrics import mean_squared_error as mse
from skimage.metrics import peak_signal_noise_ratio as psnr
from skimage.metrics import structural_similarity as ssim

img = (color.rgb2gray(data.astronaut()) * 255).round().astype(np.uint8)
f = img.astype(float)


def clip(a):
    return np.clip(a.round(), 0, 255).astype(np.uint8)


def report(name, t):
    print(f"{name:38s} MSE {mse(img, t):8.1f}  PSNR {psnr(img, t, data_range=255):6.2f}  "
          f"SSIM {ssim(img, t, data_range=255):.3f}")


rng = np.random.default_rng(0)
report("brightness +10", clip(f + 10))
report("shift 1 px right", clip(np.roll(f, 1, axis=1)))
report("shift 2 px right", clip(np.roll(f, 2, axis=1)))

target = mse(img, clip(np.roll(f, 1, axis=1)))   # tune blur and noise to the MSE of the 1 px shift
for s in np.arange(0.5, 6, 0.01):
    blurred = clip(gaussian_filter(f, s))
    if mse(img, blurred) >= target:
        break
report(f"blur sigma {s:.2f} (MSE matched)", blurred)

for sd in np.arange(1, 80, 0.1):
    noisy = clip(f + rng.normal(0, sd, f.shape))
    if mse(img, noisy) >= target:
        break
report(f"noise sd {sd:.1f} (MSE matched)", noisy)

for q in (90, 50, 20, 10):
    buf = io.BytesIO()
    Image.fromarray(img).save(buf, "JPEG", quality=q)
    report(f"JPEG quality {q}", np.array(Image.open(buf)))

gram/check.py

# Section 5: a Gram matrix ignores where features are, and equals an MMD with a polynomial kernel.
import numpy as np

rng = np.random.default_rng(1)
F = rng.normal(size=(8, 16 * 16))                 # K = 8 channels, N = 256 positions
G = F @ F.T / F.size                              # normalised as in the PyTorch tutorial
perm = rng.permutation(F.shape[1])
G2 = F[:, perm] @ F[:, perm].T / F.size
print("same Gram after shuffling positions:", np.allclose(G, G2),
      "| feature maps equal:", np.allclose(F, F[:, perm]))

X = F.T                                           # N points in R^K
Y = rng.normal(size=(300, 8)) * 1.3


def mmd(X, Y):
    k = lambda A, B: (A @ B.T) ** 2               # second-order polynomial kernel
    return k(X, X).mean() + k(Y, Y).mean() - 2 * k(X, Y).mean()


Gx, Gy = X.T @ X / len(X), Y.T @ Y / len(Y)
print(mmd(X, Y), ((Gx - Gy) ** 2).sum())          # equal

neural/lpips_table.py (optional, not run)

# Optional and NOT run for this guide: adds an LPIPS column to the distortions of section 1.
# Suggested pins (a matching torch/torchvision pair; adjust to your platform):
#   pip install "torch==2.5.1" "torchvision==0.20.1" "lpips==0.1.4" numpy scipy scikit-image pillow
# The first call downloads the AlexNet weights through torchvision; the linear LPIPS weights ship with the package.
import io

import lpips
import numpy as np
import torch
from PIL import Image
from scipy.ndimage import gaussian_filter
from skimage import color, data

img = (color.rgb2gray(data.astronaut()) * 255).round().astype(np.uint8)
f = img.astype(float)
rng = np.random.default_rng(0)


def clip(a):
    return np.clip(a.round(), 0, 255).astype(np.uint8)


def jpeg(a, q):
    buf = io.BytesIO()
    Image.fromarray(a).save(buf, "JPEG", quality=q)
    return np.array(Image.open(buf))


variants = {
    "brightness +10": clip(f + 10),
    "shift 1 px right": clip(np.roll(f, 1, axis=1)),
    "blur sigma 2.46": clip(gaussian_filter(f, 2.46)),
    "noise sd 17.2": clip(f + rng.normal(0, 17.2, f.shape)),
    "JPEG quality 10": jpeg(img, 10),
}


def to_tensor(a):                                  # grey 8-bit -> 1 x 3 x H x W in [-1, 1], as the lpips docs ask
    t = torch.from_numpy(a.astype(np.float32) / 127.5 - 1.0)
    return t[None, None].repeat(1, 3, 1, 1)


loss_fn = lpips.LPIPS(net="alex").eval()           # lower = more similar
with torch.no_grad():
    for name, v in variants.items():
        print(f"{name:20s} LPIPS(alex) {loss_fn(to_tensor(img), to_tensor(v)).item():.3f}")

# Exercise for the style-transfer sensitivity (not provided as code): in the PyTorch tutorial, repeat the run with
# style_weight in {1e4, 1e5, 1e6, 1e7} and content_weight = 1, and compare the outputs. The original formulation
# uses a different loss normalisation and ratios (alpha/beta of 1e-3 and 1e-4), so the tutorial's numbers do not transfer.

Sources

Practise this topic: Computer Vision

More guides