CS Fundamentals Trainer · Guides
Which image filter should I use? A decision guide
Start from what is wrong with the picture (noise, impulse specks, softness, uneven light) and pick the filter: box, Gaussian, median, unsharp mask, Laplacian or gradient. How to size the kernel, why sizes scale with resolution, what separability saves, how to treat the border, and the common mistakes.
9 min read
Most image filters are easy to apply and hard to choose. A box blur, a Gaussian, a median, an unsharp mask and a Laplacian all take a few lines of code, and each one fixes one problem while making another worse. This guide is a decision guide: start from what is wrong with the image, pick the filter that matches, then size it and check the border. It does not derive the filters; it tells you when each one is the right tool and what it costs.
1. Ask what is wrong first
Before choosing a filter, name the problem in one sentence. Almost every real case is one of these:
| What you see | What it really is | Family of fix |
|---|---|---|
| Fine grain over the whole image | Noise that varies a little at every pixel | Smoothing (average-like) |
| White and black dots, a few pixels | Impulse noise ("salt and pepper") | Median |
| The picture looks soft | Edges lost their contrast | Sharpening |
| One side is darker than the other | Uneven illumination (shading) | Large lowpass, then divide |
| Everything is dull or too dark | Narrow range of intensities | Point operation (see the point-versus-neighbourhood guide) |
| You want only the big objects | Small details are irrelevant | Lowpass, then threshold |
Two of these rows are not about filters at all. If the intensity range is the problem, a tone curve fixes it, and no kernel will.
Diagram: Choosing a filter from the symptom
2. Smoothing noise: average or rank?
An average-like filter (box, Gaussian) replaces each pixel by a weighted mean of its neighbours. If the noise at each pixel is independent, averaging n pixels with equal weights divides the noise standard deviation by √n. A 3 × 3 box averages 9 pixels, so the noise standard deviation drops to a third (a measured example: noise of standard deviation 10 becomes 3.3 after a 3 × 3 box). The price is that everything else is averaged too, so edges and thin lines blur.
A rank filter (median, maximum, minimum) sorts the neighbourhood and takes one value. The median ignores extreme values instead of mixing them in. Take a 3 × 3 window with five values of 100 and four of 255 (salt noise): the mean is 168.9, which invents a new brightness that was never there, but the median is 100, which is a clean value. The rule is that the median survives as long as fewer than half of the pixels in the window are corrupted: 4 of 9 is fine, 5 of 9 is not.
| Box | Gaussian | Median | |
|---|---|---|---|
| Kind | linear | linear | nonlinear (rank) |
| Best against | mild random noise, quick tests | random noise, general smoothing | impulse noise (salt and pepper) |
| Edges | blur them | blur them, less abruptly | step edges stay close to sharp |
| Output values | new values (averages) | new values | always one of the window's values |
| Separable | yes | yes | no |
| Watch out for | favours horizontal and vertical directions | needs a larger kernel than a box for the same blur | thin lines narrower than half the window disappear |
If you apply an averaging filter to salt-and-pepper noise, each bright dot is spread into a grey stain around it instead of removed. If you apply a median to mild Gaussian-like noise it works, but an averaging filter of the same size usually reduces that kind of noise more.
3. Box or Gaussian?
For general smoothing prefer the Gaussian:
- It is isotropic: its response does not depend on direction, while a box favours the two perpendicular directions and leaves visible blockiness on diagonal detail.
- It falls off smoothly, so edges blur more gracefully. It also models a defocused lens better than a box.
- It is the only kernel that is both circularly symmetric and separable, so it keeps the cheap computation.
A box kernel is still fine for quick experiments, and when you need the cheapest possible blur (a running sum can do it).
4. Choosing the size
The size is the decision that most often goes wrong, and it has one governing rule: the kernel must be larger than the details you want to remove, and smaller than the details you want to keep.
- A Gaussian is defined by σ. Values more than 3σ from the centre are negligible, so the kernel only needs a size of about 6σ, rounded up to an odd number: σ = 3 gives 19 × 19, σ = 7 gives 43 × 43, σ = 3.5 gives 21 × 21. A bigger kernel adds cost and essentially nothing else (an 85 × 85 kernel with σ = 7 differs from the 43 × 43 one by less than one grey level). Shorter kernels are common in libraries and are fine for many uses; the 6σ size is the safe choice.
- A Gaussian needs more pixels than a box for the same blur, because its outer weights are small: a σ = 7 Gaussian (43 × 43) blurs about as much as a 21 × 21 box.
- Scale with the image. The same kernel blurs less on a bigger picture, because the objects are bigger too. If you move from a 1024 × 1024 to a 4096 × 4096 version of the same scene, both the kernel size and σ must grow by 4 to get the same look.
- Stacking filters adds up in a predictable way. For Gaussians the variances add: σ = 3 followed by σ = 4 acts like σ = 5. For equal-size kernels of size m, Q stages combine into one of size Q(m − 1) + 1: three 3 × 3 stages act like a 7 × 7.
5. Sharpening: which tool?
| Tool | What it does | Strength | Weakness |
|---|---|---|---|
| Unsharp mask / highboost | adds k × (original − blurred) | simple, one knob for strength (k) and one for scale (σ) | large k makes halos and can clip |
| Laplacian | second derivative, added back with the right sign | very fine detail, isotropic | amplifies noise strongly |
| Sobel gradient | first derivative magnitude | strong edges, built-in smoothing | thick edges, not meant to be added back as it is |
| Laplacian × smoothed gradient | detail only where edges are strong | quiet flat areas, so little noise | more steps to tune |
Every derivative-based sharpener amplifies noise, since noise is also a sudden change. Two habits help: smooth a little before sharpening (unsharp masking does this by construction, since the mask uses a blurred copy), and use the gradient as a mask when the image is noisy.
The sign matters for the Laplacian: with the kernel that has −4 at the centre, the sharpened image is f − ∇²f. With a +4 centre it is f + ∇²f. Adding with the wrong sign darkens edges instead of sharpening them.
6. Cost and speed
The cost per pixel of a linear filter with an m × n kernel is about mn multiplications, or m + n if the kernel is separable. On a 1024 × 1024 image with an 11 × 11 kernel that is about 127 million multiplications against 23 million, an advantage of 5.5. For a square separable kernel the advantage is about half the kernel width, so it grows with the size: 10.5 for 21 × 21, 256 for 512 × 512.
- Box and Gaussian kernels are separable. Test any other kernel by computing its rank: rank 1 means separable.
- The median is not separable and needs sorting, so it is slower than a box of the same size.
- For large non-separable kernels the frequency-domain route can be cheaper, but not automatically; it depends on the image size, padding and implementation (see the spatial-and-frequency guide), so measure.
7. The border
Near the border, part of the kernel falls outside the image, so something must be supplied there.
- Zero padding darkens the border when smoothing: a 3 × 3 box on a constant image of 90 gives 40 in a corner and 60 along an edge. Avoid it unless you want that.
- Replicate padding (copy the nearest border pixel) suits borders that are nearly constant.
- Mirror padding (reflect the image) suits borders with detail. When in doubt, mirror.
8. Mistakes to avoid
- Using an averaging filter on impulse noise. Use a median.
- Using a kernel size picked for another resolution.
- Sharpening before denoising. The sharpener amplifies the noise, and the denoiser then has a harder job.
- Normalising wrongly: smoothing kernels should sum to 1 (or the brightness changes), while derivative kernels sum to 0 (and the result needs rescaling for display).
- Forgetting that correlation and convolution differ for asymmetric kernels. For box and Gaussian kernels it does not matter, but for a directional derivative it does.
- Trusting a result that looks fine on one image. Check a flat area (should stay flat), an edge (should not ring or shift) and the border.
9. Starting points, not rules
The settings below are first guesses. The right values depend on the image content, the noise level, the resolution and the implementation, so look at the output.
| Goal | First choice | Typical settings |
|---|---|---|
| Remove salt-and-pepper | Median | 3 × 3, then 5 × 5 if dots are larger |
| Reduce mild grain | Gaussian | σ of 0.5 to 2 pixels, size ⌈6σ⌉ rounded to odd |
| Remove small details, keep big shapes | Gaussian, then threshold | σ larger than the details to remove |
| Estimate illumination | Gaussian | σ about the size of the largest scene feature, kernel support about ⌈6σ⌉, then divide |
| Sharpen a soft photo | Unsharp mask | σ about the width of the blur, k between 0.5 and 2 |
| Sharpen a noisy image | Smoothed-gradient-masked Laplacian | smooth the gradient with a 5 × 5 box |
Further reading
- Digital Image Processing, 4th edition, chapter 3 (spatial filtering, smoothing and sharpening).
- Kernel (image processing)), Gaussian blur and Median filter.
- Unsharp masking and Discrete Laplace operator.
Practise this topic: Computer Vision