Small shifts.

Photograph a duck twice, and between the shots it drifts a pixel to the left. It’s still the same duck, and a network should give the same answer for both photos.

A layer with that property is shift-invariant: if the input moves by a few pixels, its output does not change at all. Convolution doesn’t have this property. It uses the same weights at every position, so when the input moves, the output map moves along with it, by the same amount. The values are the same, but they’re in different places.

Max pooling is often described as the fix. It helps, but less than that description suggests.

Following one row

Take the row [0, 0, 9, 1, 4, 0] and max-pool it with a window of 2 at stride 2. The windows are [0, 0], [9, 1] and [4, 0], so the output is [0, 9, 4].

Now shift the row one pixel to the right. A 0 enters on the left and the last 0 falls off the end, giving [0, 0, 0, 9, 1, 4]. The windows are now [0, 0], [0, 9] and [1, 4], and the output is [0, 9, 4] again. Every pixel moved, but the output didn’t change:

• The 9 moved within its window, so it’s still that window’s winner.
• The 1 crossed into the third window, but it’s smaller than the 4 already there, so it doesn’t win.

Average pooling sees the same shift differently. Before, it outputs [0, 5, 2]. After, it outputs [0, 4.5, 2.5]. The 1 left the middle window and joined the last one, and an average counts every pixel.

Shift the original row two pixels to the right, though, and it becomes [0, 0, 0, 0, 9, 1], which max-pools to [0, 0, 9]. The 9 crossed into the next window, so that window has a new winner. That’s the rule: a max-pooled value stays the same when its window’s largest value stays inside the window and nothing larger moves in.

In two dimensions

Figure 5.3 starts with the book’s duck, drawn at 16×16, shifted one pixel to the right. The first pane is the shifted image and the second is its pooled map, 8×8. The last two are the original image and its pooled map. Each pooled square is one 2×2 window; point at one to see which pixels it came from.

With max pooling, 10 of the 64 outputs change. Switch to average and 40 change. The other directions are similar: a one-pixel shift changes 10 to 16 of the max-pooled outputs and 35 to 43 of the averaged ones.

Look at where the red outlines sit. Every one is on the duck’s outline or the edge of the pond. A window that lies entirely inside the duck, or out in the open sky, holds the same pixels after a one-pixel shift, so its output can’t change. Along an edge, max pooling only changes a square when that square’s winner leaves the window or something larger arrives. Average pooling changes whenever any pixel in the window changes.

The texture image has no background at all, so every window holds detail, like a real photo. There a one-pixel shift changes 31 to 39 of the 64 max-pooled outputs and 59 to 62 of the averaged ones.

Now load the row image: the same row, 9, 1 and 4, on a black background, shifted one pixel to the right. With max pooling none of the 64 outputs change. It’s the shift you just did by hand. Average pooling still changes 2. Try the other directions too. Shifting left pushes the 9 into the next window, and 3 outputs change with either kind of pooling. Shifting up moves the whole row into the row of windows above, and 4 change.

So max pooling reduces the effect of a small shift but doesn’t remove it. How much it helps depends on what the image holds and where it sits relative to the window grid, and the network controls neither.

Real CNNs stack several pooling layers, and each layer forgives small shifts at its own scale, so the network as a whole gets more tolerant. But it never becomes fully shift-invariant. A 2019 paper by Richard Zhang, Making Convolutional Networks Shift-Invariant Again, showed that standard image classifiers can change their answer when the input moves by a single pixel.