Not every pixel counts.

The field size says which pixels can affect a unit. It doesn’t say whether they all affect it equally. They don’t.

Counting routes

Go back to two 3×3 convolutions. A pixel’s value reaches the layer-2 unit by passing through a unit of layer 1. A route is one such path: a chain of taps, one per layer, that carries a pixel’s value up to the unit. Along a route, the value is multiplied by one weight at each layer.

The center pixel of the 5×5 field lies inside the patch of all nine layer-1 units that the unit reads, so it has 9 routes. A corner pixel lies inside the patch of only the corner unit, so it has 1. Along one row of the field the counts are

$$1,\; 2,\; 3,\; 2,\; 1$$

and the count for a pixel is its row’s count times its column’s count. The pixel just left of center, in the middle row, has $3 \times 2 = 6$ routes.

Add a third layer and the counts along a row become 1, 3, 6, 7, 6, 3, 1. The center pixel has $7 \times 7 = 49$ routes, and each corner still has 1. Every layer piles more routes onto the middle, so the counts take on a bell shape.

Figure 6.4 shades each pixel by how many routes it has, so bright amber means many routes. With the routes shading on, the view from the side prints the count along the row whenever the cells are wide enough. Add layers and watch the bright center tighten relative to the field: the field keeps growing by 2 per layer, but most of the routes stay near the middle. Switch the shading to field to compare with the plain field.

The effective receptive field

A pixel with more routes has more ways to change the unit. So the edges of the field matter less than its middle, even though they’re inside it.

The effective receptive field is the part of the field that has a noticeable effect on the unit in practice. A 2016 study by Wenjie Luo and colleagues, Understanding the Effective Receptive Field in Deep Convolutional Neural Networks, measured it in real networks. They found it has a roughly bell-shaped profile and fills only a fraction of the field that the formula gives.

Route counts are only a rough guide. They treat every weight as equal, but trained weights aren’t equal, so some routes carry more than others. And with max pooling, only the winning pixel of each window passes its value on for a given image, so most routes through a pooling layer carry nothing on that image. The counts show the shape of the field when nothing else is known about the network. They aren’t a measurement of a trained one.

So read the field size from the formula as an upper limit. A unit of VGG-16’s last map has a 212-pixel field, but it doesn’t use all 212×212 pixels evenly. Most of what it responds to comes from the middle.