Stride and pooling.

Stacking 3×3 convolutions grows the field by only 2 per layer. Real networks see much more of the image than that, because they also shrink their maps with pooling or stride, and a smaller map changes how much each later layer adds.

The jump

After 2×2 max pooling at stride 2, neighboring units of the pooled map summarize windows that start 2 pixels apart. Step one unit to the right in that map, and its field moves 2 pixels to the right in the image.

The jump of a map, written $j$, is that distance: how many input pixels apart the fields of two neighboring units sit. The input has jump 1, since neighboring pixels are 1 pixel apart. Every layer multiplies the jump by its stride, so after two pools at stride 2 the jump is 4.

Now put a 3×3 convolution on the pooled map. Its three taps read neighboring units, and neighboring units now sit 2 input pixels apart. Its outer taps reach 2 pixels past the center on each side, not 1, so the layer adds $(3 - 1) \times 2 = 4$ to the field instead of 2.

The rule

For each layer with kernel size $k$ and stride $s$, reading a map with field size $r$ and jump $j$:

$$r_{\text{new}} = r + (k - 1)\,j, \qquad j_{\text{new}} = j\,s$$

Start at the input with $r = 1$ and $j = 1$, and apply the rule one layer at a time. With dilation, use the effective size $d(k-1)+1$ in place of $k$. For the stack in figure 6.3:

Layer	r	j
input	1	1
3×3 conv, stride 1	1 + (3 − 1) × 1 = 3	1
2×2 max pool, stride 2	3 + (2 − 1) × 1 = 4	2
3×3 conv, stride 1	4 + (3 − 1) × 2 = 8	2

Add another pool and another convolution, and the field reaches 18. The jump column in the listing tracks $j$ for every layer.

In the view from above, each pooled map has fewer, larger squares, and each square sits over the $j \times j$ block of pixels its unit summarizes. The view from the side shows the same thing as spacing: after a pool, the units in a row sit further apart, and the lines from the next layer fan out wider.

A strided convolution works the same way as a pool. Set the middle layer of a three-convolution stack to stride 2: $r$ goes 3, then $3 + (3 - 1) \times 1 = 5$, then $5 + (3 - 1) \times 2 = 9$. The stride takes effect on the layer after it, through the jump.

If a layer’s stride is larger than the field below it, neighboring fields stop overlapping, and gaps open between them. A 1×1 convolution at stride 3 followed by a 3×3 convolution reads pixels 3 apart, with two unread pixels between each pair.

VGG-16

Load the VGG-16 stack. Its 13 convolutions are all 3×3 at stride 1, arranged in five blocks, and each block ends in 2×2 max pooling at stride 2. VGG-16 was built for 224×224 images, and on those the rule gives the last pooled map a field size of 212, with jump 32. Each unit of that 7×7 map has a field nearly as large as the whole image.

On the 32×32 duck, five pools shrink the map to 1×1, and the one unit at the top has the same $r = 212$. Its field covers the whole duck, and the dashed square reaches far into the padding on every side. The field size depends only on the layers, not on the size of the image.