Stacking layers.

Put a second 3×3 convolution on top of the first. A unit of layer 2 reads a 3×3 patch of layer 1’s map, and each of those nine units reads its own 3×3 patch of the image. The layer-2 unit’s field is everything those nine units can see.

Follow one row. The layer-2 unit at column 16 reads layer-1 units 15, 16 and 17:

• Unit 15 reads pixels 14, 15 and 16.
• Unit 16 reads pixels 15, 16 and 17.
• Unit 17 reads pixels 16, 17 and 18.

Together they cover pixels 14 to 18, five pixels. The same happens down the columns, so the field is 5×5 and $r = 5$.

Figure 6.2 starts with these two layers. In the view from the side, the lines from the blue unit fan out to three units of layer 1, and from those to five pixels of the input.

Each layer adds the same amount

Add a third layer, and the field grows to 7. Add a fourth, and it grows to 9. Each 3×3 layer at stride 1 adds 2, one pixel on each side. The outermost units it reads sit one pixel left and one pixel right of center, and each brings its own field with it.

In general, a layer with kernel size $k$ at stride 1 adds $k - 1$. A stack of $L$ such layers has

$$r = 1 + L\,(k - 1)$$

For 3×3 kernels that’s $2L + 1$: 10 layers give $r = 21$. With dilation, use the effective size $d(k-1)+1$ in place of $k$.

Keep adding layers. After 16 layers the field size is 33, larger than the 32-pixel image. On the input, the dashed square now hangs past the edge of the duck, and the part that hangs over is padding.

Small kernels, stacked

Two 3×3 layers see a 5×5 field, the same as one 5×5 layer. Three 3×3 layers see 7×7, the same as one 7×7 layer. The stacks use fewer weights:

Field	One big kernel	Stacked 3×3 kernels
5×5	25 weights	2 × 9 = 18 weights
7×7	49 weights	3 × 9 = 27 weights

The two aren’t the same function, since a stack computes three dot products in a row instead of one. But they read the same pixels with fewer weights. VGG, a 2014 network, is built almost entirely from 3×3 convolutions for this reason.