Padding.

The Stride module counted where a 3-pixel kernel can start on a 16-pixel row: 0 through 13, which is 14 positions at stride 1. So even at stride 1 the output is smaller than the input: 16×16 becomes 14×14. A 3×3 kernel loses one pixel on each side, because its center can never sit on the outermost ring of pixels without part of the kernel hanging off the image. Stack ten of those layers and you’ve lost 10 pixels on every side before the network has done anything useful.

Padding is a border of extra pixels added around the image before the kernel slides over it. The added pixels are all 0. Padding 1 adds a one-pixel border, so a 16×16 image becomes 18×18. Padding 2 adds a two-pixel border, making it 20×20.

With padding 1 the 16×16 image becomes 18 pixels wide. Counting as before, the last start is 18 − 3 = 15, so starts run 0 to 15: 16 positions. The output is 16×16, the same size as the original image. With padding 2 the image is 20 wide, the last start is 17, and the output grows to 18×18.

Same padding

Padding that keeps the output the same size as the input is called same padding. To find it, picture the kernel centered on an edge pixel. A 3×3 kernel reaches 1 pixel past its center on each side, so it needs a 1-pixel border to fit. A 5×5 kernel reaches 2 pixels past its center, so it needs 2. In general a kernel of odd width $k$ reaches $(k-1)/2$ pixels past its center, so at stride 1, same padding is

$$p = \frac{k-1}{2}$$

The cost of padding

Padding also fixes a fairness problem. Without it, a corner pixel falls inside exactly one kernel position, while a pixel in the middle falls inside nine, so the edges get far less say in the output. With padding, every pixel can sit at the kernel’s center.

It has a side effect, though, which you can see in figure 3.2b. With padding on, the border of the output is darker than the rest. The blur kernel averages the nine pixels under it, and at the border some of those nine are padding zeros, which pull the average down toward black.