Get your mind out of the gutter. It didn’t start out as an expression we use today. At the height of the sailing ship era, war ships and freighters brought along round iron cannon balls which they had to prevent from rolling around the deck. They did that by stacking them in pyramids, 30 stacked in rows of 16, topped by 9 topped by 4, then 1. Problem was, on a moving ship, the bottom row wanted to slide away. They stabilized the bottom of the pyramid with a Monkey, a brass plate with 16 slots for the balls.
Another problem: rust held the Money and cannon balls (both iron) together. So, they started making the Monkey out of brass. Great, except that when the temperature fell, the brass contracted quicker than the iron, meaning the shrinking indentions caused the balls to roll off. Because it was “Cold enough to freeze the balls off of a brass monkey.”