Because men don't have bony structures that could get in the way, they can have a disproportionately large penis for their bodies. Really. Compared to many other primates, the human penis is larger. This helps with attracting child-bearing partners.
But the trade-off is that the human male's testes are tiny compared to other species like chimpanzees.
When a female chimp goes into heat, she copulates with as many males as she can in order to ensure that she produces offspring. All the adult males then protect the child because it could be theirs. So chimp testes need to be big to produce large quantities of sperm because their sperm fight with other sperm for the right to the egg.
Because human males theoretically don't have to compete with other men, their testes don't need to produce an oil rig's worth of sperm. Their testes don't need to be as large (and thank goodness-considering how many times men's groins have been greeted by an errant baseball).