Research shows that after age 50, all men experience a decrease in the production of androgen and testosterone, at the rate of 1% per year. Other sources say that testosterone rates start to decrease after age 30. This is just part of the normal aging process, and symptoms can develop any time after age 40. Other contributing factors include stress or other underlying conditions that may affect hormone production. What this means is that if you have lived a high-stress lifestyle, your testosterone levels may be affected.