Interesting question and one that is not easy to answer. When you first start exercise you are changing up your routine and this may take you away from some previous activities in which you were involved. However, taking part in exercise will not only benefit you but will benefit those around you. By you taking care of your health will help increase your awareness of others and potentially assist them with theirs.
We must all be selfish for ourselves and if that means doing so through physical activity then those who love you should truly support your decision. Again, this time taken will benefit you and those around you.