I think the question to answer is: what is your purpose for pursuing post-grad training?
Is it to enhance your knowledge as a physician? A prerequisite for career progression? Or purely a personal goal from medical school days? etc.
Otherwise, I feel the best time to pursue post graduate education is a time when you may have fewer commitments and your passion is still strong, usually early on in medical training.
One should definitely identify an area of interest, firstly, before diving into the particular specialty. Fellowship training might require some degree of seniority/experience, so that will come eventually after the masters.
Having said that, competition and limited places for residency is also a barrier to entry and will be a test of perseverance for most.
