if the retainer has warped or stretched out over time (this tends to happen with clear plastic retainers as the plastic degrades after some time) this gives your teeth some wiggle room to move while inside the retainer.
Our jawbones are constantly being remodeled, and our bone continues to grow (at a very slow rate) throughout our lifetime. The teeth have to compensate for the changes in the bone by shifting around. Should you lose a tooth later on in life, this may also induce drifting of the other remaining teeth around the space.
Long story short, as long as you are alive with all your metabolic processes functioning normally, continued tooth movement is a fact of life! Hence, lifelong retainer wear is a must.
For best results, I would advise plastic retainers be replaced every year to ensure maximum fit and retention.