Benzoyl peroxide helps to treat the painful inflamed lesions of acne by killing the bacteria that causes them. If there are no more active inflamed acne, you can consider stopping its use.
However, the non-inflamed comedonal acne i.e the whiteheads and blackheads which are the precursors of inflamed acne continue to form even after the inflammation has subsided, so these have to be controlled to prevent acne from relapsing.
The best control for these non-inflamed acne comedones is the regular use of topical tretinoin or adapalene at night over acne-prone areas. This maintenance treatment must be continued for several months or even longer after your inflamed acne has cleared. This is what you should consider before you stop benzoyl peroxide.
Hope this helps!