You are right to say that. Most cataracts actually are not usually threatening.
The only time you want to do surgery is if it affects your vision, making it blurry vision, making your life miserable, or it causes other complications like glaucoma, causing something called angle-closure glaucoma and flukes in vision.
Maybe there is something wrong with the retina at the back of your eye and cataracts get in the way which makes investigation or diagnosis of retina conditions a problem. That is more reason to undergo surgery.
