Ultimately, as much as it was your choice to have the ICL inserted, it is similarly ultimately your decision to have them removed.
Though, it would be prudent to have a deep discussion with your surgeon to help you weigh the pros and cons of ICL removal, whether your fear or paranoia can be resolved in other ways apart from its removal.
Also read my complete guide to icl surgery in Singapore.