Endoscopy is recommended only if there is a strong reason to do so. The type and urgency of an endoscopic procedure depends on the nature and severity of your medical condition, which may be reflected in your presenting symptoms, medical history, family history or abnormal blood tests or scans.
Common indications for endoscopy include persistent abdominal pain, suspected blood in your stools, raised serum tumour markers, personal or family history of colonic polyps and so on. Other indications include cancer screening and abnormal scan findings discovered incidentally, and the list goes on.
What I would advise you to do is to discuss with your gastroenterologist on the reason(s) for the procedure and find out on the urgency to do so and possible outcomes for not undergoing the procedure. If you are still not convinced, do get a second opinion and do some research on your own.
Ultimately, you have to understand why you are undergoing the procedure and accept the risk(s) of endoscopy and be convinced of the benefits you will get from undergoing the procedure.