I just don't know what to do. I'm a mess. Really don't want to take medication.

Backstory is I had my first seizure when I was 15, grand mal. Had a couple of SP and GM's between then and 18. Never really had any diagnosis or medication or whatever, then they just stopped. For 6 years I was med free and seizure free and lived a normal life. Had a grand mal out of the blue in sept 2015 after a heavy night drinking. Went in for tests and such and they had found evidence of a previous bleed on my brain, likely due to a condition I was born with that I was always told I had "grown out of". No idea why they never picked up on this bleed previously. Anyway, I saw the epileptologist in december and he prescribed Lamotrigine, I didn't take it though, I was too scared. Since then my anxiety has gotten worse and worse, to the point of becoming agoraphobic and I've been signed off work for the past 9 months because of it. Since the GM in 2015 I've been having sporadic SP seizures, closest together was 9 days apart, longest apart a few months. No more GMs. My epileptologist still wants me on the Lamotrigine. Being the hypochondriac I am, I've looked up people's stories and the effect this medication and other AEDs have had on them and I just think - is it worth it? It feels like the possible cons outweigh the benefit. I have this irrational fear of being out of control of my own mind/thoughts, and the thought of losing my cognitive ability, memory or worse terrifies me. The thought of the long term effects terrifies me, because I think well, maybe all I'll have is SP seizures that'll last a few mins and be done with it. I could be setting myself up for a life of having to deal with feeling lousy every single day, for the sake of stopping some small seizures every couple of months. I know there is a flipside and the seizures could worsen, but I keep thinking should I just deal with that when it comes to it? I don't drive, never have, don't have kids and never will, nobody is dependant on me apart from my dog, so the "safety to others" issue doesn't apply. Does anyone here not take meds? Anyone in a similar situation where their epilepsy isn't so severe? I feel like my increasing anxiety surrounding the seizures is adding to their frequency, and I'm trying to put all of my focus into working on my mental health. Just need some advice/stories/words of encouragement etc. Cheers Steve