Marriage and Epilepsy

I'm engaged to a wonderful man who was diagnosed with temporal lobe epilepsy at age 12 (most likely due to a childhood fever), had a partial lobectomy as a teen, and now has seizures originating from the temporal lobe opposite the lobectomy. I love him so much, and admire his strength at continuing a full-time job and volunteering in several capacities, but sometimes the cognitive and social issues produced by impaired memory are difficult to handle without frustration. I don't want him to feel that I'm treating him like a child, or that I'm his teacher or coach. Sometimes it feels that he has difficulty connecting with me and others at a deep emotional level. I also wonder if in a way he has to keep "reconnecting" with me because his seizures sometimes destroy the relational memories we've already built. Also, I have my own issues (clinical depression and mild anxiety somewhat controlled by medication) and sometimes this affects my ability to be compassionate and loving to him. I'd be thrilled if anyone had advice about how to have a deep, fulfilling marriage despite the "abnormal" circumstances. (By the way, I now HATE any classification of what is normal or not. It just becomes a reason for people to write others off as not worth the attention or effort).