Temporal Lobe Epilepsy or panic attacks?

I am a bit concerned with what my neurologist has told me. I'm 18 years old and ever since I was 15, I have been getting these unexplainable minute-long periods of detachment and confusion and extreme panic. During these episodes, I just kind of walk around in circles, my hands clutched near my chest (like I'm really afraid) and I just keep saying "I don't know" over and over until it passes. I feel unconnected to reality and I almost completely lose all of my senses when it happens, though I've never lost consciousness. It's very hard to put into words what really happens when I have this, but afterwards, I have a horrible headache on my left side of my head and I'm physically wiped out. When I went in for an EEG, it came up normal, but I felt these attacks happen pretty bad both during and after the test. 

I saw my neurologist recently and he is almost positive I'm experiences panic attacks. I tried to explain to him that I just do not feel like that's what it is, but because my EEG came up normal, he says he's positive I haven't had a seizure. I have a very large history of epilepsy in my family (More than half of my father's side is on some kind of medications to control their symptoms) and after talking about this with my dad, he said his seizures started at the same age and the same way- Detachment, confusion, panic, and then exhaustion. My dad has also had 2 of his 5 EEGs read as normal (though his grand mauls were and still are quite bad) so he also told me to be slightly skepitcal about normal results. 

Really, why I'm posting here is to make sure I am not crazy for questioning my neurologist. He is setting up a video EEG for a few months out, but as these episodes only happen maybe once a week, I'm worried it won't happen while I'm there and that if I do have one, the machine won't pick it up again. I just have a gut feeling that these aren't panic attacks, but am I just crazy? Should I try to press this issue with him harder or give in and try the antidepressants he said he'd give me should the next EEG also be normal? I've heard the worst horror stories about those and forgive me for trying to do anything I can to not put myself through that!