No, you're not being silly, and you're not alone.
I am a medical 'phobic', I hate going to the doctor!
I get white coat syndrome everytime my blood pressure is taken, it reads as MEGA high when I actually have low-normal blood pressure - always have done, even while pregnant.
Once I had a full blown panic attack waiting to see a new doctor
I know it's irrational but I can't help it!
Mine stems from one time I fainted and I had to have a whole barrage of serious tests - carotid artery scan, echocardiogram, CT scan etc etc - only for my own GP to tell me months later they were not needed at all, it was just a simple faint as I do have low blood pressure!
I know the docs at the hospital were covering all bases but it put the fear of god into me.
Ever since I dread going to doctors, and like you, fear the worst every time.
Like you, I know its irrational but I can't rationalise it in my silly head!