You haven't said what strength of patch you are on but it is almost certainly a very low dose, and the presence of vaginal atrophy tells you that you have very little estrogen being delivered to the tissues.
I would use what you need in order to adequately treat your condition, and don't give up advocating for yourself.
If the GP cannot manage your treatment, ask for a referral to the local menopause clinic.