It's hard isn't it. I started having problems when I came off the pill in 2014 because I was having a breast reduction. Started with low mood, extreme anxiety and night sweats. I ended up reducing my self employed work because the anxiety was crippling my confidence. I continued without HRT and noticed that by the time I was 47 I would go without a period for months at a time. I felt more level in my mood during these long stretches without a period but the mood was quite flat and then when a natural period came (after ovulation) I would get almost high and a bit manic. After a very long stretch without a period I decided to give HRT a go. Initially it was good, the first couple of months but I have never really found an HRT that keeps me on an even keel - that said I was always best in the Progesterone phase after about 5 days in. I tried continuous but I think the oestrogen dose of one pump was too low.
I feel for you with the anxiety but what I would say is depending on where you are in peri, for me it was part of the process of heading to menopause and I wonder if HRT can only do so much because of the fluctuations with your own hormones kicking in. The anxiety phase did end for me but only when my periods pretty much stopped. As I say I only ever took a very low dose of Oestrogen therefore I wonder if it wasn't really making a difference in the end. I might be wrong but since I have been off it I don't seem to feel any worse, possibly a bit less foggy.