Do you have a clear medical reason for trying to stop it every year or just a vague belief that it is somehow dangerous or that you aren't deserving of hormone treatment?
Many people have subconsciously internalised negative and scientifically inaccurate messaging that there is something harmful, immoral, weak, vain or frivolous about replacing our ovarian hormones, or that "toughing it out" is strong and admirable, and this causes them to put pressure on themselves to come off a beneficial treatment when there is no medical reason to do so, often at considerable detriment to their wellbeing.
Any treatment is a choice and won't be for everyone, but any decision you make should have a sound rationale behind it.