Open communication is vital. You seem to be "suffering in silence"? This is futile. Frankly, unkind. Sorry. It would be different if he were harming you in any way, but this doesn't seem to be the case whatsoever. Why not have a chat with Relate? I believe one party can do this, possibly by e-mail even and I know it helped me massively years ago. Yes, to leave, but to leave with grace.
An external, neutral sounding board can be very helpful to give you the confidence to air whatever needs aired and to explore your true feelings so that you can be real when the two of you talk meaningfully and kindly together, whatever the outcome. Things may all surprise you once a dialogue is opened up? It just all feels like you are keeping this big secret and that's not healthy for either of you, ultimately. Good luck!