I think we all get affected by different things in different ways so giving it a try at a time when you can deal with any bad effects is probably the way to go as you might not have any issues at all. It would be a shame to give up on something without knowing.
Personally (and I have lichen sclerosis as well as VA) I find that I can swim occasionally without any issues. I make sure i shower well when I get out of the pool and again when I get home and then slather myself in the moisturiser that I know works for me. Another thing I do, if I'm on holiday or am lucky enough to have a spa day, is to make sure I don't sit around in wet bottoms, I take a change of swimmers with me.
But, we're all so different!
Suzie