As the saying goes "looks are only skin deep" and what really matters is the person. Whilst I was attracted to OH, and boyfriends before him, it was their personalities that really mattered to me. Those people who dumped you did you a favour. You didn't waste your time with them hoping it would go further, and it meant you found your OH, who sounds like a lovely person.
It's the same with friends, plus I'm also an introvert, so I've never had a large group of friends. In recent years I made the decision to shrink the friends group further, preferring quality over quantity. As to being rejected, not sure if it's the same thing, but since my surgery I've had people stare at me, tell me I'm stupid, treat me like I'm an idiot, assume I need help to order food, and then, of course, there was the woman who ran away from me! People will be cruel and the only thing we can control is our reaction to them. We can let them hurt us or we can realise they are the ones who may well be stupid, etc and move on.