I really feel for you. I have a BIL, who has been hostile ever since I met my husband. His behaviour has been at best indifferent and at worst abusive. His behaviour worsened as the years went by. He was indifferent to the children. My MIL apologised for his behaviour. Now he will not come here, though my husband occasionally meets up with him. He would love to have a rewarding relationship with him, but has accepted this will not happen. They were close as children and he is he only relative now apart from our children.
You can't change people. This was never about us; this man is ill, (his problems extend to the way he relates to everyone else). You SIL is il too. You wont be able to change her nor is it your job to do so. But what you can do is look at how you respond to her and be responsible for your own reactions. Be calm. We have the choice to never have anything to do with my BIL, but I think you want to be there for your brother and Mum. It sounds as if she is being abusive towards him, trying to control him and isolate him from his family. One day she will take a step too far and he will want to break free. This is when he will need his family's support. I expect his self esteem and confidence is undermined. This woman is in a bad place emotionally, but there is never an excuse for abusive behaviour.
There are several women on this forum who have experienced abusive behaviour from their ex partners. It would be interesting to hear what kind of support they would have appreciated and responded to while still in the relationship.