EX WIFE DRAMA :-(

Please post anonymously.

This is going to be quite long so I apologise.
In October of 2014 I met a man who was (at the time) perfect. From the first meeting we clicked. He has 2 children from a previous marriage and he had been separated for just over a year. I had one children from a previous relationship and had been single for 3 years.
We bonded over everything. We were both co-parenting with our Exs and we all actually became friends. More so me with his ex. In that time we were all so comfortable that I was looking after his children on a regular basis.
Me and my sons dad were close on a friendship level but everything revolved around our son.
Their relationship however, I started noticing (after removing my rose coloured glasses) was way to close for comfort. It got to the stage that EVERYTIME we would spend time together their would be some reason she needed him there... He started spending a lot of nights there... Uncomfortable right?
So after 4 months I told him that the idea of marriage to me means a lot and that he really should give it shot with his ex because obviously something is still there...
I then found out I was pregnant.
Mind you a month previous she had falling pregnant to one of the several men she was interment with, sadly she miscarried. My partner left my home to take her to hospital... We both then helped her through that hard time.
When I found out I was pregnant I panicked... Like badly. He assured me everything would be OK and that he was in a way pretty happy... So I decided to sit back and look at all my options.
He decided to sit down and tell his ex and her first response was "when she booked in to get rid of it"... I was disgusted... He then had the same opinion... He was completely against it. Said that his ex had mentioned divorce papers and he was ok with being separated but she had made it clear she just wanted a couple of years break from marriage...
He was in and out the entire pregnancy.
Up until the day I gave birth. Which was a beautiful 4 months ago.
Our little one is pure perfection and my son loves his sister more then anything.
The night before I gave birth me and the baby's dad had a pretty heated argument because he decided to inform me of this.. "Me and (ex) have decided that the custody will be me being there with you for a night once a month and the weekend in between ill be having my other 2 kids for the weekends"... I said no... He said "I'm sorry but that's all I can offer because then the time is spread evenly"...
Now since my daughter has been born his ex has decided that her children are to have nothing to do with my daughter. Her children only found out about having a sister One month ago. In which she told them without consulting me or my daughter's father first. She has encouraged her children to only refer to my daughter as their half sister.
Now, now that they know they obviously want to meet their sister however, they are not allowed to do so unless I drop my daughter off to her father and leave.
My daughter is breastfed and has been relying on me for four months for comfort, for food, and for love. She has only been with her dad every third weekend since birth and a few random visits in between if we're lucky. He sleeps through her crying or doesn't recognise when she just wants to be held.
In between this already confusing situation he also managed to miss our daughter's first Christmas due to his ex saying that we werent important enough to spend that day with and that his real family is what matters, mind you his ex lives six corners away from my home and he still couldn't even come over for five minutes. I had to text him at 6 PM and remind him that he had had no contact he then called and within 30 seconds his ex started screaming at him telling him that me and my daughter shouldn't be interrupting her day. I know there is a lot of gaps in the story but there is no screen big enough to tell the whole thing. There has been both emotional and physical abuse throughout the marriage and throughout the separation from her behalf. She was quite open about hitting her husband when they were married and abusing him after. My daughter is suffering, and losing out on time with her father due to his ex threatening to take his children away if he tries to change the custody arrangement.
So basically what I'm asking is do I persevere and fight for my daughter's right to have her dad and hope that one day he can get away from his ex and her manipulative tendencies.
Or do I I remove myself and my daughter from the situation so his ex can keep doing what she's doing, she has also threatened to take her children away from their dad's family also, his family are scared that she will actually do that.
His family has been in contact with me and have been quite clear that this emotional abuse from her behalf has been going on since the day they got married and that he is programmed to have to listen to her because he has known no different for a long time. I know that if I push for him to spend more time with our daughter and make it even between the children that she will remove the kids from him and his family and I'm afraid that I will be resented for it.
I don't know what to do but any help or advice is appreciated. I have been seeking counselling for over a year to deal with and another person X marriage and I'm over it.
He is to have no contact with me on the weekends that he has his other children however the one weekend we get with him a month his ex is constantly calling and texting. There is too many double standards to even explain in one story.
She claims she wants nothing to do with our daughter at all but is the one calling the shots for everything and I'm expected to go along with that because according to her I ruined her chance to have her family back.
Please I need advice on how to handle and emotionally abusive, manipulative, irrational and unreasonable ex-wife.
I suffer from anxiety pretty badly and this has caused it to get 1000x worse! I'm at a loss...