Absentee abusive ex

Sorry for the long story but any advice would be appreciated. I have a four year old with my ex. He left us when she was 8 weeks (on Xmas eve) when I was suffering severe postnatal depression. We have tried to reconcile on and off but he is not interested in anything that remotely involves responsibility or adulthood. If I'm not interested in being with him he makes little to no effort to see our child. It's nothing for him to go six month without seeing her and then suddenly demand to see her but only lasts a few weeks. I have found recently - after a failed attempt at reconciliation that was really just me trying to get him involved in his child's life - that he has been lying to me and everyone for the entire time we have know each other. He told his family I kicked him out even though he left us and they, as a result, called me and abused me calling me a home wrecker and swearing at me. He has been verbally and emotional abusive to both me and our child. Regularly "forgets" to pay child support. Left us drowning in debt. We have seen two psychologist both of whom have advised me to never leave him alone with our child for they fear he is highly unstable and dangerous. He recently told me he doesn't want to be around us for fear that his anger is so put of control he can't trust himself. I told him to stay away. That you can't do this to a child flitting in and out of their lives, saying you'll be there and not turning up. Not even calling to say hello. Of course he suddenly wants to start visiting again but after one visit mysteriously gets "sick" and doesn't come around anymore. I let our child call him and he sounded well, in fact he sounded like he was at a party.
I want to stop him seeing our child. As a mother it breaks my heart but I can't handle seeing her crying and asking why her daddy doesn't want her. I fear for her and my safety along with her emotional well being. But I don't want to be the evil ex that kept him from his child. I've done everything I can to create a bond between them and he doesn't care. What do I do???