Should I take legal action

I was with my partner for 7 years. He didn’t technically live with me, as he worked and lived regionally. We saw each other mainly on weekends (rent was not paid to me, but he did contribute financially and was generous). I thought our ‘long-distance’ relationship worked, I thought he thought so also. However, he was always emotionally unavailable and a non-communicator. I would try to get him to open-up to me on many occasions. I tried to break up with him a number of times but he would say he would do better and open up, I would stay, and of course this pattern repeated. He assisted financially in the renovation of my house. Which we always said we would then move to his farm and build a life in the country together. I always intended to pay him back if/when we broke up. About a year before we finally broke up our intimacy had dropped to non-existent. When I finally called it over, he didn’t fight for me this time. I sold my house so I could pay him back for the renovations, I paid him $80,000. When I think I probably should have paid him have the profits instead, which was $60,000. I have recently found out, over a year later after we broke up, that he was having an affair. With a woman married to his friend and co-worker, whom they have two children. The affair started around the same time as our intimacy had stopped. He and her had gotten pregnant and had an abortion around Christmas time, two months before we broke up. I’m heartbroken. Now that this has come to light and he was betraying and lying to me for over a year, I have confronted him and asked him that had I known, I would not have given him the full amount. I have asked him for $20,000. He believes I have no right and he owes me nothing. He and her are still together by the way. What do you think?