My son is driving me crazy!

Hi IM's. I'll try and make as much sense as I can but I am quite stressed and upset.
I have a beautiful 3.5 year old boy. It has been just us from day one and I work full time to provide a decent life for us. He spends his days between daycare and my Mum or sister once a week.
The last 6 months or so his behaviour has been awful, more so the past couple of months. He does not listen to me at all, throws stuff all over the place and at me, hits and slaps me and swears. Runs around flicking switches, slamming doors and touching everything I tell him not to. He has his moments where things are calm and smooth at home but the big guns come out the moment we go anywhere or someone comes over. He just goes absolutely feral! He is swearing at daycare, hurting other children and the carers, throwing food and anything else he can get his hands on and not listening at all. It's got to the point they have to segregate him from the other children often.
When we go out, he screams and carries on, spits you name it. I am constantly turning down invitations to go out because as awful as it is to say, his behaviour embarrases me.
He has developmental delays so I understand it's harder for him to understand some things his peers do but he should know the difference between right and wrong!
I am at a loss where this had come from, he has not learned any of this behaviour at home and he is the main antagonist at daycare apparently. He is also like this at Mum's when she watches him, he just trashes her house and will not listen.
His father was diagnosed with ADD as a child. I am wondering if this is something to look into... any advice please would be so appreciated! I absolutely lost my shit tonight which is not normal for me because I had hit breaking point and he was quite shocked but didn't stop the bad behaviour. Please help!