I really don't know what to do

I am a 33 year old single mother with a 2.5 year old son. Up until recently, life was fine, I worked part time and I was somewhat happy. Then somewhere down the line for reasons I still cannot explain or understand I started getting severe panic attacks and fell into a deep depression. They got so bad I could barely get myself going every morning.
I thought a lot of my anxiety was coming from the demands of my job, so I made the decision to resign to spend more time with my son. I partially felt better after this decision but then it all came rushing back to the point I had a breakdown and spent a few nights in hospital for some rest and to get some help. I have seen a psychiatrist and she has put me on some medication to help with the panic disorder.
My problem now is I am constantly afraid to be home alone. I have not had a relationship since my son's father and have no idea how to go about meeting people, especially in a small town where I know everybody.
I have spoken to my Mum about us moving back in with her for a while and she is fine with that. I am happy there and so is my son. We have a room all set up nicely. The problem is my 25 year old younger brother. He has just moved back home also and is a lazy bum who smokes weed all day and is argumentative, selfish and generally an a**hole. He does not listen to a word my Mum says, does not contribute a cent to the household, she has no control over him whatsoever yet she will not put him out as he has nowhere to go.
As much as I need to be with my Mum until I get back on my feet, I am unsure if taking my son there is a good idea. (My brother does love my son I'll admit that) Yet the thought of living here much longer alone sends me into a state of panic.
I am very vulnerable and confused and do not know what to do. Sorry if this doesn't make much sense but I need some advice as I don't know if his negativity will be good for my mental health right now.