Early pregnancy and sex worries.

So we are 6weeks pregnant. I've had 2 previous misscarages. My partner is not ready for this baby and is not so exited but getting better about it. He tried to instigate sex but as he started to play around I jumped. He got upset and walked away. I love sex with my partner but it's usually quite rough. I'm just worried something would happen. I no it's probably stupid cause Bub is safe but I can't go through another miscarriage. Am I being stupid and will it be ok or was I right to worry??