Coupla stories and a bit of a bio ...
When I was 17, a very good friend was rear-ended on his ZZR250 which resulted in a back injury. He hasn't been on a bike since, although he's drooled over my ZZR.
A friend came off doing 120 around a corner recommended at 40. He hit gravel on his Bandit 250, broke numerous bones and nearly lost his kneecap. 3 months later he still had little feeling in the calf muscle on that side due to nerve damage. He's recovered but hasn't been on a bike since. His brother used to ride a ZXR250, haven't heard from him in a while. Both crazy boys.
Another friend came off in an altercation with a car. Was in hospital for a couple of weeks, lost his spleen. Hasn't been on a bike since to my knowledge, although I haven't seen him in over 2 years.
The moral of these stories? Sometimes it's ok to say it's just not worth the risk. Maybe you're not meant to ride? Only you can answer that.
I started riding at 16 on an old Suzuki TS100. 2 stroke thumper. Would do all of 55 mph flat stick with a tailwind down on the lucerne paddock. Had a knobby on the back and a (well-worn) street tyre on the front. I swore blind I'd never ride on the road, 'cos offroad (even on a midget street bike) was enough fun. I was wrong.
The chap on the Bandit crashed about a year before I got my licence. I thought long and hard, looked at a few statistics, and figured the odds weren't as stacked against me as I'd thought. So, when it was time to renew my licence I did the few extra questions and got my learners.
When I finally got to ride on the road, I was as nervous as I'd ever been. Strung waaay tight, but managed to pass all the required tests until about 1/2 way through the final (3hr ride) test something made me loosen up. Until then I'd been getting through on determination and stubbornness. From that point on I've not been on a ride I haven't enjoyed.
About 2 months after I got my licence I still hadn't scraped together enough money to get a bike. Then the chap who lost his spleen crashed. Every one of my friends and family were telling me, "See? That's what road bikes do to you. They're too dangerous."
I began to think that way myself, scared myself again. Same situation as you, I think.
Then I got my first road bike given to me. Gave it back 3 months later when I got a different (bigger and older) bike which I rebuilt. I still have it, but it's needing a few bits. Riding was fun again, and all the nay-sayers in the world won't get me off bikes. I can't help it, and don't want to. My family still hates all things motorcycle and tries to rub in every bike accident that occurs around the joint. It makes absolutely no impression other than to make me even more aware of my surroundings because I know they
can not understand why I ride. I'm not even sure I can explain it.
Should you quit? Don't ask me, I'm biased!
I will say, however, think long and hard about it. Only you can make that decision. If you decide to quit no-one will think the worse of you for it, and most will understand. If you decide to continue, goodonyermate!
And that last para is the only advice I can give.