It is not the time to give up. No one has told you that you would not be safe to drive.
Don’t stop yourself from having a go at the driver assessment.
You could be missing something valuable.
By the way, I’m still only just back at driving. In my first 1000 miles. I have had exciting enjoyable trips out, everything is new. On the other hand I also sometimes had to deal with confidence issues getting used to being out and about again. Simply finding my way around again was a challenge to be met.
If you haven’t already got one apply online now for a blue badge. They take quite a while to come through. You don’'t have to be a driver. You can use them as passenger to get preferential treatment, free parking for the vehicle you are riding in and so on.
Take the challenge and good wishes. 
All the best Mark.