Honestly, its up to
you if you want to drop if it feels hard. I actually have some advice as a fellow year 11 student! Basically, to improve in maths, yes, you are supposed to do it more regularly so I advise that you start implementing some plans into how much you will study for maths, like doing the Cambridge maths textbook, to just be able to practise the questions you do in class.
Another tip which I feel like should help you, its something I'm currently trialing/working on, but I'm not sure if it works long term but:
Every time I sit down to do maths homework and/or extra questions, I have a sheet of paper with me with 4 columns:
1. Question I got wrong
2. What did I do wrong
3. What is the Correct Approach
4. What will I do differently next time
No MATTER WHAT you get wrong, you should always put it in there and
NONE OF "I would've gotten this but like it was a silly and simple mistake" even a simple mistake like wrong number or misreading your own working out(Lol thats me).
EVERY MISTAKE COUNTS.
BTW, I STRONGLY SUGGEST using physical paper and pen to write it down instead of typing. It will help you.
This should help you with recognising the common mistakes you make in a lot of the questions you do, should possibly help you fix them since you will know what to do. Also, I advise using ChatGPT to show you the working out and process, and you can even ask your teacher if your working out is up to their standards.
AND EXTRA TIP:
This is going to probably sound stupid but if it weren't for this advice, I wouldn't have been doing Ext right now, but in your exam if you're making silly mistakes,
BE CAREFUL, like double check your answer and if it seems sus or it doesn't quite click in your mind,
cross check with another method of working out.
(Thats why we have multiple methods of working out for the same solution)
But try not to be too paranoid or anything it could eat up time in your exams
Anyways, the choice is yours if you want to drop EXT Maths but if whether you are or not pursuing EXT Maths I still heavily recommend this just to try to see
if it works for you.
GOOD LUCK! YOU CAN IMPROVE. DON'T WORRY 
EDIT: I agree with Study to Success, if it is mainly just silly mistakes dropping to Standard wouldn't really benefit you, use what I said: it could work for you!