I strongly advise that you keep chemistry. I started poorly in physics. My first test in year 11 I got 37%. I improved greatly and in the HSC tasks got 100% for the first task/exam came thirs overall. So it is too early to consider giving up, with time and hard work you will improve.
Chemistry...