Nope, nowhere near as deep. Most of the math you do in the other subjects are pretty specific and you can revise pretty quickly (like graphics is only about matrix algebra, and pretty straightforward ones at that)
Do you want a life?
I did higher comp and higher phys when I started. Didn't do higher math - I don't have a thing for maths. I think it was alright. Phys can be a bit tough at times, and higher comp is very enjoyable.