It could well be an advanced Year 10 question (or Year 11) but certainly not the kind you'd be tested on unless you are at like a selective HS:
It could be done by substituting:
a = A^2, b=2B^2
f(A,B)= a^2+b^2
f(A,B)= a^2+b^2+2ab - 2ab
f(A,B)= (a+b)^2 - 2ab
f(A,B)= (A^2+2B^2)^2 - 4A^2B^2
And...