This is not necessarily true, only algebra is needed for physics and most physics students are able to do basic algebra anyways.
Anyways, do physics if you will enjoy the subject, not because of scaling. I personally hated physics and found it boring (idk blame my teacher?) and hence did not study for it at all.
Scaling is certainly a factor to consider when choosing subjects, no matter what people say, it still does play a role in the ATAR you get relative to the effort you put in. The subjects that you like aren't always filled with what you like/want to do (e.g. Like art ==> go into VA ==> Theory is cancer (you get bored/ sleep in class)). The other argument about turning hobby/passion into work is that there may be a removal of the ability to control when you engage with this interest of yours because you have a need to interact with the subject regardless of your mood/feeling towards the subject in the moment. (aka. When does working on something you like become 'work')
With what Hiva said above, I would disagree with placing Chemistry beneath Physics in terms of difficulty when one excels at logical/maths-based thinking. The math in physics is much more straight-forward, you either apply a formula or derive a formula, and explain what the equations are if asked (just look at the components of the equation), while chemistry's maths is abstract and weird, and absolutely a hellhole (if you think that maths is going to carry you in any way in the subject, it's really not). Yes, there is maths involved, but basic arithmetic is not what is going to get you the marks, it's first working out what to do with the information, deciphering and allocating the correct method/formula (really different from the straight-forwardness of physics) to find a method is what I believe as the 'maths' people refer to.