A lot people harbour the misconception that strong performances in the HSC require absurdly high marks, e.g. consistently upwards of 90%. All marks are relative. I typically scored around 85%~ for most of my exams (got 50% for a Biology assignment in Year 11, and 70% once in a Chemistry exam during Year 12... whoops), and I still managed a 99+ ATAR in a public comprehensive school ranked in the 90s during 2014.
My advice is don't worry too much about what marks you get, but focus on consistency and effort, and aim for 80-85% in exams. The HSC is an examination of your ability for self-management. Things won't always go your way, and it's how you acknowledge and handle those situations that define your performance in the HSC. A 50% in an exam isn't the death of your ATAR (writing from experience). Similarly, getting 100% in an assessment won't solidify your final HSC result. Apply yourself steadily and consistently, and don't let disappointment sway your focus.
Now onto some specificities. Your preliminary results do not influence your HSC results at all. You can literally rank the lowest for all of your subjects in your Preliminary studies, and you won't technically be at any disadvantage in thriving for a high ATAR against people coming first in their courses. The Preliminary year is designed to simulate what the HSC is like, such as the density, difficulty and pacing of the courses. Hence while your Preliminary results do not factor into your final HSC results, it's good practice for the real thing. Preliminaries only really matter if you are considering enrolling into Extension courses, which I presume is of interest to you. If such is the case, then you need to keep your English and Mathematics results up. In my school, we only run one class of Extension 2 Mathematics and it was by invitation only to the top 24 of Extension 1. However since you are going to a selective school, it's probably easier to enter into Extension 2 Mathematics and English because typically selective schools run multiple Extension 2 classes (plus the fact that they often encourage you).
I also did 6 units of science during my Preliminary year. It was perfectly manageable. (Yeah, I did get 50% in Biology once, still no idea how that happened, but I came 1st, 2nd and around 4th for the yearly Preliminary Chemistry, Physics and Biology exams respectively.) It's all about how you manage your time such that efficiency and productivity are maximised. It's a no-brainer that you should spend more time on subjects you're doing relatively worse in. Make sure you familiarise with your assessment schedule ahead of time; e.g. if you have a coming English essay in a few weeks, do your maths and science homework ahead of time to free up your personal schedule. Quite literally, preparing for preparation.
Don't be afraid to do more than 10 units in your HSC year. If your 12th unit just happens to be one that you find highly interesting (Economics in my case), stick with it. It could help you decide your tertiary study (as it did for me), and hence your career. Furthermore on this point, it's very difficult to predict how things will end in the HSC. At the end of my preliminaries, I predicted that Economics won't count towards my ATAR. At the end of my HSC, it turned out Chemistry, traditionally my strongest elective apart from maths, broke my Band 6 streak.
The bulk of your worry seems to come from your entering into a selective cohort. While I've never been to a selective school myself, I have many friends who enrolled into selective schools. Yes it's competitive, yes you will find that it's a little harder to compete against the average (ranks are relative; it's meaningless to compare your rank in your new cohort to your old cohort), yes ranks can be quite volatile, and you'll have people who worry about the ranks of themselves and others much more than they should. But this is a rather ubiquitous quality, as you'll find similar going-ons in private schools, and comprehensive schools alike. The jump from Year 10 to Year 11 is one that is more significant and perhaps even disorientating than the transition between a comprehensive to a selective environment. Ultimately the best way to settle into the selective environment is to find friends who are in your situation. And I don't mean the situation of you moving to a selective school, but rather the situation of you moving onto Year 11. And therefore, you'll find that many of your own anxieties are shared with your selective peers, and that there's not that much difference between your peers and yourself.
Just because a significant amount of your selective peers are tutored, doesn't mean you need it. Our dux (99.75 and Economics state ranker) wasn't tutored at all. During his HSC year, his effort was consistent, worked only slightly ahead of the content taught during class (as a part of his time management strategy), and played Runescape -
- which just so happens to lead onto my next point. Relax. Make sure you always have time for yourself. Hang out with friends. Watch movies. Play games. Do whatever that you enjoy and relaxes you. Don't stay up working too late (I capped myself at 8pm everyday, even if there is an exam the next). Taking long walks was the most relaxing thing for me. Even during the weeks of intense studying for the trial and HSC exams, I took two hour walks every single day. I also found the time after school every once in a while to find some place to eat. It's the small things you do that you find yourself really enjoying and appreciating.
Those are the main things that I feel you should understand. The most memorable, and often the best years of schooling come from the final two years. Everyone at your new school is in the same situation as you, trying to adapt to a new environment. Relax as much as you can, and come into your new school year with a fresh and clear mindset, and you're set to go!