Ok well your talent for math will certainly come in handy, especially with the relativity stuff. You doing spec will also be a big help as mechanics/kinematics and trying to use models are a big part of physics 3/4. However, a large part of the year 12 course is not covered in year 11(only like the motion/energy stuff, which it seems u r familiar with, electricity and maybe some electromagnetic wave but it takes like 4 seconds to learn. One thing to be aware of is that AOS 3 is a big prac which you have to design, create and discuss yourself, this is something the people who did the 1/2 have done before and so may already have this set up, the prac can be as easy as dropping a ball and measuring efficiency but it better be a damm good report to get a very high grade with that lol. It is not that big a deal but definitely something you should think about and be aware of sinc eu lack experience writing a report (easy to learn tho). Beyond that tho, you really won't be at that big a disadvantage, especially with ur capabilities with math and the fact that a lot of the content is new for everyone. The physics stuff is definitely clear and personally, way more fun. (shhhhhhhhhhh), bio can be a messy subject in terms of order and how content but doesn't mean it ain't fun.
In the end, the last thing to consider would be what ur planning to do after year 12, how much you are enjoying bio and maybe chatting to the physics teacher to see what they are like, mine is great (also my spec teacher) so the subject is a heap of fun and easy grades. I have almost the exact same work load as you (Physics, chem, spec, methods, English Language, Italian and religion and society 3/4) and i think it is manageable. Before u talk to ur teachers, take a peak at the study design to see whats up, but don't stress about some of the language or weird formulas they r easier than they look (most of the time). Or even try to drop bio 1/2 and pick up physics now in the mid semester.