Physics is a different story. I find it boring and I have no intentions of doing a course that involves physics (engineering etc) at university. My class has already finished most of motion/energy/Newtons laws this semester, and I'm not finding any of it engaging.
The general rule I follow is that I only pick subjects I enjoy and am willing to put in the effort.
I can really only talk for physics, but you'll probably hate Unit 3 if you don't enjoy motion already. Relativity was good fun, best part of the physics course, but most schools pick S&M for the detailed study, so that's more motion-type stuff. Electronics & Photonics was extremely lightweight area of study.
I'm assuming you'll be doing spesh, as you're doing GMA this year? Specialist will probably give you a more enjoyable experience with motion - as the problems are a bit more complex than the lame VCE physics stuff. Unit 3 motion does go a bit beyond Unit 2 (projectile, circular, gravitational motion), but still pretty toned down stuff. I do think that it's the VCE physics course making it boring, keeping the mathematics simple, having to revise thoroughly to ace the exam. I find motion a lot more interesting when I'm not bound by what we have to do in the classroom - in the middle of year 12 you'd probably not have as much time to waste on stuff like that (or seeing as you said you don't find the subject interesting, you might not bother with it).
Unit 4 I find more interesting (especially the light and matter stuff), I don't know if you will though. Most people seem to hate it.
Physics does require a bit of effort and if you're not willing to put that in, you might not get as much out of it. Uni maths seems 100x more interesting though. I would think more interesting is really what matters. If you find a subject more interesting and enjoyable, you'll put more effort in which would (hopefully) result in better grades.
It wouldn't be worth doing physics if your only reason was to get a high score. The same thing would go for uni maths.
I guess that Mark Twain quote would be appropriate: "I never let my schooling interfere with my education"