Didn't do physics in VCE as well. In Monash, cruised through foundation physics with close to 0 hours of study throughout the whole semester, but when I joined mainstream physics in semester 2, I got torn apart..... Would be lucky to scrape through with a pass.
The huge jump in the difficulty of physics hits when you transition from Foundation Physics to mainstream physics, as you compete against other physics students who has a 2year advantage on you. You get lecturers and tutors assuming that you already know said formula and theory and it's just utterly confusing...
That said, we had 4 lecturers for physics in Monash and 1 was great, 2 was reasonable and one was bad (but isn't lecturing any more next year). I wouldn't say that the teaching quality is bad, but the jump in the level of difficulty between foundation physics and sem 2 physics is HUGEEEE.