Hey SnekiSnek!
I do agree quite a bit with the fact that Unit 3/4 is where the subject really gets interesting, however, there is some assumed knowledge that you will need to know to catch up.
If you look at Unit 2, you will need to understand basic electricity such as adding resistors in different circuits and getting familiar with concepts such as finding current, power and voltage (This comes in handy for Unit 3 AOS 2 Electromagnetic Induction).
You will also need to catch up on everything motion-based. Some topics that I would definitely recommend to study before moving into Unit 3/4 would be everything kinematics related (constant acceleration, Newton's Laws of motion, energy relating to gravitational, kinetic and elastic, momentum/impulse, work and the difference between vectors and scalars should be plenty).
For VCE Physics, the maths knowledge required is pretty much Year 10 Maths (To account for the people that may not be studying maths at VCE level). It is definitely not hard to 'cram' all these topics in. I think if you talk to your teachers they may be able to recommend some work you do prior to starting 3/4. There is also these worksheets
electricity and
kinematics that I highly recommend you try as it covers everything you will need for 3/4.
Physics can be a hit or a miss depending on the person. I personally would say that VCE Physics is one of those subjects that you really learn how to understand, rather than using formulas. If you like to think conceptually and can trust 'the physics', I would definitely recommend this subject!
Hope that helps