If you want to do proper physics, do Specialist Maths at VCE and then do Physics at Uni.
But don't do VCE Physics unless you enjoy pictures of playgrounds and see-saws 
I would think that not doing VCE Physics would make uni physics quite a struggle, especially seeing that Specialist Maths only really focuses on statics/dynamics emphasising the maths and not the concepts. You'd be missing out on understanding a lot about newtonian mechanics, about electricity, about electromagnetism and about light etc.
Doing specialist maths if you want to do physics at uni is probably a good idea though... as far as I can tell with my limited experience of 5 weeks into first year physics anyway.
Physics is a whole heap of fun. What you learn in Units 1&2 only really helps in Unit 3, so you won't be at any significant disadvantage. It could be harder if you start in Unit 3, but it's still easily doable if you put in the effort to quickly go over what you missed. I definitely would recommend you give it a shot.
I'd probably agree strongly with this. Spend an hour or two looking over the entire textbook quite closely yourself (try out a few questions etc.) and then decide if VCE Physics will be what you think it is and go for it. It does have quite a lot of flaws (e.g. see pi's 'unbiased' review/why people would tell you not do the subject), but for the most part there is a lot of interesting stuff that you can learn about in the subject. For the most part you'll find that the interesting stuff (well for myself I found reading about all the concepts/theory and trying to understand it being the interesting part) is up to yourself to study though, with the textbook explanations being often... quite dense and
a lot of exam questions (and hence the focus of the course) being... not as interesting.