Hi there,
Personally, I loved both VCE Physics and Specialist, albeit for different reasons, and think you'll greatly enjoy both. The challenge provided in spesh is unlike most other subjects, and its really rewarding to master those difficult skills. On the other hand, the questions in VCE Physics (and the math) are comparably straightforward, but the concepts covered (relativity, electromagnetism, wave particle duality etc.) are super interesting.
With regards to your main question, I don't think you'll be face any disadvantage that you can't overcome picking up 3/4, as a lot on Unit 1/2 focuses on topics not covered in the 3/4 study design, such as circuits and thermodynamics (so don't worry about these). What you will have to catch up on is motion, which includes straight line motion (constant acceleration/suvat), horizontal and vertical circular motion, as well as momentum and energy transformations (kinetic, elastic, gravitational potential). None of this is too hard, and there are ample resources online to help you get a grip on it (feel free to post any questions you have here on ATAR Notes!); so I'd focus on that over the summer break. Having done well in spesh won't directly benefit you (although if you've already done motion in spesh that would help), as VCE physics is more concept based than tricky mathematics; most of the maths is simple, however it does suggest VCE Physics and catching up on 1/2 work will be within your grasp.
If I was in your position; I'd probably keep specialist, as I can't really see VCE physics giving you any significant advantage going into uni as most of the course can be picked up relatively easily, unlike the mathematical skills learned in spesh, which may be harder to develop. That said, if you're not enjoying spesh, or would feel more comfortable going into uni with the foundations of physics, go for it, you won't be disappointed (as long as you don't expect to do much maths in VCE physics!)
TL;DR: Both are great subjects, you'll be able to transition to 3/4 fine if you brush up on motion (read the study design), but don't discount the valuable skills you could learn in specialist either; they may actually be more beneficial going onto further education.