I reckon it depends what you want physics for.
If you're interested in an ENTER - I would drop it. There is little point stressing yourself out with an extra subject (which, as your worst subject, will probably not even make it into your top 6). Specialist covers some of the same material (if you're interested in overlap to reinforce), but generally only the boring stuff and the spesh approach is a little different. If you want to improve with physics questions, you need to practice practice practice practice!! Heaps of questions kind of become the same after a while.
For the sake of broader learning, I would say physics is one of the rare subjects where VCE provides some good intuitive knowledge of concepts and is worthwhile doing.
If you want to do further physics study in university, I would definitely do VCE Physics since you get rushed a lot more through concepts in uni and good foundations are pretty vital.