Specialist isn't really harder or easier than methods - it's different. Often people will say it's harder because you need methods knowledge to do it, which ends up making methods easier, however if you look at the content blankly and consider each exam differently, they're no harder or easier than the other. Similarly, physics is different to specialist, you can't really compare them like that. (granted, I didn't actually do VCE physics, I just did Monash's crappy bridging unit)
If you're unsure what to do after uni, I'd say do physics so you still have a physical science in case you decide to do science at Monash or Melbourne (although I think Monash accept methods as a relevant science VCE unit). However, if you like maths more than you like physics, or BusMan more than you like physics, choose those instead. You should always choose what you enjoy.