It's true Specialist Maths has a bit of Physics, as in the calculations and concepts anyway (like kinematics), however, it is still very different from Physics itself. As the name suggests, its all about the Maths, you don't need to like Physics to be able to do it!
Besides, you don't need to look at Specialist Maths like it has Physics, just think of it as another unit of maths, with coincidentally crossovers and similarities to certain components of Physics.
Personally, the only reasons why you would do Specialist Maths is because A) you like maths B) you have nothing else to do or C) scaling (aww yiss). I'm biased but I think it's a really good subject to take advantage of (especially the scaling, not going to lie). I don't think it's much more difficult than Maths Methods, but that's my opinion.
It is sort of applied maths, but definitely not more or less applied than methods, in my opinion, and no where near as applied as Physics. Take a look at one or two past VCAA exams, and see what you think

I think it's a great maths subject, and I really enjoy it right now! (Or i hate the rest of my subjects, one of the either)