physics is similar to spesh in that you study motion in both. However in spesh its using calculus a lot more, and in physics your using pre-determined formula to solve pretty simple(relative to that in spesh) problems.
spesh doesnt really help with physics because by the time you study motion in spesh, you dont need it for physics anymore(unless your super speedy at finishing the spesh course). However doing physics helps with spesh because there is a bit of constant acceleration(from physics) in spesh towards the end, so you've got an understanding already.
Which one is easier? well thats probably physics for most people, it has a lot more simple maths and the concepts aren't that hard. Also it scales less, so thats a bit of an indication as well.
And no, you dont have to be great at maths for physics 3/4, it helps if you are, but theres no significant disadvantage of not being a maths god. One of my friends did physics and his only maths subject was further, and he understood the maths in the course just fine.