Specialist maths is one of the most competitive subjects in VCE.
To put it into perspective, here are a few rough figures: there are 40,000 people in the state doing maths in VCE. Out of them, only 4500 are good enough to do specialist maths, and out of them, only 400 score a 40+ raw score. So you would have to be in the top 1% of VCE maths students to score a 40.
Having said that, it is definitely achievable if you are a strong maths student. I don't want you to feel discouraged, but at the same time, I want you to be aware of what you're up against.
For me, I have done roughly 120 hours of specialist maths studying this year (which equates to about 45 minutes per day) and I have scored 100% on every SAC so far.
If you're already doing 3+ hours of studying each day by year 10 level, then you should do very well in VCE. Just remember to study the correct things. For example, if you are going through your textbook in chronological order and you come across a topic that you know really well, then skip it. There's no point wasting your time repeating something basic which you already know (eg. basic algebra). Study effectively and you'll be set for VCE.
Good luck
Are you sure about those statistics? I'm fairly sure that for most subjects with substantial cohorts (Spec. being one of them), 40 raw = top 8-9% of the cohort.
EDIT: @Op, to answer your question, I'm a firm believer in the saying that if you have to ask "how much studying" you should be doing, your already doing it wrong. You should be doing exactly as much studying as it takes to you be comfortable with the content.
I agree with Paul in that you should be doing the most possible revision in the least amount of time, now for me personally, during exam preparation time or the time where every SAC seems to occur at once that can amount to 3-5 hours a night either doing revision or practice exams - during the term where I only have the occasional SAC and exams are still a while away, I do just enough to stay 1-2 weeks ahead of the class (or in the case of Biology last year where we fell pretty far behind, to stay far enough in front of the course that I'd have ample time to prepare for the exam).
How long that takes me depends on the difficulty of the content we're covering, although I have to say that whenever I feel like I haven't done enough/can't be bothered studying my "hard" subjects I just do accounting lol, it's good to have one of those subjects you can do work in when you're feeling guilty about not doing enough.
Ultimately, your in year 10 - it's good that your worried about this now because many people either go into Spesh or drop it for incredibly stupid reasons, but with the maths knowledge you know how and the relatively huge period of time between Year 10 Maths and Year 12 Spesh you're not really in a position to make any decisions on how much you should be working. Take GMA next year, see how you go in all your 1/2 subjects, and you'll know yourself how much you need to work to achieve your aims.