As a year 11 student your scores are calculated under the new system. You'll get an ATAR calculated from your 5 best subjects (after scaling) [or it can be four general subjects + a VET subject but this doesn't seem to be your case]. English is the only subject type were you have to have passed at least one of them, so failing/dumping spec is fine - they'll just use your other subjects for your ATAR calculation.
Units 3&4 go to the ATAR but units 1&2 are also important for eligibility for the QCE (even though they don't go towards the ATAR).
To be awarded the QCE you need at least 20 credits (so make sure not to fail your other subjects & talk to your teachers if you think you're in danger of that. There's also rules abut where these credits come from but you don't need to worry about these
) and to meet the literacy and numeracy requirements. If spec was your only maths you may want to check how you would meet the numeracy requirements.
TLDR; ATAR calculation will be fine but you should check that you're meeting all of the QCE requirements
I know this is all a bit complicated and much to take in so let me know if any of this is unclear and/or you have more questions 
Edit: improved clarity
I think this is how you reply but thank you thank you very much for clearing it up

. I'm also taking math methods so my numeracy should be alright.
I just want to clarify that the assignment and test I just had for Spesh was only based in UNIT 1. We are currently starting UNIT 2 but I'm gonna ditch the subject next week and move to accounting.
I was just wondering if the qce points are counted like
UNIT 1 = 1 QCE point
or UNIT 1 + 2 = 1 QCE point
per subject
If it helps my subjects are:
English
Methods
Chem
Physics
Art
Spesh (--> accounting)
I have passed the rest of them

assuming I pass all these for the rest of the year (end of grade 11), how much would my qce points roughly around? Anyways, still thank you very much

.