It all depends on the subjects that you take. In some subjects the end of year exam is worth 33% of your study score, while others can be 50% -70%.
To be blatantly honest, doing poorly on exams does affect your ATAR considerably. I'm not entirely sure why this nervousness happens (trust me, I'm exactly the same) but I find it more difficult to sit in a classroom with all my peers and us doing a 3/4 SAC which counts 10% rather than in an isolated exam with phony examiners.
I'm not entirely sure how the system works (if you get a high sac score, and a poor exam score, does your exam score get pulled up to the mark of your sacs? )
But yes, I would imagine that it does affect your ATAR and will. Honestly, I know you have been struggling with this your whole life, but there is always a way to overcome fears;)