You definitely need to sit down with someone (career counsellor?) at your school and sort this out. I'm not sure what options your school would offer, but I highly doubt not doing any English is a viable option.
Just to check, are you currently not enrolled in either English or Lit? If you've started Lit this year already it should be fairly easy for your school to slot you back in; otherwise I'm sure you could be put into a one of presumably many mainstream English classes. Not only to most courses require a pass, but I'm under the impression you need an English subject to even get an ATAR.
As an absolute last resort, you could look into doing English via distance ed. but that seems unnecessary if you've got a school that can cater for you. Generally speaking dist.ed. subjects require a lot more work anyway because you don't have the same access to resources that a school can provide.
With regards to fears of not passing, don't worry, it's easier than you think! Giving up on an entire subject after only a few weeks of studying it is definitely not the way to go about this. Objectively, you'll probably prefer English and it'll be easier to score well because it's such a huge cohort. Literature tends to appeal to students who are already good at English, so it's usually considered more fiercely competitive.
So assuming you try and get back into English, your entire year will consist of learning how to write three essay types, most of which will already be familiar to you from previous years. The oral (which I'm assuming is difficult becuase you don't like public speaking?) is just a single SAC. It's over in 5 minutes, and it's worth the least out of everything. Most teachers don't mark harshly because they know how tough it is, and I'm sure if it was sever enough you could have a word to your teacher about your anxieties then they'd be prepared to work out an alternative arrangement (ie. you just perform in front of them rather than a whole class - which you'd probably have to do anyway if you were transferring in so late.) Once you learn how to hit the criteria, it's really not hard to do well; it's only people on the far ends of the bell curves that have to struggle. If you're far enough to the left (ie. can't write essays, perhaps should be in EAL but wasn't eligible, don't understand the texts, etc.) then just scraping a pass can be a challenge. By contrast if you're on the far right, the efforts needed to move from a 9/10 to a 10/10 are monumental compared to, say, a 4/10 to a 6/10.
When you were doing English in Year 11, what aspects of the course did you find difficult?