If you are an average English student, stick to mainstream English. Literature is for students who like reading a lot of fiction. You have to be really good at text response essays, understand the author's intentions, character interactions and personalities. In general, you have to be able to develop an in-depth understanding of all the novels you read in the course in order to excel in this subject. If you are good at this stuff, then go ahead.
A lot of people underestimate the difficulty of English Language. There is a reason why it is the most difficult English subject. If you are a top English student and you are passionate about English, then English language is a possibility. If you're finding it difficult to score high marks in your persuasive analysis tasks and essays in mainstream English, then switching to English Language is not recommended.
This is my personal opinion. You could try out all three English subjects before making up your're own mind. Check out the VCAA study designs for the English subjects and see what interests you. After all, you are more likely to perform well in a subject you like. If you don't like English at all, stick to mainstream English because you really don't have a choice.
Good luck buddy