There is one compelling argument for keeping VCE English over VCE English Language:
The cohort in VCE English Language will tend to be more fortunate on average. The schools that provide it generally have the surplus of resources to do so, and it's not farfetched to believe there is a correlation between fortunate schools and smart students. This hurts your rank in VCE English Language. VCE English, on the other hand, contains pretty much the whole state, perhaps minus a few fortunate students who can do Literature or EL instead. This helps your rank -- and no offence to the state, but it's easy to outperform. This is also the reason why MHS makes VCE English compulsory. To do EL or Literature, you must do it in addition to VCE English.
Although, rank effects are not necessarily going to outweigh the effects of the enjoyability of a subject, I think it would be safe to keep VCE English.
Ultimately you have to weigh up the pros and cons -- and for the future, wonder why this weigh up now differs so drastically from the weigh-up you did in the start of the year. For example, were your expectations too high?