I can definitely empathise and understand your concern. Between the restriction on maths subjects and the compulsory English subject, there was always the possibility that I would end up with my two lowest-scoring subjects in my primary four, and that would have really annoyed me.
With that in mind, I intentionally neglected Specialist this year (since I did well in Methods last year, I assumed that Methods would be in my top four, which actually didn't happen). I know it feels wrong to neglect a study, but just think of it as a sacrifice.
If you can manage to study a lot for all your subjects, then go ahead. But otherwise, prioritise what's going to contribute more to your ATAR. I dedicated half the Term 3 holidays and all of my exam period solely to English Language, and I just managed to make it my 4th highest subject score.
Don't worry about it too much, though; just keep it in mind when dividing up your study time.