I "technically" did 4 VCE subjects in year 12, although, I did add on a MUEP subject so really 5
As others have said you can definitely only do 4 subjects in year 12 - that will give you a total of 5 vce subjects completed by the end of year 12. This will impact on the potential of your ATAR (if you are going for high 99s) but other than that this can be an effective way to maximise your time studying since the max aggregate addition you could've got would only be 5 (since they only take 10% of your bottom 2 subjects). So, for most people that 6th subject would actually only be 3-4 points since not everyone gets 50s/high 40s in all their subjects.
Anecdotally I know of a few people who only did 5 subjects but they still managed 98+ ATARs. So you can still do really well if you ace those 5 subjects.