There's not really any correlation between VCE and AMEB - 5th grade practical means you should be able to play a bunch of the pieces, and 3rd grade theory should mean you'll be fine with most of the theory (don't remember what AMEB theory includes, but VCE performance theory is pretty not hard, so at the very least you'll pick it up reallll quick with that background.) VCE only requires one instrument - if you do group performance, you can choose to play multiple, though. Group also has a larger emphasis on performance as opposed to solo, wherein solo there is also technical considerations to your performing.
The real problem lies in that VCE assesses different things - in terms of your practical, AMEB is a lot more technical than VCE, whereas VCE places a lot more emphasis on performace and sounding good. (I know of singers who have gotten perfect marks in the performance exam without knowing what a scale is) Also, some of the theory in VCE is more about musicology than practical music theory - to memory, its about 1/3 aural work (identifying rhythms, melodies, scales, chords, progressions, etc.), 1/3 written (writing scales, chords, etc. in your chosen key signature) and 1/3 describing (in terms of things such as rhythm, texture, tone colour, etc.) Also, some instruments are a lot more competitive than others, as bangali has mentioned (piano and guitar being the two most competitive instruments)
I will ask, though - have you considered investigation? Investigation defs doesn't require having done VCE before, you get a much broader selection of songs you can perform, and IMHO is just a lot more fun.