It's when you occupy yourself with so much content, techniques, things to learn etc, that your mind becomes quite overwhelmed. You become irrational. You get stressed and the stress takes its toll. Your mind needs a break as well as much as your body does from activity. It's like running a marathon only you keep on going when it's over - you'll get exhausted and eventually will collapse from fatigue. So I suppose it's like fatiguing of your mind. You hear stories of people going 'blank' or having 'mental blanks' when they see the exam - it's one of the side effects of over-doing it. It's the running joke that you see the first question and go, quite literally, wtf? Your thoughts are all jumbled up that when you see a question a whole lot of things you've studied jump at you in a completely random order and it makes no sense.
Eventually if you do get to this stage, simple questions trip you up and you just cannot think logically and hence make 'silly mistakes' that you really shouldn't otherwise be making. It happened to me for unit 3 - all I wanted to do was make my Physics cheatsheet (Which took forever with the amount of content I wanted to fit) and do as many past exams/Checkpoints questions, which wasn't fun, strenuous, tiring and very demanding. When it came to the exam well, let's just say there wasn't much energy left to give the exam a good crack.
That's why something as simple as taking a walk or getting some fresh air in between study is so helpful - it lets your mind unwind itself a little and rest itself temporarily. So when people claim they have 'burned out', they have pretty much over-studied. It goes by the saying that it's better to study efficiently, rather than longer, and that studying longer does not mean you've gained more than you normally would. It's quality vs quantity.
Also, something pretty important to understand in VCE, is that you actually can take a break without compromising your ATAR. You can go take some time off for yourself, your life for the year isn't completely dominated by VCE (Although it mostly is). Just don't get too carried away.