Although I strongly feel that VCE physics doesn't test real physics ability and is strongly watered down, it provides a basis (albeit simplified) for understanding the way in which the world works. The course itself isn't hard - you either understand it, or you don't, and if you comprehend the concepts you won't find it hard. However, you've got to realise that most of the people who take physics enjoy the subject - it's not a prerequisite for too many degrees.
I'm only doing 1/2 Chemistry, but 3/4 Chemistry seems quite analytical. I'm not sure why most people find it so difficult, but keep in mind that a lot of people who take Chemistry only study it because it's a prerequisite for the health sciences. This means that you have a lot of people who may not enjoy the subject studying it so that they can pursue specific careers. I know that this happens a lot at my school - there are students attaining 40%s in 1/2 with the intention of undertaking the subject next year because they want to be doctors. This might be why Chemistry's developed a reputation for being "hard" - some of the students who study it don't have an interest and struggle with the concepts. On the other hand, it does seem to require a little more understanding than physics at VCE level...