Although I strongly feel that VCE physics doesn't test real physics ability and is strongly watered down, it provides a basis (albeit simplified) for understanding the way in which the world works. The course itself isn't hard - you either understand it, or you don't, and if you comprehend the concepts you won't find it hard. However, you've got to realise that most of the people who take physics enjoy the subject - it's not a prerequisite for too many degrees.
I'm only doing 1/2 Chemistry, but 3/4 Chemistry seems quite analytical. I'm not sure why most people find it so difficult, but keep in mind that a lot of people who take Chemistry only study it because it's a prerequisite for the health sciences. This means that you have a lot of people who may not enjoy the subject studying it so that they can pursue specific careers. I know that this happens a lot at my school - there are students attaining 40%s in 1/2 with the intention of undertaking the subject next year because they want to be doctors. This might be why Chemistry's developed a reputation for being "hard" - some of the students who study it don't have an interest and struggle with the concepts. On the other hand, it does seem to require a little more understanding than physics at VCE level...
appian don't you think you're being a bit harsh on your classmates? I don't think the reason chem is considered harder has anything to do with it being a prereq for health sciences. I think if they really want to become doctors they'll work hard and do well anyway because there's a reason it's a prereq.
I think chem is harder than VCE physics basically because of the structure of the two courses. Physics skims the surface, and doesn't really try to go that deep, and while chem doesn't go that deep either I personally just find chem exams more fun than physics exams. Like in unit 3 where you look at the spectroscopy and try and deduce the structure. I found extra ones on the internet they were pretty fun to do. I think that if you like chemistry, and enjoy it, and you reckon you can put the effort in for units 3+4, you'll find it to be perhaps a more solid earner in VCE than physics. Because you can cement your concepts in physics - but that's not that difficult to do, and most people can do it. As shown by the A+ cutoff for this year's physics midyear, there's a lot more chance at play in determining your physics study score that goes beyond your knowledge.
At the end it's a personal choice. As some people have noted, don't assume success in 1/2 translates to 3/4. But do consider that you have set a strong foundation for yourself and if you keep working then you may indeed do very well at VCE chem. And it also depends on whether you want an easy subject, an interesting subject, or a solid study-score earner.