Ok... I don't do accounting... but you’re the best judge on whether you’ll enjoy the subject/ be capable of it

Make sure you look thoroughly over the syllabus before you switch, because you may end up finding you actually prefer physics. (I did this with a couple of my IB subjects LOL, I switched subjects and found that I'd actually prefer being in the ones I was in previously.)
Also, as for the work component, although I do not do accounting, I believe all subjects require hard work and a lot of effort. Don’t listen to people who say, “oh, you don’t need to do ANY work for [insert subject here]!” because it’s often not true. It’s more satisfying to try your hardest in any thing you do, and you’ll achieve a lot more!

Good luck!