You absolutely should.
There's really no other way about it. Maths is critical to understanding a lot of higher level chemistry (especially physical chemistry), and you'd be surprised how often it pops up in biochemistry, especially if you wind up doing biophysical chemistry - and that's ignoring statistics. To be honest, I'm pretty surprised that the chemistry department hasn't made first year maths compulsory yet for the major, when I did third year chem it was not possible to do some questions without calc 2 knowledge (triple integrals) and linear algebra made many topics in second and third year much, much easier.
Maths is hard, for sure, but you'll be giving yourself a definite edge in later years. All you need is calc 2 and linear algebra, plus calc 1 if you didn't do specialist.