Just checked. There are only 2 places that offer a Bachelor of Primary Teaching directly, being Swinburne (which only needs a 25 SS in English or equivalent/ 30 SS for English) or Deakin (which needs some form of maths + same English requirement). All others are labelled slightly differently (Childhood and Early Years, Education - Primary Teaching, etc.), where a good portion of these include the same English requirement and a requirement of some type of maths (including either 2 of U1/2 General, Methods and Spesh OR a U3/4 of any maths). Psychology generally doesn't require maths, however, if you're trying to do so under a Bachelor of Science, there are a number of institutions which say you have to do a little maths as their requirement.
Bottom line is please keep going with maths. You'll probably need it to open up your chances.