Personally I think there are realistic limitations.
If a person has struggled with maths all their life, and never fully understood basic concepts even with help, then yes I think they would struggle with methods, spesh and uni maths. I don't think that people who tried hard for Cs and Bs in middle school maths will magically get A+s in spesh. Some people just don't get maths, no matter how hard they try.
However, for an "average" person as you say, I think it is possible. I don't find any difference in difficulty between methods and spesh. Even though spesh is meant to be more difficult, the concepts aren't that hard to understand if explained well and if you do enough practice. I know naturally intelligent people who are failing spesh, and "average" people getting As. I think it is a combination of determination, hard work and, yes, intelligence.
But I think natural maths intelligence is probably needed for 45+.