I'm assuming your having trouble with b:
if we let C and D be the points such that OC=a-hat, OD=b-hat we see that we want the vector that bisects angle COD(as COD=AOB). However the triangle COD is isosceles (as |OC|=|OD|=1) hence we just want a vector that bisects the line CD. if M is the midpoint of CD then OM=OC+ 0.5CD. Figure out what this is, then "scale it" to a unit vector. (draw picture for clarity's sake).
Also, what does it mean for 3 vectors to be mutually perpendicular to each other?
Pretty sure that means "pairwise perpendicular". ie: any pair of two vectors out of the three is perpendicular, like i,j,k for example. As opposed to something like say i,j,2j; where not all pairs are perpendicular (the pair j,2j in this case) even though some are (like i,2j)