Have you seen Harvard/Yale/Princeton? (If you have, ignore the rest of this paragraph) I went to visit New Haven last year and I absolutely loved it, it would be a wonderful place to study, I didn't really like Boston/Cambridge very much though, strangely. But yes, New Haven is actually quite a lot like Canberra except the buildings are nicer/older. I'm sure if you visit you would become excited about the idea of studying there!
In the end it's totally up to you and there are lots of things to consider, but I'd totally go for it. And, if after your first year you don't like it, it would surely be easy enough to move back. You can take a year of absence at ANU rather than leave out right, so coming back wouldn't be difficult.