First off, lemme say I'm neither for or against having a HR Act. But seeing as you need reasons agasint it...
Simply put: we don't need one. We HAVE rights. They're found through Common Law and various Acts of Parliament, like the Equal Opportunity Act 1995 (Vic) and the Crimes Act 1958 (Vic). And of course, in the Constitution. The entrenched rights that we have in the Constitution can't just be changed through a simple Act of Parliament -- you need to have a referendum and that's no easy process. Yes, admittedly having a bill would mean that all our rights are found in one place and it'd be nice and easy to find. However, there's also bad things about it too:
Setting rights down as specifically as they would be in a Bill of Rights -- well, society's views and values change over time. Then what happens? Lets take the US for example. The right to bear arms was perfectly fine and relevant in 1791... but now we don't really need it anymore and it's caused more problems than it's solved, but yet, it still exists.
Having rights in Common Law and through various Acts of Parliament is better for this reason -- when the need arises, we can alter the rights that we have so that it suits better. Law should be rigid so that it's all consistent and so on BUT it also needs to be able to keep up with the times.
Also, the rights that we have in the Constitution CANNOT be limited, under ANY circumstances. South Africa's Bill of Rights, although it's in the Constitution, some of these rights can still be limited if the Government sees fit, which kinda destroys the point of having the rights in the first place.