To hack websites, even by defacing phishing pages, could maybe seem funny but is still NOT cool (a.k.a : legal).

A less intrusive move to help is to notify people involved (owner, registrar, hosting provider, anti-phishing teams and toolbars...) and law enforcement agencies.

And for those who don't know The BOFH, here is the Wikipedia entry.