This is especially useful in instances, but even with the new 3 minute cooldown, it should not be used as a staple of holding aggro. It is best used if a given fight looks to be getting out of control, where multiple enemies are starting to split off and attack different group members, or unexpected adds join in and start attacking the squishies. Ideally, skillful use of the much faster cooldown single taunt and rapid target switching should be enough to keep everyone focused on the main tank in normal fights.
As with Taunt, this ability has little effect on enemies already attacking the warrior beyond preventing them from switching targets for the duration, but unlike Taunt does not actually create fresh threat itself. Enemies will switch back to their original targets after 6 seconds, unless fresh threat is generated by other means.
It has no real use while soloing, and like Taunt, has no effect on enemy players in PvP.
Like Taunt it will affect player pets, allowing for a limited usability in PvP, such as pulling pets off of a caster.
Specifically a party-only ability, Challenging Shout is used in only one situation: A party member has generated a large amount of aggro and they need to be pulled before they die. It can also be used if a single mob has resisted Taunt and/or Mocking Blow and you need to pull that mob off your party member.
With the new three minute cooldown, this means that Challenging Shout can be used every third pull in an instance (assuming a minute for each pull). Because of this, warriors don't need to worry so much about the cooldown, but it should still not be used willy-nilly.
Patch 2.3.0 (13-Nov-2007): The chance for challenging shout to land is now affected by hit rating.
/Patch 3.0.2 (14-Oct-2008): Cooldown decreased to 3 minutes, from 10 minutes.