It can proc off both the main hand attack and the off hand attack, if there is a sword or axe equipped in the appropriate slot.
The extra attack is always done by the weapon in the main hand. Thus, if you have a good fist weapon, it is a good idea to spec fist and sword/axe, and keep a fist weapon in the main hand and a sword/axe in the off hand. In that situation your extra attacks from sword/axe specialization procs will be done by the fist weapon with +5% crit chance.
Patch changes
Patch 3.2.0 (04-Aug-2009): Sword Specialization: This talent is now called Hack and Slash and applies to axes as well as swords.
Patch 1.3.0 (07-Mar-2005): Added a visual to Sword Specialization "proc" that will now show up in the combat log as well.