A nuke generally refers to an effect that causes ranged magic damage (resistable) as opposed to physical damage (defensible), but some call Hunter attacks nukes, since they may include a magical effect with their physical damage.

Additional info:

• Damage dealt by a nuker (caster).
• A spell that deals all or most of its damage immediately rather than damage over time (such as Fireball or Wrath).
• Channeled spells like Arcane Missiles can be referred to as nukes, so the distinction blurs slightly; nuke can mean any heavy damage spell.
• Alternatively, any damage spell at all; mages are called "nukers".
• Overnuking is where a player deals so much damage (therefore generating excess threat) that it pulls aggro away from the tank.