From WoWWiki

Damage Per Second (DPS) Formula

The general formula for your weapon's DPS is:

((Min Weapon Damage + Max Weapon Damage) / 2) / Weapon Speed

So, for example a [Sand Polished Hammer] with minimum damage of 97, a maximum damage of 181, and a weapon speed of 2.60, the formula would be:

DPS = ((Min Weapon Damage + Max Weapon Damage) / 2) / Weapon Speed
DPS = ((97 + 181) / 2) / 2.60
DPS = (278 / 2) / 2.60
DPS = 139 / 2.60
DPS = 53.46
DPS = 53.5

Some weapons also have extra damage, e.g. "Holy Damage". This should be factored in by adding the min and max values to the the above averaged weapon damage. Suppose the above hammer dealt an additional 20 - 40 Holy Damage. Then the formula would look like:

DPS = ((Min Weapon Damage + Max Weapon Damage + Min Extra Damage + Max Extra Damage) / 2) / Weapon Speed
DPS = ((97 + 181 + 20 + 40) / 2) / 2.60
DPS = (338 / 2) / 2.60
DPS = 169 / 2.60
DPS = 65.0

Formula for: Tooltip DPS & Damage Range:

Damage Ranges (for melee):

Minimum Range = (((Minimum damage / Weapon Speed) + (Melee Attack Power / 14)) * (Weapon Speed)) * (Dual Wield Penalty)
Maximum Range = (((Maximum damage / Weapon Speed) + (Melee Attack Power / 14)) * (Weapon Speed)) * (Dual Wield Penalty)

Note: Dual Wield Penalty only applies to off-hand weapons and can change depending on how you are spec'd.
Example: A rogue with dual-wielding specialization at 5/5 would use 0.75 as the penalty value, for dual-wielding specialization 0/5 you would use 0.5 as the penalty value.

Damage Ranges (for ranged):

Minimum Range = (((Minimum damage + Scope damage bonus) / Weapon Speed) + (Ranged Attack Power / 14) + Ammo DPS bonus) * Weapon Speed
Maximum Range = (((Maximum damage + Scope damage bonus) / Weapon Speed) + (Ranged Attack Power / 14) + Ammo DPS bonus) * Weapon Speed

Note: "Weapon Speed" is your actual fixed value on your weapon, not your Attack speed.

Tooltip DPS

DPS = ((Minimum Range + Maximum Range) / 2) / Attack Speed

Note: "Attack speed" is not your fixed value on your weapon. It is adjusted for any speed enchants/buffs/procs.

Example #1 (melee):
(Rounding error is considered in all formulae.)

1354 Attack Power
5/5 Dual-wield specialization = 75% damage from off-hand = 0.75
Slice and Dice activated (30% speed increase)

Main Hand: [Vindicator's Brand]

Weapon speed: 2.60
Minimum damage: 147
Maximum damage: 275

Minimum Range = ((147 / 2.6) + (1354 / 14)) * 2.6 = 398.46
Maximum Range = ((275 / 2.6) + (1354 / 14)) * 2.6 = 526.46
Attack Speed = 2.60 / (30 / 100 + 1) = 2.00  (would be 2.60 if no haste effects were on)
DPS = ((398.46 + 526.46) / 2) / 2.00 = 231.23 (would be 177.87 if no haste effects were on)

Off Hand: [Fireguard]

Weapon speed: 1.60
Minimum damage: 94
Maximum damage: 176

Minimum Range = (((94 / 1.6) + (1354 / 14)) * 1.60) * 0.75 = 186.56
Maximum Range = (((176 / 1.6) + (1354 / 14)) * 1.60) * 0.75 = 248.06
Attack Speed = 1.60 / (30 / 100 + 1) = 1.23 (would be 1.60 if no haste effects were on)
DPS = ((186.56 + 248.06) / 2) / 1.23 = 176.56 (would be 135.82 if no haste effects were on)

Example #2 (ranged):
1323 Ranged Attack Power
5/5 Serpent's Swiftness = 20% increase in Ranged Combat Attack Speed
[Clefthoof Hide Quiver] = 15% increase in Ranged Combat Attack Speed

Ranged Weapon: [Valanos' Longbow]
With a [Adamantite Scope] attached (+10 damage)
Firing [Halaani Razorshaft] ammunition (Adds 34 DPS)

Weapon speed: 2.80
Minimum damage: 130
Maximum damage: 242

Minimum Range = ((((130 + 10) / 2.80) + (1323 / 14) + 34) * 2.80 = 499.80
Maximum Range = ((((242 + 10) / 2.80) + (1323 / 14) + 34) * 2.80 = 611.80
Attack Speed = (2.80 / (20 / 100 + 1)) / (15 / 100 + 1) = 2.03
DPS = ((499.8 + 611.8) / 2) / 2.03 = 273.8