First, you need to understand that a great number of casts is not needed. We are not doing statistics, we are using the games mechanisms to analyse spell damage more efficiently.
In order to test the coefficient of a spell, we need to look at the range of damage the spell can do without any spell power. The best way to do this is to go to a database site, find the relevant rank of the spell, and look at the tooltip.
Then, in order to test efficiently, make sure there are no buffs that will soon run out. Make sure the amount of bonus spell damage will remain static. Take away any items which may proc bonus damage effects. Also, you will need to either remove all your talents, or to account for them in your calculations, but be careful, as this can be quite tricky.
The most important thing to know about spell damage is that it increases your spell's damage by a flat value. It does not act randomly at all. The range of damage which you can deal is equal to the range of damage you would deal without. Let me illustrate:
Imagine you have a spell which is lucky enough to use 100% of spell power. Say that spell, according to the tooltip, deals between 600 and 700 damage. A range of 100 damage. So if you have 150 bonus damage, the spell will always deal between 750 and 850 damage, the same range.
So essentially, we will be researching the range rather than an average. If we were researching the coefficient for this spell, with a tooltip of 600-700 and a 150 bonus spell damage, we would cast the spell a high number of times with hopes of getting a highest and lowest value with a range of 100. Once you do get that range, you can easily calculate the way in which bonus damage is applied.
You can look, for example, at the highest value each time. Without spell power, it would be 700. With spell power, it is 850. The difference is 150 damage. That is exacly 100% of your spell power.
The damage your attack does will always be an integer. However your spell coefficient times your spellpower may not be an integer (even more so after you include talents that increase all damage by 3%). What WoW does is compute the spell damage using floating point math, and then round either up or down (probably based on a random number generator) to one of the nearby integers. For a really precise spell coefficient, you will want to account for this.
Suppose you have a spell with a damage range from 445 to 488. You cast the spell several times with 901 spell damage and your lowest and highest (non-crit, non-partial-resist) damage values are 952 and 976.
You know know that 445 + 901 * coefficient is at most 952.999. If the coefficient were any larger, you could have never seen a cast that did only 952 damage.
Your 952 might have been from a base damage of 445, and in floating point might have really been 952.999. We can use this to compute an upper bound on the spell coefficient: The spell coefficient is no more than (952 + 1 - 445) / 901 = 56.38%. If the spell coefficient were any higher, the spell would have always done at least 953 damage.
Use a similar formula (against the max damage numbers) to compute a lower bound on the coefficient: (976 - 1 - 488) / 901 = 54.05%. If the spell coefficient was any lower, every cast would have done at most 975 damage.
At this point you know the spell coefficient is between 54.05% and 56.38%. To get a tigher range you'll need to run more tests.
And that's how you do it. If you give it a little thought you won't need to untalent, and you can even use crits in your analysis, but be careful!