
There seems to be a lot of miscomprehension on this subject, so I thought I would attempt to write up the definitive answer once and for all, and hopefully this post can be stickied for the benefit of the forum.
My understanding is that two calculations are done  one for chance to hit, then another for resistance. Your base chance to hit is as follows:
Your level 3 99%
Your level 2 98%
Your level 1 97%
Your level 96%
Your level +1 95%
Your level +2 94%
Your level +3 83% (87% if it is a player)
Your chance to hit is capped at 99% always. Your final chance to hit, then is:
min (99%, base chance + talents + gear)
This means, for instance, that in lvl 60 PvP, the maximum useful amount of +% to hit gear and talents is 3%. Another important conclusion is that, at lvl 60, more than +5% will only help versus lvl 63 bosses.
The second calculation is for resistance. Mobs sometimes have inate resistances, and players develop resistances from gear, talents, buffs and racials. This resistance is adjusted by 5 for each level of difference between the attacker and the target. So, against a base resistance of 50, the effective resistant score will be:
Your level 3 35
Your level 2 40
Your level 1 45
Your level 50
Your level +1 55
Your level +2 60
Your level +3 65
Note also that spell penetration effects (curse of elements, Don Rodrigo's Band) can reduce a target's effective resistance. It is not possible to reduce a target's resistance below zero and therefore not possible to achieve vulnerability damage bonuses. (Note that there are mobs with negative resists, and therefore vulnerability bonuses, in the game, but it is not normally possible to achieve this).
The effective resistance rate is capped at 300, and floored at 0, so the formula is:
max (0, min (300, base resistance  spell penetration + 5 x (target lvl  attacker lvl)))
This leads to the conclusion that the maximum useful amount of fire resistance gear to use against lvl 63 bosses like Ragnaros, is 315.
The effective resistance rate moves in steps of 15 , so 13 is in practice the same as 14, but 15 is in practice 15 better than 14 FR.
From this number a chance to resist is calculated. Overall the damage reduction is equal to the effective resistant rate divided by 4. Since the effective rate is capped at 300, that means damage mitigation is capped at 75%.
For binary spells, like silence or frostbolt, the outcome is always zero effect or full effect, but on average damage mitigation is equal to the above calculation.
For nonbinary spells, like fire, there is a chance of the target taking 0%, 25%, 50%, 75% or 100% damage. The sum of these probabilities still gives the same average damage mitigation as we calculated above.
Misses are known as 'white' resists, because the 'resist' message on the mob will be white. Full resists due to resistance show up as yellow, hence 'yellow' misses.
These two calculations are separate  no amount of +% to hit gear will change your yellow resist chance, and no amount of spell penetration will change your white resist chance.
I think this last point has caused confusion because Tseric said that one roll is used:
Q u o t e:
For binary spells only, there is an additional modifier for the resistance of the victim to your particular spell school: fire, frost, shadow, nature, arcane. That modifier is multiplied by your hit chance to get your actual chance to land. This is done with binary spells only, because they never do partial damage.
Example:
Eyonix the Mage (level 60) fires a frost bolt at Yeti of Doom (level 63). Eyonix is also wearing a total of +6% spell hit gear. Yeti of Doom has frost resistance such that he takes 50% from level 60 frost attacks. So, heres the hit calculation:
0.83 (83% for +3 levels mob) + 0.06 (+6% spell hit) = 0.89
0.89*0.5 (50% damage from frost) = 0.445.
The game will roll a number between 0 and 1, and if its less than 0.445, the frost bolt will hit for full damage. Otherwise, a resist message will appear.
While this may be one calculation, I believe the formula is as follows:
Chance = min (99%, base chance + talents + gear) * (100%  (max (0, min (300, base resistance + 5 x (target lvl  attacker lvl))) / 4))
I would say that it is much more intuitive to regard these calculations as completely separate.
The full post from Tseric is here:
http://forums.worldofwarcraft.com/thread.aspx?fn=blizzardarchive&t=37&tmp=1#post37[ post edited by Romaneva ]
Epitaph: I told you I wasn't feeling well
