Using RMX: ESF penalties and bonuses only change the unmodified failure (ne fumble) range of a spell, so when making the roll, the "Spell modifiers" don't apply, since the ESF only applies to the unmodified roll.
Using RM2: I consider ESF, Spell Failure and Fumble rolls to be akin to Critical rolls, in that a modifier only applies to them specifically. . .none are "actions" or "Maneuvers" affected by injury penalties. The only 2 modifiers I can think of that would affect all of the above, in the sense of "+/-X to ALL rolls" are the "Lucky" background option, and some of the spells on "Wyrd Mastery" that affect luck/probability.