It was fun if you were a spell user. For the non-spell users, not so much. Casters could do everything.
Now, you posted this question in the general RM forum, so I don't know if you are planning on using RMFRP or RMC, and there is a BIG difference...
In RMC, the CORE rule is that if you are wearing the proscribed type of armor, you cannot cast at all, period.
The ESF mods for wearing armor are an option. Using them increases the power of the caster.
RMFRP increases the power of spell casters by making ESF be core.
If you want to remove ESF mods from armor, then I would suggest replacing them with some other sort of drawback. For example, increase the number of PP required per spell based upon AT (i.e. 1/2 AT, rounded up in extra PP). Thus wearing AT 5 would mean that all spells cost Level + 3 PP to cast. For those wearing AT 20, that would be an extra 10 PP per spell.
Then for magical/material bonuses, add an extra PP per +5 bonus. Thus, Mithril Chain (AT 16 +20 DB) would require that all spells use an extra 12 PP to cast.
This way, they have to balance what they want to wear against how often they want to be able to cast spells. No minuses to casting, BUT there is still a real effect that has to be handled.
As for beefing up the non-casters. Use some of the stuff from Combat Companion. While pure casters still have to learn 1 weapon at a time, allow Non-Spell Users to learn weapon groups. Also allow them to semi-freely gain Combat Maneuvers for each weapon group as well.
For example, Fighters start with 3 Combat Maneuvers (each maneuver is only usable with a single weapon group - thus the Fighter COULD get the same one for 3 different weapon groups). Rogues, Thieves & Warrior Monks start with 2 each, and Semis start with 0. Then for each level beyond first, they get to select 1 additional Combat Maneuver.
The combination of these two steps should help balance things out between the two main types.
