I don't know if this is spelled out anywhere in RM2/RMC or RMSS/RMFRP, but in RMU the rule is:
Light versus Darkness: Magical darkness completely hides non-magical sources of light, but is overcome by magical light. Utterdark spells suppress even magical light, except for Utterlight. Normal fires, torches, and the like will still produce heat in an area of magical darkness, and will continue to burn (and ignite combustible materials), but will not produce any light. Upon exiting an area of suppressing darkness, normal or magical light will spring up again (unless extinguished or expired in the meantime). Likewise, upon the removal of a suppressing magical (Utter)light, magical (Utter)darkness will resume if not expired. Actually eliminating, rather than suppressing, magical light or darkness requires dispelling it.
I think this is generally the intent in previous editions even if it wasn't always as clear. So, magical darkness is not going to be defeated by lighting a lantern (because if it was, why isn't it also defeated by sunlight or whatever light sources are already present?). It's only going to be defeated by a light spell.