I'd determine a penalty for the size of the person, then another for the size of their weapon.
a halfling in a 5' tall passage is +0, using an 8' long halbard -20
but
a human in a 5' tall passage is -20, using a 1' long shortsword +0.
It's a judgement call, as to how impaired a person is by the surroundings. . .it may help to list your player's sizes and decide logical "how constrained is body motion" decisions, then list the weapons they commonly use and select penalties. . .looking at all of it in one place may help you gut check it for consistency, and if it feels right to you.
Check to see if any of your players have RAC (Restricted Area Combat) skill. . .I think that skill may also list some suggested penalties.
(You playing RMC or RMSS?)