The racial modifier to the stat modifier is already part of the character's stat modifier. Page 23 of the core rules literally tells you that every time you apply your basic stat mod to something, you are supposed to include the associated racial stat mod, too. It is literally considered part of your character's stat mod. This even applies for situations where a single stat mod is applied multiple times, such as the resistance skills, your defense bonus (doubly so if you're benefiting from the agile defense/swashbuckler talent!), etc.
I agree that simply lifting a heavy object for a brief moment so someone pinned beneath it can get away should be easier than carrying said weight across town. Cutting the penalty in half seems a decent compromise, if the weight lifted is below your carry limit.