Well, you're talking about something smaller than you vs equal sized or bigger than you.
So, my first thought on this line of reasoning is: If you're a roughly average adventurer, is it harder for you to hit, let's say, a 2.5 foot wide and 6' tall wall rather than a 5' wide and 12' tall wall?
Even taking into consideration that you're talking about a moving target I don't think it's really all that different from a basic size standpoint.
Now, from a mechanics standpoint, you also have consider that if you did have a bonus for both directions you'd essentially be doubling that bonus. Because both combatants would be getting bonuses/penalties for size differential rather than just one.