I would disagree with that latest point, since swimming wasn't at all common in the Middle Ages, only people who lived near a river or had to go into the sea or rapid waters to fish or work were swimmers. Using your description of physical skills being the most common used, I would trade swimming for climbing.
That would depend on where one is located. England is an island, criss-crossed with many streams, rivers, ponds, and lakes.
Even Italy has many bodies of water in which one would have to learn to swim. Most of Western Europe and the Mediterranean swimming would be common.
Climbing is likely not included as it is also a skill for climbing up castle walls, or walls of houses to break into such. Climbing that would be common would not require a roll, thus it is not placed in Physical. Climbing that does require a roll is more uncommon, thusly placed in Athletic.
In any case, you can move climbing to Physical, or swimming to Athletic.
It all depends on the background of the character and world.
If playing in a mountainous world, climbing might be very common. If playing in a heavily treed world with very large trees climbing might be very common. However if the world does not have rock outcroppings, cliffs, and very large tress that are common; the character is less likely to have climbing skills. Thus the climbing becomes an athletic skill.
If playing in an island world (mostly water with lots of small islands), swimming might be very common. If playing in a heavily water world (lots of streams, rivers, ponds, lakes, et all) swimming might be very common. However if the world larger bodies of water that are common; the character is less likely to have swimming skills. Thus the swimming becomes an athletic skill.