With a loaded and cocked crossbow, there's always a chance it could accidentally go off, and a chance the bolt could fall out. If you are walking around with your loaded and cocked crossbow in your hands, on a nice even floor, the chance is negligible. But if you stumble on something, and certainly if you need your hands free to get across a narrow ledge or climb over rubble, it could be greater.
If you take the bolt out, you won't accidentally shoot anyone but the crossbow could still accidentally dry fire, which is not great for the crossbow and even worse for your fingers if they happen to be in the wrong place.
A misfire, with or without the bolt, could also be startling if it happens while you are trying to concentrate on something (like balancing while crossing a narrow ledge).
I would say for normal walking around, let them do it. If there is any situation where they need to make a moving maneuver, roll and check for a fumble (same chance as a fumble in combat) which represent the chance of it going off. Likewise for other situations with movement and they need their hands, e.g. while on a horse. If there's a situation like the narrow ledge, they might take a penalty on the moving maneuver or make an extra roll for it because they were startled.
Warl's idea of breakage checks leading to a small penalty as the string stretches is also a good one.