Mind you, humans, elves, etc. will have similar problems visiting non-cosmopolitan areas of dwarven or halfling or like-sized folk. Whereas in Giantville, it will be the Trolls who have the least problems "fitting in". The size issue is all relative, so it gets down to what is normal in the campaign.