I'm close to jdale on this, but I have to admit empirical evidence also shows I'm a better world-builder than I am player, so I went with my strength.
I have all the material I've used for adventures and my world going back to the late '80s. Some of the early stuff I would say isn't quite what I want in the world now, and since I'm on my 5th or 6th set of players in that world, I do have the freedom to change it. But mostly it's been almost 4 decades (oof) of building, adding detail to areas I still use, and filling in areas that were blank on the "map". Each new group of players prompts different questions about how something works in part of the world, and I fill it in.
I don't know how many here remember the old list-serve for Rolemaster. One of the things I loved on that (and here, but it's a different beast) was the amazing expertise people had on real-world historical topics, and fantasy literature. I still have some printed posts from doctors who had studied medieval medicine, historians who had worked on bronze-age cultures, etc. I have, in all honesty, learned more about the real world and the people in it from gaming, than I learned in getting my Ph.D.
And when I go senile, hopefully at some distant date, I have no doubt where I'll be living in my mind...