Handling a refugee camp of 3,000 is a huge task - trying to handle one for hundreds of thousands would require a full government, complete civil support, and entire logistics/supply line.
3,000 is the correct number, and the group can survive on the supplies found in storage of Belynar until such time as they are able to situate themselves and begin to settle. Also, hundreds of thousands would instantly draw an army from the various other races, while 3,000 simply draws iinterest. The only issue with the 3,000 would be that a party of 5-8 adventuring caliber people would certainly be visible and known among the populace. Certainly if someone goes out on a mission or achieves something of note in Belynar, the achievement would be well known and their "fame" would spread through most of the population. (Think of a star athlete at a major university of 20K+ students - everyone knows them and most know of their success after a big game)