Why sin't disease resitance genetic? Half elves get a big boost.
Anything from being rooted in divine favor to (as in Middle-Earth) elves and humans differing in the types of their souls. You can make it genetic if you want, in your world, but that's hardly the only way to go.
A latent genetic disorder waiting to pop up in one in a hundred thousand or more, sure, but inbreeding won't cripple a population.
inbreeding != interbreeding
This has nothing to do with anything anyone else has posted to this point.
There is no need to kill babies.
There is if you want to kill all elves, as you claim. Look at those pointy ears. Clearly not "one of us".
Now, if we humans were just a little less adaptable, then perhaps destinct species of humaniods might have survived over time via well defined habitats, ignoring natural world wide disaters such as super volcanoes and killer asteroids...except we are so adaptable that no ecosystem has proven immune to human occupation and development, except under the seas of course. In time we will develop there too, along with space.
Humans only establish large populations in unfavored areas where they can alter the habitat. Humans deal with forests but cutting them down to create grasslands, not by being excellent forest-dwellers. Some humans live in forests, but their populations are low. Against a similarly (or more) advanced people that was as intelligent and specialized for forest living, they would be out-competed in that environment. Humans live in arctic and desert environments, but marginally. A specialized intelligent species there would outdo humans. As for underground dwelling, that is very rare and barely scratches the surface when it is done, and so far as I am aware, only done as a form of fortification, with the food obtained mainly from the surface. No living on cave lizards and fungal forests or whatever dwarves do to keep their cities fed.
Humans have managed to live almost everywhere, because they didn't have comparable competition. Even with modern technology, many places are marginally habitable. For a people with similar intelligence, technology, and magic (in a fantasy setting), which found those places a pleasing environment, such areas would be easily held against the sort of population drift that historically sent humans into such places. A concerted effort by a large, organized human civilization could be a threat, but then the ,e.g., desert-folk would present a counterthreat to that civilization, which would, of necessity, be advanced enough to negotiate with to avoid a conflict that might destroy either or both of them.
