D&D dark elves were the only dark-skinned elves, and they were evil. Dark skinned dwarves were evil. It's not a big jump to expect that dark skinned versions of other races are also meant to be evil in that case. D&D dark elves were also D&D's most prominent matriarchy, and not only was it evil, it was also dominatrix-based. I don't think it's much of a jump to read something into that. Even the good drow were fetishized, written as literally dancing around naked. So.
In the real world, things that live underground, in darkness, tend to become pale and white, not dark. Albino cave elves.
Personally, I prefer the idea that darkness is in their hearts and not on their skin. I played an elven enchanter in a game with that kind of dark elves, the elves with the closest ties to nature could sense that difference. Others could not. I had some concern that my character had actually crossed that line (ok, he had the Mind Disease spell list, it was a valid concern), and steered clear of those kinds of elves. "Am I dark?" Interesting.
My regular elves in my campaign have a more olive complexion, they are well suited for being out in nature all the time, not pale and ethereal as Tolkien had them. I have dark dwarves, but the darkness is that the earth they work with will not respond to the earth magic of other dwarves, it is "dark" to them. There's nothing visual about it. It's not inherently evil, just separate. There are no dark elves in that sense, that is not the path of evil that elves walk down, they have their own ways....