It's true, LACK of racial diversity doesn't necessarily equal racism.I am well aware of this. However, he was only questioning the issue of racism in the Empire.
Is Japan racist because most of its jobs and positions in government are filled by non-blacks, non-whites, etc (in fact, anybody but japanese 'oriental' ethnicity?)
Perhaps non-Whites among humans are even MORE rare in Star Wars than they are on earth. Now coming from Europe or North America, whites are a fairly large group, but in the rest of the world they are far outnumbered.
Realistically, the casting of mostly whites probably has more to do with the ethnic/racial makeup of the countries that produced the movies and the thoughts about race at the time they were made. But... I think the implication is that the Empire doesn't bother.. because they want everybody to be the same, and so they can exclude minorities on principle. People like Grand Admiral Thrawn and Admiral Daala were considered unusual, and that's only in the EU. Even in those cases they were given remote outposts and secret bases to defend, hidding them away from the rest of the galaxy.