Actually, some of these labels were used to keep poor whites down. These are the "artificial distinctions" W.E. B. DuBois pointed out prevented the union of the "freed" slaves with poor whites, which could have overthrown the oligarchies of both the North and South. (See his Black Reconstruction in America).

That is not to say that these labels are "racist." But these, and all such labels, make solidarity more difficult. Solidarity is the opposite of racism, and (I believe) the only way to truly defeat it.