I agree that we should not get stuck on "semantics and labels." Nor should we be casting blame or demanding apologies. The "harm" that was caused by Whoopi Goldberg's comments was that she only "relates race to her experience as a Black person" and diminishes anything else as just "White people" who are "going to fight amongst yourselves."
What I am urging is to go beyond our own "experience as a Black person" or a "Jewish" person, to expand our empathy to all who are oppressed. As you say, let's "not get sidetracked by labels" but stand together against "racism, bullying and general victimization. Let's not run when the Klan comes marching down the street, but stand together.