Affirmative action, if done thoughtfully, benefits everyone. I got into Boalt - now Berkeley - Law School only because of its affirmative action program at the time. Not that they needed more White Jewish Men in the law school. Instead, they had a committee, including existing minority students, look carefully at each applicant, rather than relying just on grades and LSAT scores. Even though I graduated from a college with no grades, and my test scores were just good, not stellar, they looked at my work with farmworkers and the resulting thesis I wrote. Once in, I enjoyed another privilege of affirmative action. Diversity, in and of itself, is an advantage to all, giving us all the privilege of sharing the experience and perspective of a wider group of people.
Alas, this is when the Bakke decision came out (some of my professors had worked on the briefs). Everything went downhill after that. I've been trying to pay forward the privilege I received, but it has seemed to be a losing battle. (See Andrea Guerrero, Silence at Boalt Hall: The Dismantling of Affirmative Action (UC Press 2002) for an account of those years).
The problem, I believe, is looking at affirmative action as a zero-sum equation, in which the gains of one group must be a loss to another. Instead, we should look at it as a problem (e.g., low rates of minority admissions out of sync with the overall population) for which we need to take "affirmative action" for a solution, recognizing, as Guerrero put it, that "inequalities resulted from structural features and not just individual actions." We need to identify and address those "structural features" and pursue diversity as a goal in and of itself beneficial to all.