The United States Supreme Court ruled against affirmative action in college admissions. In a case involving Harvard and the University of North Carolina, two of the oldest colleges in the country, the court said colleges could no longer use the race of the applicant as a determining factor in whether they were granted admission.
In practice, that meant that blacks and Hispanics were more likely to be admitted than those of Asian or white heritage, everything else being equal.
A little history: Brown v. Board of Education ended the Plessy v. Ferguson legal doctrine of separate but equal. For the reason obvious to anyone who actually looked, that separate was not equal. All black schools on the wrong side of the tracks did not provide an education equal to all white schools in the suburbs. Brown said that a public education "must be made available to all on equal terms." Justice Sandra Day O'Connor's opinion in 2003 in Grutter v. Bollinger watered that down to allow the affirmative action we have lived with for the last 20 years.
O'Connor herself said that in 25 years affirmative action would no longer be needed. The Roberts court shortened that to 20 years, saying "Eliminating racial discrimination means eliminating all of it."
If it were not for the fact that the vast majority of colleges in the United States accept federal aid, either directly or in the form of aid to students, affirmative action would not be of much interest to us. Libertarians believe that the First Amendment provision for freedom of assembly implies the freedom to assemble with whomever you wish to. If a person wished to assemble with a student body more heavily weighted to people of color, we're fine with that.
Where libertarians (and most other people) have a problem is when we must, as taxpayers, fund the education of students with lower academic qualifications at the expense of students with higher academic qualifications. That's what affirmative action does.
The 22 or so colleges in the United States which do not accept government funding in any way are free to practice affirmative action however they want to. And we have no problem with that.