Pearbotham's Law on the Persistence of Achievement Gaps

Men do not make laws. They do but discover them.
 
                                                         -- Calvin Coolidge, 1914           

Mentor. Not at all Prodigy. I want you to meet my friend, Judge Pearbotham.

Prodigy. The famous Medal of Freedom winner? Judge Aldan Pearbotham? I am honored.

Mentor. You know Prodigy, Judge Pearbotham is retired from the bench. He is now on the faculty of the Barnyard Law School, so we see a lot of each other. He teaches a course in "Race and the Law." We were looking at his final exam for next semester when you knocked.

Prodigy. Next semester?

Judge. Yes, I write the exams before a course begins. It helps keep me focused on the course goals.

Prodigy. Well, I did not mean to interrupt ...

Judge. I welcome your input Prodigy, but you must promise not to spill the beans if you are in the vicinity of Grosshall before the course ends.

Prodigy. Of course, but I know so little about the law I doubt I can add much. I am, however, curious about legal training in a renowned law school like Barnyard that has produced so many distinguished leaders. This is an extraordinary opportunity for me to learn from none less than Judge Aldan Pearbotham.

Judge. Are you considering law as a career?

Prodigy. It had crossed my mind.

Judge. Well, a good head, a sturdy spirit and an appetite for blood are the basic ingredients of a good lawyer. Ha, ha, ha. Why don't you look at my final exam while I make a call. See what you make of the first question. I won't be ten minutes.

Question 1. (Seven hours, open book)

   The State of Flyover requires high school students to pass an exit exam before obtaining a diploma. Students are given eight tries at the exam beginning in their sophomore year. To pass, a student must answer 70% of the questions correctly. Ninety-eight percent of white students ultimately pass, but only 85% of African Americans. Ninety-two percent of whites pass on the first try, compared to 66% of blacks.

   A coalition of civil rights groups sued the State demanding that the exit exam be discontinued. Plaintiffs charged that the exit exam discriminates against minority students by producing disparate graduation rates. They argued also that because fewer African Americans pass on the first try, they are disparately subjected to stress associated with uncertainties attendant upon not knowing whether they will ever graduate.

   Arguing for the State, Attorney General Morgan Cantekoupe contended that the test ensures that all graduates meet a minimum standard of competency, and that without such a standard a high school diploma would be meaningless to a prospective employer or college.

   Cantekoupe provided expert testimony establishing that the exit exam maintained a constant level of difficulty at each administration. He presented evidence showing that first-time pass rates for African Americans had improved from 30% in 1995 to a current rate of 66%, a gain of 36 percentage points. Cantekoupe attributed the improved pass rates to State-sponsored remediation. Indeed, every student who fails the exit exam in Flyover is assigned to a remedial help clinic. He observed that whites also improved, their first-time pass rates going from 68% in 1995 to 92%, a gain of 24 points. Most importantly the State argued that because of its efforts a dramatic 12-point reduction in the racial achievement gap was obtained in just a few years.

---

   You are a federal district judge in the western district of Flyover. The case comes to you today on cross motions for summary judgment by the civil rights coalition and the State. With all of the above facts stipulated, how do you rule and why?

Judge (returning). Have you digested Question 1 yet, Prodigy?

Prodigy. Mentor has kindly offered me the use of his computer. A Web search revealed this case to be similar to one brought in Texas in 1999. There, a coalition of civil rights groups and parents (Forum et al) sued the Texas Education Agency. Plaintiff claimed that because Hispanics and African Americans passed the Texas high school exit exam (TAAS) at lower rates than the majority whites, the test had an adverse impact on minorities and should not be a graduation requirement. In fact, with the exception of the charge of disparate stress, the issues are identical.

Judge. Good, my boy! I modeled my question on that case.

Prodigy. In its ruling, the Court in Texas acknowledged that the exit exam had an adverse impact on minority students, but it nevertheless ruled against the plaintiffs, saying that adverse impact was "not impermissible." Texas argued, like Flyover, that racial gaps on the exam had narrowed because of the State's efforts at remediation. The Court gave this argument important weight. Judge Prado wrote in his decision: "The Court finds as an inescapable conclusion, that in every administration of the TAAS test since October 1990, Hispanic and African American students have performed significantly worse on all three sections of the exit exam than majority students. However, the Court also finds that it is highly significant that minority students have continued to narrow the passing rate gap at a rapid rate."

   Though Judge Prado found gap narrowing "highly significant," I am not sure if any real narrowing occurred. It is possible that the perceived gap narrowing might be nothing more than a mathematical artifact.

Judge. Mathematical artifact?

Prodigy. Yes. A black-white gap shows up on just about every paper and pencil test, SATs, MCATs, LSATs, IQ tests, police and fireman promotional exams and more. Furthermore, the gap is remarkably constant, about one standard deviation (SD) on each test. Claims of gap reductions must be viewed against this experience, that is, cautiously. I think ...

Judge. Centuries of slavery and racism have taken their toll on African Americans. They have been oppressed and depressed, infused with self-defeating attitudes. Poor performance by African Americans on standardized tests has been explained by Professor Claude Steele of Stanford. It derives from the extra pressure brought on by the widely-held expectation that blacks are not going to do well. That aside, we must not take centuries to undo the damage. Every advance, no matter how small, needs to be acknowledged and applauded. Any reduction of any racial gap is cause for celebration.

   But you have not answered. What exactly do you mean by a mathematical artifact?

Prodigy. Let's review the facts. In Flyover, from 1995 to the present, the exit exam maintained a constant level of difficulty and a constant passing requirement of 70%. At each administration, African Americans failed at a higher rate than whites. With time, however, pass rates of both blacks and whites increased, black more than white, causing a 12-point reduction ...

Judge. Precisely! SATs and MCATs are irrelevant here. We need to consider the facts at hand. When a real effort is made to close a racial gap, progress is made, whether it be in Texas or Flyover. In Flyover, the gap went from 38 points in 1995 to 24 points in just a few short years. With hard work and goodwill we can reduce all racial gaps to zero. The problem, Prodigy, is the goodwill -- or rather the lack of it. Racism clouds every facet of American life. You need to take my course, lad.

Prodigy. I would benefit immeasurably from your course, Judge, but indulge me for a moment more. Imagine a circumstance in which all students were so weak that almost no one could pass an exit exam. In this event the racial gap would be vanishingly small because a difference between two near-zero pass rates is also near zero. Now suppose, because of improved instruction or remediation, students improved. No longer constrained to zero, a gap would appear. As students continued to improve the gap would vary, but once again vanish for similar reasons at the other extreme when all the students became so proficient that pass rates neared 100%.

   Constrained to zero at the extremes, when performance is either extremely poor or near-perfect, the gap must reach a maximum value in between. As a function of performance, the gap might look something like this:

   Though we know the achievement gap is some function of student proficiency (measured here by the white pass rate), we do not yet know the function's details. We have learned, however, that because the gap vanishes at its endpoints and is nonzero at points in between, as students improve there will be times when the pass-rate difference will increase and other times when it will decrease. And this variation in the gap, Judge, has nothing to do with real relative changes in group performance. The difference between the mean scores of blacks and whites could remain perfectly constant, while all this is taking place. That is, the perceived gap variation could simply be a mathematical artifact!

Judge. Do you mean that there was no real gap reduction in Flyover?

Prodigy. Be patient, Judge, all will come out in the wash. First we need to find the function depicted in my sketch. It should not be difficult.

   Let f_W and f_B be the fractions of white and black students, respectively, who pass the exit exam. Also, let P(x) and P(x+Δ) be ability distributions of whites and blacks, respectively. The quantity, Δ, is the mean difference in the ability distributions. We may then write,

where λ is the passing score (in standard units), and in (1) we have used the transformation:

   Equations (1) relate the two passing fractions, f_W(λ) and f_B(λ), parametrically. The percentage point gap, Δ, is given by

   Using (1) and (3) the gap may be evaluated as a function of the white pass rate. I did this while you were out Judge, and plotted up the result for a Gaussian distribution with Δ = 1.0 SD. Have a look.

   You see, Judge, gaps can come and go without a dime's worth of change in the difference between the mean scores. In this diagram the black-white mean-score difference remains constant at 1.0 SD, while the apparent gap (measured in percentage points) varies all over the map, from zero to near 40. In short:

Pass-rate gaps measured by percentage point differences can apperar to change dramatically without meaningful change occuring.

   We can call this Pearbotham's Law. It was, after all, your final exam that inspired it.

Judge. Yes! I will frame it and hang it on the wall right next to my Medal of Freedom.

Prodigy. That would really impress visitors, especially distinguished ones like Supreme Court Justices.

Judge. Indeed! You know if Dukakis had won in '88 I would be sitting on that bench instead of that Tom. With a new president, Pearbotham's Law might give me the inside track again.

Prodigy. I couldn't think of a better choice.

Judge. But Son, there is one little problem. This is all theory. We need to buttress it with facts. The law is about facts! About evidence!

Prodigy. Be patient Judge. Lots of data have been accumulated, much of it publicly available. We don't even need the Freedom of Information Act. We have enough to test Pearbotham's Law over a range of pass rates. Here's how.

   Pass rates can be varied in several ways:

1) We can look at pass rates at different achievement levels, say satisfactory and excellent. It is common practice to report results in this way.

2) Pass rates vary with the quality of the test taker and the nature of the test. High school students, for example, generally do not pass standardized tests at as high a rate as medical students taking a licensing exam.

3) By improving instructional methods or curriculum, pass-rate changes can be observed over time.

   By these methods, used singly or in combination, we can find data to cover a wide spectrum of pass rates. While you were out, Judge, I located pass rates (by race) for several large-scale tests. I used only huge data bases to avoid anomalies that might crop up in smaller data sets, say data from a single US county.

   One excellent source of test results is the Maryland State Performance Assessment Program (MSPAP). Its tests are given every year and taken by every student in Maryland. Most importantly, the results are reported fully disaggregated by race. MSPAP is an excellent source of data for people who study group differences. Pass rates are reported at both the satisfactory and excellent level, supplying points for low and intermediate pass rates. I located seven consecutive years (1993-99) of eighth-grade MSPAP math results.

   At higher reaches of competency, I obtained data from three sources. The first, bar exam data, should be dear to your heart. You will be happy to know that a very high percentage of lawyers pass the bar exam, either at a first or subsequent try. The Law School Admission Council, in a huge study, tracked all 27,000 students who enrolled in US law schools in the fall of 1991. It followed them through to the bar exam. The Council issued its report in 1998, including first-time and total pass rates by race.

   More high pass-rate data came from the benchmark National Board of Medical Examiners (NBME) test required of every medical student in the US. Students take this exam two years before graduation and must pass to become licensed. A comprehensive study of NBME pass rates by race was published in the Journal of the American Medical Association (JAMA, 272:9, 674-9, 1994). It included first-try pass rates for all medical students in the US for the years 1986-88.

   Finally, at a lower (but still high) pass rate, I obtained first-time pass rates on the California Basic Educational Skills Test (CBEST). Every new teacher in California must pass this exam. Eleven years of test results (1985-95) go into this point.

   The data are summarized in a table which you may take with you, Judge. Better still, here are the results displayed graphically, suitable for framing and hanging on your office wall.

   The points are color coded to match the table entries. The function (3) with a 1.1 SD black-white mean difference is plotted for reference. The fit is not too shabby.

Judge. What about the Flyover points?

Prodigy. Come now, Judge, you made those points up.

Judge. Ah, so I did. What about the exit exams in Texas?

Prodigy. That was discussed in "Standardized Tests: the Interpretation of Racial and Ethnic Gaps," (La Griffe du Lion, Vol 2, No 3, March 2000). Texas has succeeded in narrowing the 10th grade exit exam gap a bit though not nearly as much as claimed.

Judge. I think I am going to lecture on this stuff, Prodigy. Problem is, my audiences will be made up primarily of lawyers, and frankly my boy they are not rocket scientists. I need something that they can grab onto -- something concrete.

Prodigy. Can they read a contour map?

Judge. What's that?

Prodigy. I'm sure you've seen some, like this representation of a hill. Each curve is a contour of constant elevation. At the bottom you're at 100 feet, near the top at 500 feet.

Judge. I think I get it.

Prodigy. Good. Now I'll prepare a group-difference "contour map" for your lawyers. Thus armed, they will be dynamite.

Mean Difference Contour Map.  Each curve is a path of constant mean-score difference (in standard units).

   Civil rights lawyers should carry a copy of the map in their wallets. They can use it in three ways:

1)  They can translate pass-rate gaps from percentage points to mean-score differences. Tell them to find the intersection of the observed gap and the percent of whites who pass. The point will be on or near one of the constant mean-difference contours. By interpolation they should be able to pinpoint a group mean-score difference to within one one-hundredth of a standard deviation.

2)  For a known (or suspected) group mean difference, they can predict the percentage of blacks who score at a certain level or better, given the corresponding percentage of whites (or vice versa). For example, suppose 70% of whites pass a police promotional exam. From the chart, assuming a 1.0 SD black-white mean difference, we expect a gap of 38%. Thus, 70% - 38% or 32% of blacks are expected to pass. Anything much different is cause for suspicion.

3)  They can test for real gap reduction over time, as claimed in Texas and Flyover. Tell them to plot the gaps vs. the percentage of whites that pass. Meaningful gap reduction takes place only when contours are crossed.

   Finally, there is nothing special about blacks and whites as groups. Any two groups can be compared using the map. Just replace whites with the higher scoring group and blacks with the lower scoring. Sometimes, whites will be the lower group. Asians, for example, usually score higher than whites on math exams. In this case make Asians white and whites Asian. Simple, eh Judge?

Judge. It works my boy! Let's put everything together so I can take it with me. I must leave now to deliver a lecture on race relations at the National Black Motivational Society. Would love to stay and chat some more, but I can't be late at $40k a pop.

   Think how you can put your talents to work for the benefit of the oppressed, to lift up those beaten down by hundreds of years of exploitation. You have many years to serve mankind. Make the right decisions and you will do it well. And be sure to sit in on my course in the fall. Of course, you won't have to take the final exam.

   Goodbye Prodigy. Goodbye Mentor. The three of us must do this again over dinner.

Mentor. Goodbye Aldan, I hope your talk to the Motivational Society is as uplifting as your visit here today.

Prodigy. Sometimes, Mentor, you can just feel when you are in the presence of greatness.

# # #