Michael LeRoy of the University of Illinois Law School has posted a working paper to SSRN that is getting a lot of blogosphere attention: Storm's Employment Law; Workplace Prof Blog. Unfortunately, the paper follows a disturbing academic trend of correctly compiling quantitative data sets that are qualitatively ambiguous, and then jumping to a qualitative conclusion when the quantitative data is more likely to reject that qualitative hypothesis. Torturing the data set to make it confess to information it does not have, however, tends to tell us more about the confirmation biases of the author than the state of the world.
LeRoy's abstract says, in part:
State courts are creating conditions for moral hazard in the arbitration of employment disputes. The problem begins when employers compel individuals to arbitrate their legal claims, denying them access to juries and other benefits of a trial. This empirical study identifies a disturbing trend. State courts vacated many arbitration wins for employees, but not for employers. My database has 443 federal and state court rulings from 1975-2007. Remarkably, state appellate courts confirmed only 56.4% of employee wins in arbitration. But when the same courts ruled on employer victories, they confirmed 86.7% of awards. The difference in rates was statistically significant. Similarly, federal appeals courts upheld 85.7% of employer wins. Lower courts behaved like appellate courts. These state courts confirmed only 77.6% of employee wins, while federal district judges enforced 92.7% of these awards.
The lopsided results suggest a moral hazard. When courts vacate many awards that rule for employees, the individual must either return to a lengthy and costly "do over" arbitration - or worse, be stuck with a useless award, and no other recourse because Gilmer v. Interstate/Johnson Lane Corp. bars employees from suing.
Throughout this Article, the reader will sense snowballing futility for employees.
Here's the problem. There are five levels of qualitative variability in the data that LeRoy has collected:
1) Three possibilities: (A) Plaintiffs are more likely to pursue rather than settle an arbitration of a weak claim than defendants; (B) vice versa; (C) plaintiffs and defendants' claims that are arbitrated are equally likely to be meritorious.
2) Three possibilities: (A) Arbitrators could be biased towards employees, and make a disproportionate number of reversible mistakes in their favor; (B) the opposite; or (C) they could be entirely neutral, making mistakes as often in favor of employees as employers.
3) Three possibilities, because attorneys, when choosing whether to challenge awards, may not do so consistently: (A) plaintiffs' lawyers may make more mistakes than defense lawyers in choosing whether or not to challenge an award; (B) the reverse could be true; or (C) both sides could be equally likely to mistakenly challenge an award that will be upheld by a neutral judge. (After all, only a small fraction of arbitration awards are challenged.)
4) There's the iceberg effect of reporting: (A) Courts are more likely to report vacaturs of employee arbitration victories than other types of decisions relating to the challenges of arbitration decisions; (B) courts are more likely to report affirmances than vacaturs; (C) reported opinions are a representative sample of all opinions issued by courts in arbitration challenges. (A) is almost certainly true here; courts report only a fraction of their decisions, and are more likely to report the man-bites-dog unusual circumstances of a vacatur of an arbitration ruling than the straightforward and uninteresting affirmance of the ruling. (The word "unreported" does not appear in the LeRoy paper, but he misinterprets his quantitative data to definitively say that state courts vacate a "high percentage of employee wins at arbitration." Not true unless (B) or (C) is true: a high percentage of reported opinions ruling on employee wins are vacaturs, but this tells us little about the percentage of actual vacaturs, given (3) and (4). The fact that LeRoy's characterization is misleading is demonstrated by Professor Bales's comment on Workplace Prof Blog that he was surprised that employees "fared so badly"—even though LeRoy found precisely 37 instances of employees having arbitration victories partially (15) or completely (22) vacated in reported decisions in 32 years. The comparable number for employer victories partially or completely vacated was 27; the statistically significant difference in percentages come almost entirely from the fact that employees were more likely to challenge arbitration awards in reported cases, perhaps because they brought more creative long-shot legal theories to bear.)
5) Then there's the judicial level of review: (A) Courts unfairly provide greater scrutiny to employee awards than to employer awards; (B) courts unfairly provide greater scrutiny to employer victories than to employee wins; (C) courts review arbitration awards correctly, making mistakes as often in favor of employees as employers.
LeRoy has taken a dataset of 443 judicial decisions, a grand total of 55 of which reversed arbitration awards; he makes no qualitative assessment of these decisions, and merely counts winners and losers. From this, he concludes C-C-C-C-A in the multiple choice above, and then draws a long series of conclusions about moral hazard from that C-C-C-C-A intermediate premise.
The problem is obvious: There are a grand total of 3 x 3 x 3 x 3 x 3 = 243 combinations. LeRoy's data set is consistent with 211 of those 243 combinationsFN* (all of the combinations that have at least one "A", e.g., C-A-C-C-C or C-B-A-C-B), yet has definitively concluded that courts are providing too much scrutiny of employees' arbitration awards, though in 130FN* of the 211 possible combinations consistent with the dataset the answer to #5 is not "A".
For example, the sequence A-A-A-A-B is entirely consistent with LeRoy's empirical findings: if arbitrators make more reversible mistakes in favor of employees than employers, and then plaintiffs are more likely to make a weak appeal of a losing ruling than defendants, and then courts are less likely to report the affirmances of arbitration awards, it is entirely possible that judges are, on average, biased against correctly reversing employee victories in arbitration, yet still reverse employee victories more often than employer victories. LeRoy does not account for this possibility at all, but it would completely undercut his moral hazard argument, much less his claim of a "growing vacatur problem."
I don't disagree with one of his conclusions—"arbitration losers who incur liability should be required to post judgment bonds if they challenge an award. This idea borrows from civil procedure codes, and is therefore consistent with the goal of forum substitution." That's not a bad thing to put into an arbitration agreement, though it's a mistake to view employer appeals as "cost-free" as anyone who looks at attorneys' hourly billing rates can tell you (LeRoy would have a far larger dataset than 443 decisions if it were really cost-free for an employer to challenge an arbitration award). But I suggest that that's a rule that would favor employers more than employees, who may have trouble posting a bond in the event that an arbitrator's ruling was ultra vires.
[FN*] There are 35 = 243 combinations. Any combination with an "A" in it is consistent with the LeRoy dataset: there are 35 - 25 = 211 combinations that have an A in it. Of those combinations, 34 = 81 combinations will have an A in the fifth place, leaving 211 - 81 = 130 combinations with an A in one or more of the first four places but not an A in the fifth place.