One of EFF’s chief concerns about exam proctoring software—in addition to the fact that it subjects students to excessive surveillance—is the risk that it will incorrectly flag students for cheating, called “false positives.” This can be due either to the software’s technical failures or to its requirements that students have relatively new computers and access to near-broadband speeds. Last week, the California Bar released data confirming our fear of false positives: during its use of ExamSoft for the October Bar exam, over one-third of the nearly nine-thousand online examinees were flagged by the software (13:00 into the video of the California Committee of Bar Examiners meeting).

It is clear that at least some of these flags are technical issues with ExamSoft.

This is outrageous. It goes without saying that of the 3,190 applicants flagged by the software, the vast majority were not cheating. Far more likely is that, as EFF and others have said before, remote proctoring software is surveillance snake oil—you simply can’t replicate a classroom environment online, and attempting to do so via algorithms and video monitoring only causes harm. In this case, the harm is not only to the students who are rightfully upset about the implications and the lack of proper channels for redress, but to the institution of the Bar itself. While examinees have been searching for help from other examinees as well as hiring legal counsel in their attempt to defend themselves from potentially baseless claims of cheating, the California Committee of Bar Examiners has said “everything is going well” and called these results “a good thing to see” (13:30 into the video of the Committee meeting).  

That is not how we see it. These flags have triggered concern for hundreds, if not thousands, of test takers, most of whom had no idea that they were flagged until recently. Many only learned about the flag after receiving an official “Chapter 6 Notice” from the Bar, which is sent when an applicant is observed (supposedly) violating exam conduct rules or seen or heard with prohibited items, like a cell phone, during the exam. In a depressingly ironic introduction to the legal system, the Bar has requested that students respond to the notices within 10 days, but it would appear that none of them have been given enough information to do so, as Chapter 6 Notices contain only a short summary of the violation. These summaries are decidedly vague: “Facial view of your eyes was not within view of the camera for a prolonged period of time”; “No audible sound was detected”; “Leaving the view of the webcam outside of scheduled breaks during a remote-proctored exam.” Examinees do not currently have access to the flagged videos themselves, and are not expected to receive access to them, or any other evidence against them, before they are required to submit a response.

It is clear that at least some of these flags are technical issues with ExamSoft. Despite apparent knowledge of the issue months ago, many examinees using Lenovo laptops seem to have been flagged en masse for an issue with the software’s inability to access the internal microphone, though examinees did not have an issue with the practice exam. Lenovo laptops are very commonly purchased because they are affordable, and are one of the more popular PC brands overall.  

Still other flags are likely due to the inability of proctoring software to correctly recognize the enormous variability of examinees’ demeanors and expressions, particularly students and examinees who exhibit behavior such as stimming. An inability to detect eyes during an exam could simply mean a test-taker was closing her eyes to think, or to rest after several hours of focused staring. Being flagged for leaving the view of the webcam could very well be a failure of the software to recognize a student’s face, which is more likely to occur to Black and Brown students.

We implore the California Bar to rethink its plans for remotely-proctored future exams

Apparently, some applicants who received notices had even taken the test at major law firms where IT departments worked to ensure they would be in compliance with the exam’s rules and procedures. If, despite that, and despite mock exams, and despite hundreds of tech support calls (which reportedly caused some students to be flagged for using cell phones during the exam), one-third of students are still flagged, what exactly is the benefit of proctoring software?  

According to the Guidelines Governing the Interpretation and Application of Chapter 6 of the Admissions Rules, “the State Bar has the burden of establishing by clear and convincing evidence that a Chapter 6 violation occurred and that the intended sanction is warranted.” So far, this has not been done, despite the Chapter 6 Notices being sent. It is unfair for students to be forced to respond to potential sanctions before they’re given adequate information about the behavior for which they may be sanctioned. Deans of several law schools have requested [pdf] that flagged examinees be allowed to view video of alleged violations before responding to notices.  

The Deans also note that appealing a flag currently means an examinee’s score will be withheld pending resolution, during which the examinee cannot reapply for a future exam. This leaves those who received a Notice at a loss for next steps, and in the dark about whether they should reapply or not. That the State Bar has created such an impossible process for responding to these algorithmic flags flies in the face of the California Supreme Court’s claim that proctoring software will not be the deciding factor in whether or not an examinee passes.

The California Bar’s mission includes supporting greater access to, and inclusion in, the legal system. Forcing thousands of examinees to defend themselves against an algorithm that claims they’ve cheated without seeing the evidence against them quite clearly goes against that mission. Other states have canceled the use of proctoring software for their bar exams due to the inability to ensure a “secure and reliable” experience. We implore the California Bar to rethink its plans for remotely-proctored future exams, and to work carefully to offer clearer paths for examinees who have been flagged by these inadequate surveillance tools. Until then, the Bar must provide examinees who have been flagged with a fair appeals process, including sharing the videos and any other information necessary for them to defend themselves before requiring a written response.