The BBC Panorama investigative story on UK television this evening, regarding the compromise of the TOEIC exam is the story of the assessment industry’s ‘inconvenient truth’.
Despite the high cost of the exam to the candidate, the programme demonstrated how both
the e-Assessment and ‘bubble sheet’ parts of the test can be railroaded by fraudsters masquerading as ‘facilitators’. When the exam means something to a learner/ candidate (license to practice, entry into education, drive a car), there will always be a minority willing to take risks to beat the system.
Watching the investigation tonight, there are a number of elements which make uncomfortable viewing for exam sponsors, exam delivery companies: both in e-Assessment
and paper delivery. While companies such as ETS have been delivering exams securely around the world for many years, this UK issue does not necessarily equate to a global delivery issue or a failing exam, but should send shock waves throughout the industry.
Six points sprang to mind as I watched the programme.
1. Service Encounter While candidates sign-in at the reception and a photo is taken of the candidate in the exam room, the lack of systematic, controlled candidate authentication at each step of the service encounter, delivering a piece of admissible evidence, is a serious flaw. Why could a photo not be taken of the candidate at the workstation, and then at random times within the exam session? This is possible today with webcams and simple scheduling software.
What about making the candidate present a piece of ID at the workstation? This could be a
card-reader, configured to the test delivery system, which makes the request to the candidate for a fixed number of times, but at random points in the session. The ID data is captured and ‘attached’ to the exam instance. The failure to standardise the service encounter, introducing verification and authentication devices is missionary work that the industry and test owners need to do.
2. Remote proctoring Some exam programmes are increasingly turning to remote proctoring as a safety feature, or as a complete replacement for human proctors/ invigilators. This can be helpful for providers who have learners that need accreditation by exam, or exam sponsors who need to build candidate volume, without the expense of test centre provision.
However, one facet of remote proctoring which has yet to be solved adequately is that of the cultural chasm between the proctor and the candidate population. The cultural acceptability of, for example, a proctor in Texas controlling a session in Hyderabad is a recipe for an imperfect service, with potential for confusion and non-compliant behaviour, regardless of at least a mutual understanding of a common spoken language in a high-stakes, pressured environment.
Remote proctoring as an additional safeguard to human invigilation would help prevent exam sessions being railroaded as we witness in the Panorama programme.
3. Invigilator Honour Code/ Contract The growth of high-stakes exams delivered through third-party contracted centres, typically to maximise coverage and control costs means that, unfortunately, compromise is inevitable. Exam sponsors who need global delivery are broadly familiar with the challenges of delivering a standardised environment to every candidate. For example, administrator exams to check/ provide knowledge updates and understanding of compliance are occasionally used, but rarely with the same rigour as the candidate’s exam. Stronger honour codes, coupled with service level agreement-driven contracts with centres can help, but fraudsters are still happy to take the risk.
4. CCTV The use of CCTV varies widely across the globe: the UK is often thought of as one of the largest users. However, a first pass of the Panorama footage shows no CCTV system within the test lab, and no web cams on the actual workstations (for the e-Assessment). Given the large fall in the price of providing live, time-coded webstreams over the past five years, it seems that some innovations seem to be passing exam sponsors and delivery companies by.
Many countries have a cultural distaste of CCTV and surveillance, which can compromise the ability to deliver an exam programme that utilises such technology. While internationally delivered programmes are helping to raise awareness of the demands of high-stakes assessment, it is rare that exam sponsors can impose strict conditions in every single country they deliver to.
5. Test Taker Responsibilities Test taker communities should be engaged, given advice, and mobilised to name and shame/ whistle-blow these centres and individuals that put their futures at risk. While this is a slow-burn strategic/ cultural change, some exam owners actively monitor chat rooms and restrict candidates suspected of malpractice from their programmes.
In some cultures, candidates who take their tests fairly and ‘play by the rules’ are not always
considered to be heroes, and their success isn’t always celebrated as an example to others. The internationalisation of many exam programmes means a battle for minds taking place. Where cheating is ‘what you need to do/ easier/ quicker/culturally expected’, versus taking an exam fairly and abiding by the programme fundamentals, so the result continues to have currency.
6. Using Big Data The Panorama programme showed how a group of candidates answering on paper bubble sheets were given the answers to 200 questions in just over 7 minutes. With an e-Assessment, key entry strokes and time monitoring software quickly show outlier behaviour, such as unusual pass rates for test centres. With (near) real-time processing of candidate responses, exam length and results, test sponsors are now able to diagnose outlier behaviour far quicker than a paper-based equivalent.
While exam sponsors and delivery companies can close rogue centres quickly (once sufficient evidence has been collected), gaining a criminal prosecution can often mean a lengthy legal process, if the indigenous society believes that an actual crime has been committed, or the prosecuting party thinks they have a realistic chance of a conviction!
Using big data can help keep invigilators and delivery centres in ‘the circle of trust’. Making them aware of what is being captured to protect candidates and themselves from fraudsters and compromising activities. There is plenty of evidence of rogue candidate groups hearing about ‘weak’ test centres or ‘fences’ where an exam can be compromised and a pass can be obtained more easily than what was anticipated by the programme owner. The use of large data processing, analytics, forecasting and decisions backed by evidence can all help.
Food for Thought
If you want an exam to be meaningful, and it means something to someone, regardless of how much it costs, there will always be a human propensity to ‘beat the system’ or at least make it work to your advantage.
Exam owners want more coverage and volume, placing pressure on delivery companies to do it cheaper with an ‘acceptable’ level of quality. Candidates want to engage with a fair and just exam system, which is credible, but accessible.
My fear is that exam programmes rely on confidence, as does most education provision. Exam owners and delivery companies need to do a lot more in convincing test takers, the public at large, and stakeholders of the advances being made and how combating fraud is a priority.