Response to Ministry of Justice’s published full year statistics on the use of languages services in courts and tribunals
The Ministry of Justice (MoJ) published its third statistical bulletin (Bulletin 3) relating to the use of language services in courts and tribunals on the 28th March 2013. It covers the first full year of implementation of the Framework Agreement (FWA) with Capita Translation & Interpreting. It is in a similar format to the previous two bulletins even though the adequacy of this approach has been questioned as it does not present a full picture of the language services required (paragraph 141 Justice Committee report 6/2/13) and contains no data about the Tiers of interpreters used or any form of quality indicator (as recommended in paragraph 177 Justice committee report 6/2/13).
The report may have been prepared by a statistician, but it is in no way impartial and its presentation is skewed towards supporting the status quo.
This response presents key data from the published statistical tables summarised into in a simple format, which allows a numerate reader to gain an overview of the performance and trend over the year that has been obscured by the official presentation.
Table 1 shows the total requests for language services by month, subdivided into those where the FWA delivered (Bulletin 3 category Fulfilled + category Customer did not attend), those where it failed to deliver (Bulletin 3 category Not fulfilled by supplier + category Supplier did not attend) and cancellations (Bulletin 3 category Cancelled by customer). Over the year Capita delivered (leaving aside all questions of quality and what was actually delivered) in only 80% of cases. It failed to deliver 9% of the time and 11% falls into the controversial category of Cancellations.
The trend over the year is clear:
1. an extremely poor start in February 2012 (59.6% delivery),
2. an improvement to a level which was still significantly below target and reached its peak (84% delivery) in August 2012. This was the last month included in the second statistical bulletin (Bulletin 2) released on the 18/10/12 immediately prior to the Justice Committee’s first oral evidence session,
3. a deterioration over the next 4 months (September – December 2012),
4. a significant deterioration in January 2013 (to 75.5% delivery).
Cancellations
Following publication of the first statistical bulletin (Bulletin 1) it was pointed out that it is highly misleading to include instances where the interpreter arrives so late that the court cancels in the category “Cancellations by customer” rather than as a failure to deliver by Capita. Yet on page 13 of the most recent Bulletin 3 the same definition is quoted as in Bulletin 1 i.e. “Requests may also fail because… the supplier… does not attend (or arrives so late that the job is cancelled)”. On page 11 the MoJ’s chief statistician distances himself from responsibility for the classifications: “The classifications used in this bulletin… are taken directly from the management information system, and are decided according to the rules laid down by the contractor.”
The figures also suggest that this category continues to be manipulated to provide an improved headline “success rate”. The three key indications showing this are:
1. The full year figure of 14,823 cancellations or 11.3% of total requests is extremely high and does not tally with the experience of interpreters working in the field prior to the introduction of the FWA e.g. interpreter cancellations at Immigration and Asylum Tribunals were very rare, yet under the FWA there were 4,138 cancellations i.e. 18% of all requests were cancelled!
2. The MoJ and ALS / Capita admitted to both the National Audit Office (NAO) and to the Justice Committee (JC) that cancellations had been wrongly logged and claimed that this related to a training issue which was resolved, so only February and March 2012 figures were affected. Here are the relevant extracts from the NAO and JC reports:
National Audit Office report published on 10/9/2012 paragraph 3.2 (page 21):
The actual performance was a little lower than this, as some of the bookings Capita/ALS recorded as customer cancellations were, in fact, only cancelled because the company could not fulfil them.6
Note 6: This situation continued until March 2012. In April the Ministry became aware of it and asked Capita and ALS to clarify to staff the difference between customer cancellations and unfulfilled bookings. While we have no reason to believe that mistakes were deliberate, they did overstate ALS’s performance. The data that the Ministry published about the ALS contract in May 2012 contained these same overstatements (Ministry of Justice, Statistics on the use of language services in courts and tribunals; Initial bulletin, 30 January 2012 to 30 April 2012).
Justice Committee report published on 6/2/2013 paragraph 140 (page 50):
140. In addition, we heard that cancellations by the customer might include cases where in reality ALS was unable to fulfil the request, for example, if a court rang ALS when somebody attended court too late for them to be of use, or with skills in the wrong language, and HMCTS or NOMS subsequently cancelled the requirement. The NAO established that complaints and material failures were in some cases being logged as customer cancellations.244 We were assured by Capita and the MoJ that this anomaly was a training issue which had been subsequently rectified.245
If the incorrect logging of requests as cancellations was limited to an early training problem resolved by April 2012 as Capita and the MoJ claim, why is there no reduction in the number of cancellations per month or the proportion of cancellations in relation to total requests from April 2012 onwards? Table 1 shows cancellations reported in Bulletin 3 of 715 and 10.6% in February 2012 (655 and 10.4% in Bulletin 1) and 1,194 and 10.7% in March 2012 (1,160 and 10.8% in Bulletin 1). The figures for the rest of the year have very clearly not reduced from this level either in absolute terms or as a proportion of total requests. In fact the rest of year monthly average is 1,291 cancellations and 11.4% of total requests.
3. Extremely late adjustments to the figures undermine their credibility. On page 11 of Bulletin 3 there is a section with the title “Comparisons with figures in previous bulletins”. It very reasonably points out that the status of requests near the end of a reporting period are particularly likely to be updated in the following bulletin. It goes on to say, in the second paragraph: “some differences will be observed between the February to August 2012 figures in this bulletin and those included in the bulletin published in October 2012, based on a snapshot of the system on 20 September 2012. The largest proportion of differences occurs in August 2012 figures.” Indeed, but why are substantial adjustments being made to the first quarter? Table 4 shows that the 695 additional requests for February to August shown in Bulletin 3 as compared to Bulletin 2 breaks down into 167 for August, 221 for the second quarter and 307 for the first quarter! It is true a new category has been included in Bulletin 3 – MOJ HQ & NOMS HQ - which was not in Bulletin 2, but it is immaterial as there were only 28 requests in total for MOJ HQ & NOMS HQ in the first 7 months of the contract and 18 of these appear to have been reallocated from the Prisons category i.e. included in Bulletin 2.
Total reported cancellations for the first 7 months of the contract to the end of August 2012 increased by 89 between Bulletin 2 (18/10/12) and Bulletin 3 (28/3/13). Usually adjustments would be made primarily to the most recent months (i.e. August, perhaps July) yet the analysis in Table 4 shows that 49 of those cancellations (i.e. more than half) relate to February – April 2012 (which had already been revised upwards by 131 cancellations in Bulletin 2). A revision of the first three months’ figures as published in Bulletin 1 on 24/5/12 (4 weeks after the period end) in the following bulletin is understandable, but why are those revised figures published on 18/10/12 (5 ½ months after the period end) revised again on 28/3/13 another 5 months later? Could it be to bring them in line with the higher rates of cancellation as a proportion of total requests in later months to make them look reasonable?
The total requirement for language services for HM Courts & Tribunals Service (HMCTS)
As in previous bulletins, these published statistics do not show the full picture for language services required by HMCTS. They only show the element of the requirement which goes through Capita and is recorded on its system. Clearly the current position cannot be adequately evaluated without this information or at least an estimate of it. The House of Commons Justice Committee report published on 6/2/13 recommended that the MoJ statistics bulletin should include data on the total HMCTS requirement:
141. It has been difficult to ascertain what percentage of the total bookings and requirements for the courts service ALS has been operating in each month since the MoJ partially reverted back to the old booking systems. Performance figures clearly do not reflect the company’s fulfilment against 100% of the requirements of HMCTS and they should be altered, retrospectively and in the future, to indicate this.
The MoJ has chosen not to implement this recommendation in the full year statistics and in their response to the Justice Committee published on 25/4/2013 admit an estimate of the total requirement has been made, but do not reveal it and say they will include it in future statistical publications at some unspecified time in the future (at the time of writing there were no entries for Language Services in the MoJ’s online Publication Schedule).
In the absence of any published figures from the MOJ the only publicly available estimate is the one quoted by Lord McNally in the House of Lords on 9/7/12: "we are talking about a system with some 800 requests a day for such interpretation.”
Table 1 takes this figure of 800 requests a day and multiplies it by the working days in the year to give an estimate of a requirement for 201,600 jobs for the first full year of the contract. If this estimate is correct, Capita only delivered 52% of HMCTS’s requirement in the first year of the FWA.
Complaints
Table 2 shows a summary of the data relating to complaints recorded on Capita’s system. The trend follows a similar pattern to the FWA delivery shown in Table 1. There were 6,417 complaints in the year, which equates to an average of 25 complaints every working day.
On page 12 Bulletin 3 identifies an extra 313 complaints relating to the first 7 months of the contract, which should have been shown in the previous bulletin i.e. the figures presented to the Justice Committee were understated and should have been 8% higher. This is a good illustration of why Capita, as the company whose services are being complained about, should not be controlling the complaints process.
Ineffective trials
The MoJ’s statistical bulletin contains a limited amount of information about the impact of interpreter availability on the effectiveness of trial hearings (on page 17). It shows that, for the year 2012, 608 of the 26,504 ineffective trials in Magistrates’ Courts (2.3%) were due to problems with interpreter availability, while for Crown Courts the percentage was 6.7% of ineffective trials.
Bulletin 3 does not show how this has changed over time as such a comparison does not give a flattering picture of the current system. Table 3 gives the comparators for the preceding five years taken from the Court Statistics Quarterly bulletin.
It shows very clearly that on all measures the introduction of the Framework Agreement has doubled the level of ineffective trials due primarily to interpreter availability in comparison with a stable level over the previous 5 years in both Magistrates’ Courts and Crown Court.
Frequency of publicly available statistics - a lack of transparency and accountability
At the outset of the FWA implementation the MoJ’s online publication schedule indicated that performance statistics would be published on a quarterly basis. Quarter 1 to the end of April 2012 was published on 24/5/12, but quarter 2 just disappeared from the publication schedule! Interpreters had no information about when the next set of statistics would be published. Bulletin 2 covering the 7 months to August 2012 was released on 18/10/12 just prior to the Justice Committee’s first oral evidence session, but there was no public advance notice of this or warning given to interpreters’ representatives giving oral evidence to the Justice Committee only 3 working days later.
In effect performance data has been withheld in order to support the status quo. This is not in the public interest as it is not conducive to fair debate over the FWA. For example the Justice Minister and the MoJ have made repeated public claims during late 2012 and early 2013 that FWA performance was improving – an assertion not applicable to the situation at the time and not based on the data they had available at the time. Withholding performance data from the public in order to maintain the status quo shows a worrying lack of transparency and accountability.
There are currently no entries for Language Services in the MoJ’s online Publication Schedule, so who knows when it will be convenient for the MoJ to next issue performance figures? In its recently published response to the Justice Committee report (page 7) the MoJ even refuses to provide the Justice Committee with monthly performance data and suggests the JC should wait for official statistics to be published. Here is the JC recommendation:
152. We are pleased to hear that service levels have improved markedly in recent months and that this will allow HMCTS to book all of the interpreting work it requires through Capita TI in the near future. We call on the Ministry of Justice to keep us apprised of fulfilment rates, and their estimation of the volume of work demanded by HMCTS that Capita TI are being asked to fulfil, on a monthly basis until we can be satisfied with the extent of improvement.
N.B. There is clear evidence that the HMCTS has not been able to book all of the interpreting work it requires through Capita TI.
Here is the MoJ’s official response:
We are committed to publishing data regularly as an official statistic. This ensures that data has been checked according to the relevant codes of practice. Monthly data on fulfilment rates and volumes of work are not checked to the same level as the official statistics and therefore the Department proposes that it continues to provide the Committee with the less frequent but more robust official statistics.
