OCR trains all reviewers on a yearly basis, before the summer review of marking period. Assessors complete a remote training package covering both the process of completing reviews of marking or moderation and the rules and approach about when to change marks or outcomes of moderation. They then have to pass a test to show that they have understood and can apply these rules appropriately. Assessors are only allowed to undertake reviews after they have passed this test.
OCR monitors reviewers throughout the review of marking and moderation period. For reviews of moderation, we check all moderation outcomes against the review of moderation reports in order to ensure that the rationale for any changes is appropriate and consistent with the outcome. If this is found not to be the case, the review is returned to the reviewer with feedback so that they can correct it before the outcome is processed and the centre (school or college) notified. If a similar issue is then found on subsequent reviews for the same reviewer, they can be stopped from completing any further reviews and will not be used again to complete reviews for the same component in future series.
We also monitor all reviews of moderation outcomes at weekly intervals throughout the review period in order to check that there are no patterns in changes to outcomes that suggest a potential problem with the original moderation decisions made during the series. If this monitoring identifies any potential errors in moderation, these are investigated, and if necessary, reviews of moderation are extended to cover all centres that further investigation has identified as being potentially impacted. Marks and where appropriate results will then be updated to correct the effect of any error which is identified, provided that a Learner's result will not be lowered.
In addition, at the end of the review period when all reviews of moderation are complete, OCR analyses all outcomes to check if there are any patterns at reviewer or component level that suggest that the approach to reviewing and rules about when to change or confirm outcomes are not being followed consistently. If this is found to be the case, the reviewer(s) in question will be given additional feedback and training before being allowed to complete reviews in future series.
OCR monitors reviewers throughout the review of marking and moderation period to ensure that they are making appropriate and consistent decisions and only changing marks to correct marking error. For reviews of marking, OCR monitors all reviewers at weekly intervals throughout the review period, using statistical monitoring comparing the reviewer change rate with the change rate of the rest of the panel. Where a reviewer’s change rate is different to that of the rest of panel, they will flag on the weekly report. They will then be suspended from reviewing and any scripts they were in the process of reviewing will be completed by other reviewers. Suspended reviewers will be asked to review the training materials again, and to take another test to check their understanding. The outcome of this test will determine whether reviewers are allowed to continue reviewing or not. Monitoring on a weekly basis will help OCR to ensure that any reviewers who are not reviewing appropriately are stopped in a timely manner.
We also monitor all reviews of marking outcomes at weekly intervals throughout the review period in order to check that there are no patterns in changes to marks that suggest a potential problem with the original marking during the series. If this monitoring identifies any potential errors in marking, these are investigated and if necessary reviews of marking are extended to cover all candidates that further investigation has identified as being potentially impacted. Marks and where appropriate results will then be updated to correct the effect of any error, after the impact of this on candidates and the potential for adverse effect has been considered on a case by case basis.
In addition to monitoring at weekly intervals, OCR will run a final report at the end of the reviewing period, and this will be analysed to identify any trends or patterns between panels, reviewers and across time. Where the final monitoring report suggests that this has not been fully effective in achieving consistency, OCR will address this by removing any reviewers who monitoring data suggests are not consistent, and by amending training, testing and monitoring processes as required.
OCR will undertake a yearly review of training and monitoring arrangements for reviews of marking and moderation after the summer series review of marking and moderation period. Any changes to arrangements resulting from this review will be made before the next summer review series.
For more information about our post-results services, please see the post-results services section. This covers how to request a review of marking or moderation, key dates and deadlines, our charging policy and links to the fees that are payable.
Further information can also be found in the JCQ post-results services booklet.