RATED Introduction

1

Background

This is the fourth edition of RATED, a directory intended to assist practitioners in selecting and applying risk assessment tools. Whether used as aids to initial decision making or case management in wider criminal justice settings, risk assessment instruments must be applied with an understanding of their respective strengths and weaknesses. In addition, to be used with any degree of confidence, risk instruments should have a sound empirical basis and validation history. The RMA’s ‘Framework for Risk Assessment Management and Evaluation’ (FRAME) highlights the importance of how this information is analysed, evaluated and then communicated in a meaningful way to inform decision making and action.

The Equalities Act (2010) has informed the latest edition of RATED. The Equalities Act maintains that the equalities duty of the public sector is to recognise differences based on social characteristics and to thereafter tailor the needs of individuals accordingly, to ensure the outcomes are equal for everyone in society. As part of its equalities duty, the RMA (2015) will “inform practitioners of relevant equalities considerations and help them apply appropriate risk assessment tools to specific populations.” Where the empirical research is available, the applicability of validated risk assessments to the relevant protective characteristics are considered: people with a mental impairment/learning disability (mental disability); people from an ethnic minority (ethnic group); females (gender); adolescents (age).

RATED is an ongoing project to be continuously updated and revised. Feedback and suggestions about new and emerging tools or new research studies related to the tools in the directory are welcomed.

1.1 Background to RMA The report of the Committee on Serious Violent and Sexual Offenders, chaired by Lord MacLean, highlighted the need for an independent public body to ensure the effective assessment and minimisation of risk. The Committee reported to Scottish Ministers in 2000 and its work informed the Criminal Justice (Scotland) Act 2003.

The legislation introduced a new sentence, the Order for Lifelong Restriction (OLR) and established the Risk Management Authority (RMA).

RATED page updated: September 2019 © Risk Management Authority 2019

As part of its duties, the RMA is responsible for preparing and issuing guidelines on the assessment and minimisation of risk; as well as publishing standards by which measures taken in respect of the assessment and minimisation of risk are judged.

In 2006, the RMA published the ‘Standards and Guidelines for Risk Assessment,’ focusing primarily on practice in relation to the Order for Lifelong Restriction (OLR).

A year later, the ‘Standards and Guidelines for Risk Management’ was published.

1.2 Risk assessment tools Risk assessment instruments are intended to assist the practitioner in different ways by anchoring the assessment in empirical evidence to identify relevant risk factors. They may be used for a variety of purposes: aiding decision making, planning treatment, predicting the risk of recidivism, using in specialist settings or for case management in criminal justice settings. It is important that risk assessment instruments are applied with an understanding of their respective strengths and weaknesses. Fazel and Wolf (2017) highlight that external validation, performance in the population to be assessed and sound methodology are pivotal factors in deciding whether a tool should be chosen. In addition, to be used with any degree of confidence, risk instruments should have a sound empirical basis and validation history.

The publication of RATED is intended to assist practitioners apply appropriate tools as part of a structured approach to risk assessment to assist with the identification of risk factors, needs and strengths of an individual. It should be noted that the application of a tool is one step in the risk assessment process; it is essential that the emerging information is analysed, evaluated and then communicated in a meaningful way to inform decision making and action.

The evolution of risk assessment has been well documented, with the general recognition that the incorporation of the ever growing research literature delivers an incremental improvement to each new generation of risk assessment instrument (Andrews, Bonta and Wormith, 2006). Actuarial instruments are described as nondiscretionary procedures, whereby the evaluator makes a decision based on fixed and explicit rules designed to predict the future (Hart and Logan, 2011). Actuarial instruments were designed specifically to assess risk or estimate the probability or likelihood of an outcome using ‘predictor variables’ (Hart, Douglas and Guy, 2016).

RATED page updated: September 2019 © Risk Management Authority 2019

The development of actuarial instruments has evolved over thirty years. Initially, instruments were based on static risk factors that may contribute to broad classifications based on longer term risk. These have since progressed to more sophisticated approaches that contain a range of dynamic factors and are designed to assist in an understanding of an individual and his/her behaviour, with the aim of reducing the likelihood of that behaviour through appropriate interventions (Hanson, 2007).

A further suite of risk assessment instruments, commonly known as ‘structured professional judgement (SPJ) instruments, are distinct from actuarial tools in that they are guidelines designed to reflect the discipline with respect to scientific knowledge and practice (Hart and Logan, 2011).

Whilst they incorporate the empirical evidence, they also include consideration of other clinical factors and do not lead to a quantified ‘score.’ In this sense, instruments can assist with the identification of risk factors and strengths of an individual (Baird 2017). There are a range of SPJ approaches, ranging from those requiring scenario planning and formulation to others that require an appraisal of the risk and protective factors of individual cases. Several instruments are primarily, although not exclusively, developed for application with mentally disordered individuals who offend; these are most commonly used by mental health professionals, psychologists and psychiatrists. Dynamic instruments allow for reassessments to be carried out to evaluate the effectiveness of treatment in order to guide future decisions. Each SPJ tool focuses on a particular offence type, such as sexual or adolescent violence, and, as such, these instruments are predominantly utilised in specialist settings for the purposes of detailed and individualised risk management planning (Abbiati et al., 2014; Lloyd, 2019; Logan 2016).

It must be highlighted, however, that concerns have been raised about an overreliance on the findings of risk instruments in proceedings that have a bearing on sentencing and/or restriction of liberty (Singh, Grann and Fazel, 2011). The implications from this are that all types of risk instruments share a common weakness if relied upon solely to evaluate ‘individual’ risk.

RATED page updated: September 2019 © Risk Management Authority 2019

Running in parallel with this development has been an on-going academic debate concerning the applicability of group data to the individual and the relative superiority of actuarial and clinical approaches (Quinsey et al., 2006). A useful guide for practitioners is an approach that bridges both perspectives, drawing upon research work highlighting the limitations of the actuarial approach (Cooke and Michie, 2010; Hart, Michie and Cooke, 2007); while endorsing the counsel of others that abandonment of the actuarial tradition is unwise (Craig and Beech, 2010).

In line with the conclusions of researchers (Boer, 2006; Rettenberger et al., 2009), the RMA Standards and Guidelines encourage a ‘convergent’ approach to risk assessment which incorporates the use of both actuarial and SPJ tools.

Furthermore, structured professional judgement and decision-making are necessary components of all risk assessment practice, regardless of the type of instruments employed or the professional background of the practitioner. It is within this context that the RATED has been developed.

RATED page updated: September 2019 © Risk Management Authority 2019

2

RATED AIMS AND OBJECTIVES

2.1 FRAME Approach The RMA is tasked with promoting effective risk assessment and management practice. In July 2011, the RMA, along with Scottish criminal justice agencies and the Scottish Government, published the FRAME policy paper. This established the policy direction in which a shared consistent framework to promote proportionate, purposeful and defensible risk practice could be achieved. FRAME proposes an approach that is grounded in the principles of risk practice to ensure all professionals are aware of the processes and skills associated with effective risk assessment and management. It also helps professionals understand the contribution that different kinds of instruments make to this practice. Moreover, it firmly promotes the practitioner as the ‘assessor’ who skilfully, knowledgably and within the parameters of his/her competencies, applies appropriate tools in a structured, meaningful process. Logan (2016) recommends that risk judgment can be substantially structured by utilising formulation, which organises the information into an explanation for why it happened and the circumstances in which it could possibly transpire again.

The framework promotes risk assessment practice that makes meaningful use of risk assessment tools without being over-reliant on them, by ensuring that the valuable contribution of such instruments is located within a structured approach to risk assessment which recognises the strengths and limitations of such tools and the importance of professional/clinical judgement. Risk assessment tools do not necessarily provide the comprehensive range of risk, responsivity and protective factors that inform an understanding of individual risk. Extrapolating from this, these tools should be used to inform rather than replace professional/clinical judgement.

2.2 Purpose of RATED This is the fourth edition of RATED, a guide intended to assist practitioners in applying risk assessment tools as part of the FRAME approach to risk assessment to assist with the identification of risk factors and strengths of an individual. RATED also intends to keep researchers updated about the most recent developments.

RATED page updated: September 2019 © Risk Management Authority 2019

The publication of RATED aims to provide a summary of the empirical evidence to inform a balanced and individualised approach to assessment, as well as to contribute to effective and ethical practice (Andrews and Bonta, 2010). The RATED directory is intended as a resource for Accredited Risk Assessors undertaking assessments for the High Court in Scotland when the court is considering imposing an OLR, in addition to other practitioners involved in risk assessment and risk management.

The objectives of the current publication are as follows: • To provide relevant research information on each assessment tool included in the directory, which includes (but is not limited to) validation evidence.

• To provide an impartial and factual account of the strengths and limitations of each instrument.

•To provide a guide for assessors to consider when applying a tool as part of a holistic risk assessment process.

Risk assessment instruments may focus on specific types of risk developed and/or particular groups. This directory is structured by type of tools. This introductory section provides further background to the purpose of RATED and the criterion used to rate tools. The categories of tools are the following: • General Risk Assessment: those tools that can be used for general application for risk.

• Violence Risk: instruments to establish the risk of violence.

• Youth General Assessment: similar to the General Risk Assessment category, but for use with adolescents.

• Youth Violence: instruments for measuring risk of violence in adolescents.

• Youth Sexual Violence: tools designed for measuring sexual violence risk in adolescents.

• Intimate Partner Violence and Stalking: instruments that relate to intimate partner violence and/or stalking.

• Sexual Offending: tools relating to sexual violence risk or internet offending.

• Responsivity Issues: this category consists of instruments for individuals with personality disorders or learning disabilities.

RATED page updated: September 2019 © Risk Management Authority 2019

Whilst every effort has been made to ensure the accuracy of the information presented in this publication, given the evolving nature of research relating to risk assessment and risk management, the evidence base is subject to change.

Practitioners are, therefore, encouraged to keep abreast of the emerging evidence when administering the assessment tools. Additionally, practitioners are invited to contribute to the development of the evidence base and subsequent editions of this publication by providing feedback on the contribution of specific instruments to their practice by emailing RATED@rma.gov.scot.

2.3 Responsivity Considerations The ‘risk-need-responsivity principles,’ suggest that effective interventions are proportionate to the likelihood of re-offending, address the needs related to offending, and are responsive to a range of issues that may influence a person’s ability to respond to interventions (Andrews and Bonta, 2010; Taxman et al., 2006).

Hubbard and Pealer (2009) found that the more ‘issues’ an individual has, the less likely a programme will be successful in achieving what it aims to do. To that end, strategies that are informed by responsivity issues are more likely to be effective and reduce recidivism (Beaver and Schwartz, 2016).

The RMA’ s (2018a) revised Standards and Guidelines highlights the importance to practitioners of ensuring that risk assessment and management practices do not disadvantage those with ‘protected characteristics.’ This is entrenched within the Equalities Act (2010), which attributes ‘equalities duty’ upon public sector organisations. The purpose of this is to recognise differences based on a number of protected characteristics and tailor services accordingly to ensure that people are not disadvantaged on the basis of those characteristics. The protected characteristics are: 

Ethnicity



Age



Mental disability



Religion



Gender



Gender reassignment



Sexual orientation

RATED page updated: September 2019 © Risk Management Authority 2019



Marriage/civil partnership



Pregnancy



Physical disabilities

As part of its equalities duty, the RMA (2015) seeks to “inform practitioners of relevant equalities considerations and help them apply appropriate risk assessment tools to specific populations.” Since RATED is a means by which the RMA advances this objective, the equality and diversity guidance laid out in the Standards and Guidelines (RMA, 2018a) informs the content of RATED. An Equality Impact Assessment (EQIA) was also carried out to consider the role of protected characteristics in the publication of RATED.

When compiling a new edition of RATED, the available published literature relating to instruments was reviewed, including studies that have examined their applicability to specific groups. Where the empirical information is available, the applicability of risk assessments to the relevant protected characteristics is considered. Of the ten protected characteristics, a number have limited if any research evidence. Those that have attracted most attention in the literature are: 

people with a mental impairment/learning disability (mental disability)



people from an ethnic minority (ethnic group)



females (gender)



adolescents (age)

The research evidence in relation to the remainder of protected characteristics1 is lacking, with only a handful of studies available. For instance, transgender issues are explored in relation to the youth sexual violence tool MEGA♪ (Miccio-Fonseca, 2018); whilst a study by Bhutta and Wormith (2016) incorporated a measure of religiosity and spirituality into the LS/CMI to test its effectiveness in a religious country.

1

The other protected characteristics within the UK Equalities Act (2010) are: religion, sexual orientation, marriage/civil partnership, pregnancy, gender reassignment (both biological changes or social functioning ones) and physical disabilities.

RATED page updated: September 2019 © Risk Management Authority 2019

The existing research base is limited to the extent that many tools do not have published studies pertaining to protected characteristics. Moreover, when protected characteristics are reviewed in relation to risk assessment tools, the focus tends to be broad, e.g. mental ability broadly is reviewed; yet specific mental disorders receive less attention. If further research emerges then future editions of RATED may create new categories for these protected characteristics.

In the application of tools, practitioners should be aware of the available research, or lack thereof, in relation to applicability to persons with a protected characteristic.

2.4 Developments in RATED Fourth Edition A number of significant changes have been made to the fourth version of the directory. These include a range of amendments to the format, layout and content of the directory, a summary of which is provided below.

Rearrangement of the directory entries: •Learning Disabilities and Diagnostic/Personality Assessments were previously in separate categories. These four categories have now been amalgamated into a new category entitled ‘Responsivity Issues,’ documenting those tools specific to disorders that could influence offending.

• The Stalking category has been combined with the ‘Domestic Violence’ category, since these types of offences can often be interlinked (see Kerr, 2018). Additionally, the term ‘Domestic Violence’ has been replaced with the more commonly used term within the field ‘Intimate Partner Violence’ to give the new category of ‘Intimate Partner Violence and Stalking.’

•‘Internet Offending’ (offences related to indecent images of children) has been grouped with ‘Sexual Violence Risk’ and the category has been renamed as ‘Sexual Offending.’

Regrouping of reference lists: Reference lists are included as a separate document for each of the eight sections of tools. To facilitate accessibility for users, references within the lists are grouped by tools.

RATED page updated: September 2019 © Risk Management Authority 2019

Inclusion of new tools: Upon reviewing the literature for the current risk assessment tools, a number of new tools were identified. Following a review of their usefulness and applicability, the decision was made to include the majority of them in this revised edition. The additional entries within the directory reflect both the extent of the review for this edition and the speed of development in the field of risk assessment. The new additions to this edition of RATED are: 

Oxford Risk of Recidivism (OxRec) tool in the General Risk Section.



Oxford Mental Illness and Violence Tool (OxMIV) in the Responsivity Issues section.



Barr-2002R, Child Pornography Offender Risk Tool (CPORT), Sexual Offender Treatment Intervention and Progress Scale (SOTIPS) and Vermont Assessment of Sex Offender Risk 2 (VASOR-2) in the Sexual Offending section.



Terrorist Radicalization Assessment Protocol-18 (TRAP-18) and Workplace Assessment of Violence Risk (WAVR-21) in the Violence Risk section.



AssetPlus in Youth General Risk Assessment Section.



Assessment, Intervention and Moving On Project Version 3 (AIM3), The AIM Project Assessment Models for Children under the age of 12 years old (AIM Under 12s), Protective and Risk Observations for Eliminating Sexual Offence Recidivism (PROFESOR) and Technology-Assisted Harmful Sexual Behaviour: Practice (TA-HSB Guidance), all in Youth Sexual Violence Risk.



Structured Assessment of Positive Factors for Violence Risk: Youth Version (SAPROF:YV) and Short Term Assessment of Risk and Treatability: Adolescent Version (START:AV) in the Youth Vioelnce Assessment section.

A number of tools have been upgraded from awaiting validation status in the third edition of RATED (RMA, 2015) to validated in this edition: 

Brief Spousal Assault Form for the Evaluation of RISK (B-SAFER) in the Intimate Partner Violence and Stalking section.



Comprehensive Assessment of Psychopathic Personality (CAPP) in the Responsivity Issues section.



Risk for Sexual Violence Protocol (RSVP) in the Sexual Offending section.

RATED page updated: September 2019 © Risk Management Authority 2019



The Multiplex Guided Inventory of Ecological Aggregates for Assessing Sexually Abusive Adolescent and Children (MEGA) in the Youth Sexual Violence section.



The Youth Assessment and Screening Instrument (YASI) in the Youth General Risk section.

Rewording of rating criteria: The rating criteria refers to the amount of evidence available for a particular element (e.g. inter-rater reliability). The terminology has been changed in this version of RATED: • For zero bars, the wording of ‘no or poor’ has been changed to ‘insufficient.’

• For three bars, the word ‘little’ has been replaced with ‘preliminary.’

• For six bars, the term ‘some’ has been substituted with ‘intermediate.’

• For nine bars, the word ‘high’ has been replaced with ‘sufficient.’

Hyperlinks to research: To improve the accessibility to research, where available hyperlinks have been included to the studies cited in tool entries. The hyperlink will generally take the user to the journal’s page, which contains the article’s abstract. Users may then check whether they have access to the full article through their institution or they may have the option to purchase it on the journal’s landing page.

In cases where the full article is already freely available on the internet (for instance, through ResearchGate) the hyperlink to this is included to facilitate users’ access to research. In a small number of cases, the research cited is not available online and, hence, no hyperlink is included.

Hyperlinks are available within individual tool entries and in the list of references for each of the eight sections.

RATED page updated: September 2019 © Risk Management Authority 2019

2.5 Using RATED

Using RATED to select a risk assessment tool The use of suitable risk assessment tools is an important concern for practitioners and agencies involved in the management of offenders. RATED is designed to inform and support decisions regarding which tools to use by: • Describing the purpose and design; • Providing information on the evidence base for the tool design; • Evaluating the validation literature available; • Setting out the appraisal against the assessment framework.

When using RATED, practitioners should consider the following: • The type of risk that they wish to evaluate (general risk of reoffending, risk of violence, sexual offences, intimate partner violence or stalking); • The age, gender, ethnicity, mental state and cognitive abilities of the individual; • The performance of tools with respect to the criteria outlined which includes validation history, empirical grounding, inter-rater reliability and ability to identify targets for intervention.

Where they have the necessary competencies and training, practitioners can use with confidence validated tools that possess a robust validation history and empirical grounding. These tend to be tools that have also evidenced sufficient inter-rater reliability2, specificity3, sensitivity4 and predictive accuracy5 in identifying individuals at risk of re-offending. Practitioners should use tools with caution that possess many of the essential attributes and have the potential to demonstrate all of these in the future (with further studies and/or evidence). Reports detailing findings based on the use of tools that are awaiting validation should outline the limitations of their use in this respect.

Inter-rater reliability refers to the degree in which two or more assessors consistently rate items within a tool.

3 Specificity relates to the ability of a test to classify when an individual does not possess a particular characteristic.

4 Sensitivity refers to the extent to which an instrument can correctly classify when an individual possesses a particular characteristic. Sensitivity is inversely proportional with specificity, whereby one increases as the other decreases.

5 The predictive accuracy of a tool refers to its ability to distinguish between certain populations, such as those who reoffend and those who do not.

2

RATED page updated: September 2019 © Risk Management Authority 2019

Practitioners should use tools with extreme caution that possess few of the essential attributes, but appear to have the potential to meet all of these in the future (with further studies and/or evidence). Reports detailing the findings based on the use of such tools must include a justification for selecting them and should outline the limitations of their use.

In cases involving the 17-21 year old age group, practitioners should use their judgement as to whether to apply instruments designed for young people, or those designed for an adult population. To inform this judgement, specific consideration should be given to the extent to which the tool incorporates evidence of its validity with young people and the impact of child developmental processes of the young person being assessed. Similarly, one should give careful consideration as to whether the selection of a given instrument is appropriate for certain minority groups, e.g. those with mental health problems, cognitive impairments and learning disabilities. A good guide to an instrument’s suitability for a particular group is whether a sample from the target group has been included in the original research sample or been the subject of a successful pilot study.

In all cases, practitioners should have the necessary training for using the tool(s) of their choice; be aware of their limitations and the caveats of their use; be in a position to discuss these and to evidence their assessment. Importantly, these limitations should be clearly outlined in the assessor’s report to inform future decision-making. Above all, risk assessment tools should be used as part of a risk assessment process to inform rather than replace professional judgement.

Practitioners are advised to review the range of guidance documents dealing with risk assessment. These include the RMA Standards and Guidelines, FRAME and the RMA accredited manner for risk assessment for Risk Assessment Orders (RAO) or Interim Compulsion Orders (ICO) when the High Court is considering an OLR.

The Standards and Guidelines states that practitioners should select risk assessment instruments appropriate to the risk needs and characteristics of the individual being assessed, as well as in accordance with their own competencies, experience and training. Practitioners should also be aware of the validity and RATED page updated: September 2019 © Risk Management Authority 2019

limitations of any tool that they use (RMA, 2018a). RATED sets out the RMA’ s evaluation of the evidence available for each instrument and highlights any limitations or caveats that professionals should take into account when using a tool as part of a holistic risk assessment process.

The directory also highlights that there may be cases (perhaps with unusual types of offending) where available instruments have little or no validity, wherein the following activities are recommended: • Review the relevant publications to look at what the latest academic and clinical literature describes in relation to the type of offender and the approach to the risk under consideration; • Seek the views of other experts on the approach to be taken in the case; • Gather as much information as possible to support the formal risk assessment; • Seek supervision from a sufficiently experienced and expert colleague; • Use the structured professional judgement framework, as set out in this manner, to guide the process of case formulation and reaching a risk judgement; • Be explicit in the report about the limitations of the approach used.

RATED page updated: September 2019 © Risk Management Authority 2019

3

EVALUATION PROCESS

3.1 The evaluation process The RMA has developed a process for evaluating1 risk assessment tools and this has been followed for each tool presented in this document. This evaluation process will also be applied in future to assess newly developed tools. The RMA researcher working on this update to RATED used a standard pro-forma to describe each tool based on the summarised evidence and the adopted evaluation framework. This provides individual assessments of the tool against each of the framework criteria and comments to support them.

The RMA welcomes applications for inclusion in the directory, as well as any comments its users.

Adopting an evaluation framework The adopted framework comprises a number of criteria considered essential for the evaluation of risk assessment tools, building upon earlier work by McIvor, Kemshall and Levy (2002) and the findings of other researchers (Heilbrun, Rogers and Otto, 2002; Rogers 2000). Criteria considered as essential are:

• Empirical Grounding • Inter-Rater Reliability • Validation History

Compilation of evidence and evaluation Evidence collated for the current audit, comprised of publications (including ‘grey literature,’ such as doctoral theses and dissertations for Master’s Degree and other unpublished material) and evidence of ongoing research.

The following criteria were used to select research studies for inclusion in the directory:

In this document, the word ‘evaluation’ is used to refer to the assessment of the tools’ performance with respect to the criteria framework outlined.

1

RATED page updated: September 2019 © Risk Management Authority 2019

• Authorship, i.e. whether the author(s) were, or were not, involved in the validation of the tool under consideration, with greater weight given to studies conducted by independent researchers rather than the author(s) of the risk assessment tool; • Place of publication, with greater weight given to publications in peer-reviewed journals; • Date of publication, with greater weight given to more recent publications; • Sponsorship, i.e. whether the study has been commissioned and/or funded by a government or statutory agency, with greater weight given to independent studies; • Relevance to UK population; i.e. whether the study had employed a sample of individuals from the nations within the UK; • Target population, i.e. male/female offenders, forensic mental clients, and children and young offenders; • Size of study population, with greater emphasis on evidence drawn from large sample populations; • Focus of study, taking into account whether specific issues have been considered, such as predictive validity, inter-rater reliability, significance/ usefulness,2 sensitivity and so on.

The following information was provided for each tool listed within the directory: • The title and any acronym by which it is known; • The publisher(s)/developer(s) and the year of publication (where known); • A brief description of the risk assessment tool; • The type of risk addressed, i.e. the risk of general, sexual and/or violent offending, or any combination of these; • An outline of its validation history, considering studies carried out in the United Kingdom and internationally; this includes details of the study focus and setting (community, secure unit, prison or mental health institution), the size of the sample and type of population used; • Evidence of inter-rater reliability, significance/usefulness; •Strengths, practice considerations, and future research directions.

2

Significance/Usefulness of a tool refers to its evidential basis, predictive power and clinical utility in enabling the practitioner to undertake risk formulation and risk management planning.

RATED page updated: September 2019 © Risk Management Authority 2019

Tools are divided up into ‘Validated’ and ‘Awaiting Validation.’ Validated tools are those which have sufficient research studies evidencing their suitability to be used within the field. Due to the scope of research available on the, validated tools have more extensive entries than those that are ‘awaiting validation.’ A further distinction between these categories is that to rate a tool in as validated, the RMA considers the existence and quality of the validation evaluation studies, assessed on the basis of the availability of two or more independent papers written by different authors and published in peer reviewed journals.

Tools that are awaiting validation currently do not have the sufficient research evidence to be considered validated; although these have the potential to become validated in future.

The RMA liaised directly with tool authors and publishers wherever possible to compile this material accurately.

3.2 Criteria definitions and performance measurement scales The following criteria have been used to identify the strengths and weaknesses of each risk assessment tool listed within the current directory. It is important to note that these criteria have not been applied to risk assessment tools awaiting further validation. This is due to the limited research available relating to the psychometric properties of these risk assessment instruments.

3.2.1 Empirical Grounding Empirical grounding examines the scientific and theoretical underpinnings of the risk assessment tools presented in this document. For example, a risk assessment tool based on sound theoretical evidence and / or other extensive scientific findings observed in prior research would be considered to have a high level of empirical grounding. Higher levels of empirical grounding may, therefore, increase the utility of the instrument in assessing the risk posed by an individual.

3.2.2 Inter-Rater Reliability Inter-rater reliability refers to the degree to which two or more assessors are consistent in their ratings when using the same risk assessment tool. There are two types of statistics used to estimate this: the Intra-Class Correlation (ICC) and Kappa RATED page updated: September 2019 © Risk Management Authority 2019

(K) Coefficients (see section 5 for further details). Researchers must be able to demonstrate that the risk assessment instruments are reliable; without reliability results, the risk assessment’s performance cannot be replicated and validation cannot be attained. Even if a tool had predictive validity, its usefulness will be undermined if it is applied inconsistently by different practitioners (Taxman, 2018).

It is thus of utmost importance for a risk assessment tool to have high inter-rater reliability whereby assessors using the same tools score the items similarly.

3.2.3 Validation History To rate a tool in this area, the RMA considers the existence and quality of the validation evaluation studies, assessed on the basis of the availability of two or more independent papers written by different authors and published in peer reviewed journals. The papers are required to have examined the predictive validity of the tool and/or its practical usefulness for the assessment and management of risk of harm to others. This approach accommodates concerns that have been raised in the literature that different research designs may be appropriate for identifying the properties, strengths and limitations of various types of instruments.

The validation history criterion is split into four subsections which are as follows: a. General Predictive Validity This item refers to the capability of the risk assessment tool to discern between the recidivist and non-recidivist populations.

The recognised statistical measure for this is the receiver operating characteristic curve (probability that a recidivist would receive a higher rating than a nonrecidivist) (Taxman, 2018). b. Applicability: Female Offenders This item refers to the validity of the risk assessment tool for women and girls. c. Applicability: Ethnic Minorities This item refers to the validity of the risk assessment tool for ethnic minority populations. d. Applicability: Mental Disabilities. This item refers to the validity of the risk assessment tool for those with mental disorders and/or learning disabilities.

RATED page updated: September 2019 © Risk Management Authority 2019

Tools are rated on these criteria using the following performance scale:

0 Bars - No evidence of or evidence indicating insufficient empirical grounding, inter-rater reliability or validation.

3 Bars – Preliminary evidence of empirical grounding, inter-rater reliability or validation.

6 Bars – Intermediate evidence of empirical grounding, inter-rater reliability or validation.

9 Bars – Sufficient evidence of sufficient empirical grounding, inter-rater reliability or validation.

For tools that are ‘awaiting validation,’ this section is entitled ‘Tool Development’ and covers findings of studies in relation to predictive accuracy. ‘Tool Development’ will also cover the empirical grounding of a tool and studies relating to inter-rater reliability where these are available.

3.2.4. Contribution to Risk Practice This section in tool entries supports the tiered approach to risk practice promoted within the RMA’s Framework for Risk Assessment, Management and Evaluation (FRAME), in particular the first three standards relating to: (1) Risk Assessment; (2) Planning and Responding to Change; (3) Risk Management Measures.

RATED page updated: September 2019 © Risk Management Authority 2019

This section does not incorporate the ratings scales observed in the previous criterion; rather, it provides a qualitative evaluation of the properties of the instrument and its contribution to risk practice.

3.2.5 Other Considerations This final section documents other elements to be considered before using the tool, e.g. whether it has a clinical over-ride feature to allow the risk rating to be readjusted at the assessor’s discretion. Also noted in this section are any other research findings about the tool that do not relate to its inter-rater reliability or predictive accuracy. In tools that are ‘awaiting validation,’ this section is entitled ‘General Notes’ and covers similar areas.

RATED page updated: September 2019 © Risk Management Authority 2019

4

STATISTICAL TERMINOLOGY

Authors of reports and studies on risk assessment tools adopt a range of descriptions of statistical outcomes. For the sake of consistency in the tables below, the RMA has adopted the following descriptors drawing from the most prevalently used terminology in the literature.

There is a useful explanation that uses an analogy of the bull’s-eye on a dartboard: actually hitting the bull’s-eye represents accuracy; landing shots together indicates good reliability. Considering this, hitting the bull’s-eye and landing all the shots together would convey both accuracy and precision (Viera and Garrett, 2005).

Statistical Analyses Definition

Interpretation

The Brier score is

The Brier score

defined as the average

ranges from 0 to 1.

quadratic difference

The best possible

between the predicted

score is 0, indicating

probability and the

total accuracy. The

binary outcome. Its

lowest possible

purpose is to measure

score is 1, meaning

the ‘calibration’ of a set

that the forecast

of probabilistic

was wholly

predictions.

inadequate.

Intra Class Correlation

An ICC score

ICC values range

Coefficient (ICC) –

represents the

from 0 to 1 and are

descriptors;

estimation of the

typically reported

poor/moderate/excellent

correlation between two

with two decimal

scores. It measures the

points, e.g. .75.

magnitude of agreement

Cicchetti (1994)

for inter-rater reliability.

recommends the

Brier Scores

following thresholds:  <.40 = ‘poor’  .40 to .75 = ‘moderate’ RATED page updated: September 2019 © Risk Management Authority 2019

 .75 to 1.0 = ‘excellent’ Kappa (.) Coefficient –

The Kappa Coefficient

A Kappa is always

descriptors;

(.) measures the

less than or equal to

poor/average/excellent

agreement between two

a value of ‘1.’ A

individuals.

value of ‘1’ implies perfect agreement and a value of ‘-1’ implies perfect disagreement.

Recommended thresholds are:  < 0 Less than chance agreement  0.01– 0.20 poor agreement  0.21– 0.60 average agreement  0.61– 0.99 excellent agreement

Effect sizes

The effect size

Effect sizes can be

quantifies the size of the

interpreted in terms

difference between two

of the percentiles or

groups. It is calculated

ranks at which two

as the standardised

distributions

mean difference

overlap.

between the two groups: Interpretations of mean of Group A minus

effect sizes are

mean of Group B; the

dependent on the

total of which is divided

assumptions that the two groups are

RATED page updated: September 2019 © Risk Management Authority 2019

by the standard

normally distributed

deviation.

and have the same standard deviations.

Pearson Correlation

Pearson correlation

The values of r can

Coefficient– descriptors;

coefficient measures the

range from ‘-1’ to ‘1,’

small/moderate/large

association between a

with ‘0’ indicating

predictor variable and

that there is no

the outcome.

relationship between the predictor variable and the outcome.

Positive values indicate that the high scores are associated with increased recidivism; whereas negative values indicate that high scores are associated with decreased recidivism.

According to Cohen (1988), r values may be interpreted as follows: .10 are small, .25 are moderate, .40 are large.

Sensitivity and Specificity

RATED page updated: September 2019 © Risk Management Authority 2019

Sensitivity is the ability

The higher the value

of a test to correctly

of sensitivity, the

classify an individual as

greater the ability of

possessing a particular

the measure being

characteristic (e.g.

tested to correctly

offending). Specificity is

identify individuals.

the ability of a test to

For instance, a

classify an individual as

sensitivity of 62% on

not possessing a

a risk assessment

particular characteristic

tool indicates that it

(e.g. not offending).

has the ability to

Sensitivity is inversely

correctly classify

proportional with

just under two-thirds

specificity, meaning that

of individuals who

as the sensitivity

will reoffend. For

increases, the specificity

specificity, higher

decreases.

values indicate the ability of a measure to correctly identify who will not possess certain characteristics (e.g. who will not go on to reoffend).

Z+ - descriptors;

Measures the

According to Cohen

small/moderate/large

association between the

(1988), d values

predictor variable and

may be interpreted

the outcome. These two

as follows:

groups usually comprise

.20 is considered

of the (1) recidivists and

‘small’;

(2) the non-recidivists,

.50 is considered

separated by the

‘moderate’;

difference in their scores .80 is considered obtained on risk

‘large’.

assessments.

Receiver Operating

In the context of risk

Several different

Characteristic (ROC) Curve –

assessment, the ROC

indicators can be

descriptors;

curve is a plot that

calculated from

low/moderate/high

shows the probability

ROC curves. The

RATED page updated: September 2019 © Risk Management Authority 2019

that a measure will

most commonly

correctly identify

used indicator is the

persons as recidivists or

‘area under the

non recidivists

curve’ (AUC) (see

(Mossman, 1994; Rice

below).

and Harris, 1995).

It is a plot of the ‘hits’ (the proportion of recidivists correctly identified as recidivists) against the ‘false alarms’ (the proportion of non-recidivists identified as recidivists).

Area Under the Curve (AUC)

The area under the

AUC values can

– descriptors;

curve (AUC) for the

range from ‘0’ to ‘1’.

low/moderate/high

ROC curve is a useful

They can be

summary statistic for the

interpreted as the

extent to which a

probability that a

measure discriminates

randomly-selected

between recidivists and

recidivist has a

non recidivists.

worse score than a randomly-selected non-recidivist.

Values between ‘.51’ and ‘1.0’ indicate positive associations with recidivism; whilst values between ‘0’ and ‘.50’ indicate that predictions are no better than chance.

RATED page updated: September 2019 © Risk Management Authority 2019

Using Cohen’s (1988) d values as a guide, AUC values may be interpreted as follows: .556 is considered ‘low’; .639 is considered ‘moderate’; .714 is considered ‘high.’

RATED page updated: September 2019 © Risk Management Authority 2019

Reference List Abbiati, M., Laix, J., Gasser, J. and Malin, V. (2014) ‘Predicting physically violent misconduct in prison: a comparison of four risk assessment instruments.’ Behavioral Science and the Law 37(1), 61-77. Access Here.

Andrews, D., Bonta, J. and Wormith, S. (2006) ‘The Recent Past and Near Future of Risk and/or Need Assessment.’ Crime & Delinquency 52(1), 7-27. Access Here.

Andrews, D. A. and Bonta, J. (2010) The Psychology of Criminal Conduct (5th edition).

Cincinnati, Ohio: Anderson Publishing. Access Here.

Baird, C. (2017). A question of evidence part two. Summary and conclusion. Madison, WI: National Council on Crime and Delinquency. Access Here.

Bhutta, M. H. and Wormith, J. S. (2016) ‘An examination of a risk/needs assessment instrument and its relation to religiosity and recidivism among probationers in a Muslim culture.’ Criminal Justice and Behavior 43(2), 204-229. Access Here.

Boer, D. P. (2006) ‘Sexual offender risk assessment strategies: is there a convergence of opinion yet?’ Sexual Offender Treatment 1(2), 1-4. Access Here.

Cohen, J (1988) Statistical Power Analysis for the Behavioral Sciences. New York: Routledge. Access Here.

Cooke, D. J. and Michie, C. (2010) ‘Limitations of diagnostic precision and predictive utility in the individual case: A challenge for forensic practice.’ Law and Human Behavior 34(4), 259-274. Access Here.

Craig, L. A. and Beech, A. R. (2010) ‘Towards a best practice in conducting actuarial risk assessments with adult sexual offenders.’ Aggression and Violet Behavior 15(4), 278-293. Access Here.

Fazel S. and Wolf, A. (2018) ‘Selecting a risk assessment tool to use in practice: a 10point guide.’ Evidence-Based Mental Health 21(2),41-43. Access Here.

RATED page updated: September 2019 © Risk Management Authority 2019

Hanson, R. K. (2007, March) ‘How should risk assessments for sexual offenders be conducted?’ Paper presented at the Fourth Annual Forensic Psychiatry Conference, Victoria, British Columbia, Canada. [Not accessible] Hart, S. D., Douglas, K. S. and Guy, L. S. (2016) ‘The Structured Professional Judgment Approach to Violence Risk Assessment: Origins, Nature and Advances.’ In L. Craig and M. Rettenberger, M. (eds.) The Wiley Handbook on the Theories, Assessment and Treatment of Sexual Offending. Hoboken, New Jersey: John Wiley and Sons, 643-666. Access Here.

Hart, S. D. and Logan, C. (2011) ‘Formulation of violent risk using evidence based assessment: the structured professional judgment approach.’ In D. Sturmey and M. McMurran (eds.) Forensic case formulation. Chichester, UK: Wiley Blackwell, 83-106.

Access Here.

Hart, S., Michie, C. and Cooke, D. (2007) ‘Precision of actuarial risk assessment instruments: Evaluating the ‘margins of error’ of group v. individual predictions of violence.’ British Journal of Psychiatry 190(S49), S60-S65. Access Here.

Heilbrun, K., Rogers, R., Otto, R.K. (2002).’ Forensic assessment: Current status and future directions.’ In Ogloff, J.R.P. (Ed.). Taking psychology and law into the twenty-first century. New York: Kluwer/Plenum, 119–146. Access Here.

Hubbard, D. J. and J. Pealer. (2009) ‘The Importance of Reponsivity Factors in Predicting Reductions in Antisocial attitudes and cognitive distortion among adult male offenders.’ The Prison Journal 89 (1), 79-98. Access Here.

Lloyd, M. (2019) Extremism Risk Assessment: A Directory. Lancaster, UK: Centre for Research and Evidence on Security Threats. Access Here.

Logan, C. (2016) ‘Structured professional judgment: applications to sexual offender risk assessment and management.’ In Phenix, A. and Hoberman, H. (eds.) Sexual Offenders: Diagnosis, Risk Assessment and Management. New York: Springer, 571588. Access Here.

RATED page updated: September 2019 © Risk Management Authority 2019

McIvor, G., Kemshall, H. and Levy, G. (2002) Serious Violent and Sexual Offenders: The Use of Risk Assessment Tools in Scotland. Edinburgh: Scottish Executive Social Research. Access Here.

Miccio-Fonseca, L. C. (2018, Spring). Sexually abusive youth who are transgender.

Perspectives: California Coalition on Sexual Offending (CCOSO) Quarterly Newsletter, 1, 14-17. Access Here.

Mossman, D. (1994) ‘Assessing predictions of violence :being accurate about accuracy.’ Journal of Consulting and Clinical Psychology 62(4), 783–792. Access Here.

Quinsey, V. L., Harris, G. T., Rice, M. E. and Cormier, C. A. (2006) Violent Offenders: Appraising and Managing Risk (2nd Edition). Washington, DC: American Psychological Association. Access Here.

Rettenberger, M., Matthes, A., Boer, D. P. and Eher, R. (2009) ‘Prospective Actuarial Risk Assessment: A Comparison of Five Risk Assessment Instruments in Different Sexual Offender Subtypes.’ International Journal of Offender Therapy and Comparative Criminology 54(2), 169-186. Access Here.

Rice. M. and Harris, A. (1995) ‘Psychopathy, schizophrenia, alcohol abuse and violent recidivism. ‘International Journal of Law and Psychiatry 18(3), 333-342. Access Here.

Risk Management Authority. (2015) Equality Duty Progress Report. Paisley: RMA. Access Here.

Risk Management Authority. (2016) Standards & Guidelines for Risk Management.

Paisley: RMA. Access Here.

Risk Management Authority. (2018a) Standards & Guidelines – Risk Assessment Report Writing. Paisley: RMA. Access Here.

Risk Management Authority. (2018b) Literature Review – a review of the risk posed by internet offenders. Paisley: RMA. Access Here.

RATED page updated: September 2019 © Risk Management Authority 2019

Rogers, R. (2000) ‘The uncritical acceptance of risk assessment in forensic practice.’

Law and Human Behavior 24(5), 595-605. Access Here.

Singh, J. P., Grann, M. and Fazel, S. (2011) ‘A comparative study of violence risk assessment tools: a systematic review and meta-regression analysis of 68 studies involving 25, 980 participants.’ Clinical Psychology Review 31(3), 499-513. Access Here.

Taxman, F. Thanner, S. M. and Weisburd, D. (2006) ‘Risk, Need and Responsivity (RNR): It All Depends.’ Crime & Delinquency 52(1), 28-51. Access Here.

Taxman, F. S. (2018) ‘Risk Assessment: where do we go from here?’ In J. P. Singh, D. G. Kroner, J. S. Wormith, S. L. Desmarais and Z. Hamilton (eds.) Handbook of Recidivism Risk/Needs Assessment Tools. John Wiley & Sons, Ltd: New Jersey, 271284. Access Here.

Viera, A. J. and Garrett, J. M. (2005) ‘Understanding Interobserver Agreement: The Kappa Statistic.’ Family Medicine 37(5), 360-363. Access Here.

RATED page updated: September 2019 © Risk Management Authority 2019