Issuu

1 Background

RATED page updated: September 2019 Authority 2019

The Equalities Act (2010) has informed the latest edition of RATED. The Equalities Act maintains that the equalities duty of the public sector is to recognise differences based on social characteristics and to thereafter tailor the needs of individuals accordingly, to ensure the outcomes are equal for everyone in society. As part of its equalities duty, the RMA (2015) will “inform practitioners of relevant equalities considerations and help them apply appropriate risk assessment tools to specific populations.” Where the empirical research is available, the applicability of validated risk assessments to the relevant protective characteristics are considered: people with a mental impairment/learning disability (mental disability); people from an ethnic minority (ethnic group); females (gender); adolescents (age).

RATED is an ongoing project to be continuously updated and revised. Feedback and suggestions about new and emerging tools or new research studies related to the tools in the directory are welcomed.

This is the fourth edition of RATED, a directory intended to assist practitioners in selecting and applying risk assessment tools. Whether used as aids to initial decision making or case management in wider criminal justice settings, risk assessment instruments must be applied with an understanding of their respective strengths and weaknesses. In addition, to be used with any degree of confidence, risk instruments should have a sound empirical basis and validation history. The RMA’s ‘Framework for Risk Assessment Management and Evaluation’ (FRAME) highlights the importance of how this information is analysed, evaluated and then communicated in a meaningful way to inform decision making and action.

The report of the Committee on Serious Violent and Sexual Offenders, chaired by Lord MacLean, highlighted the need for an independent public body to ensure the effective assessment and minimisation of risk. The Committee reported to Scottish Ministers in 2000 and its work informed the Criminal Justice (Scotland) Act 2003. The legislation introduced a new sentence, the Order for Lifelong Restriction (OLR) and established the Risk Management Authority (RMA).

1.1 Background to RMA

RATED page updated: September 2019 © Risk Management Authority 2019

1.2 Risk assessment tools

Risk assessment instruments are intended to assist the practitioner in different ways by anchoring the assessment in empirical evidence to identify relevant risk factors. They may be used for a variety of purposes: aiding decision making, planning treatment, predicting the risk of recidivism, using in specialist settings or for case management in criminal justice settings. It is important that risk assessment instruments are applied with an understanding of their respective strengths and weaknesses. Fazel and Wolf (2017) highlight that external validation, performance in the population to be assessed and sound methodology are pivotal factors in deciding whether a tool should be chosen. In addition, to be used with any degree of confidence, risk instruments should have a sound empirical basis and validation history.

As part of its duties, the RMA is responsible for preparing and issuing guidelines on the assessment and minimisation of risk; as well as publishing standards by which measures taken in respect of the assessment and minimisation of risk are judged. In 2006, the RMA published the ‘Standards and Guidelines for Risk Assessment,’ focusing primarily on practice in relation to the Order for Lifelong Restriction (OLR). A year later, the ‘Standards and Guidelines for Risk Management’ was published.

The publication of RATED is intended to assist practitioners apply appropriate tools as part of a structured approach to risk assessment to assist with the identification of risk factors, needs and strengths of an individual. It should be noted that the application of a tool is one step in the risk assessment process; it is essential that the emerging information is analysed, evaluated and then communicated in a meaningful way to inform decision making and action.

The evolution of risk assessment has been well documented, with the general recognition that the incorporation of the ever growing research literature delivers an incremental improvement to each new generation of risk assessment instrument (Andrews, Bonta and Wormith, 2006). Actuarial instruments are described as nondiscretionary procedures, whereby the evaluator makes a decision based on fixed and explicit rules designed to predict the future (Hart and Logan, 2011). Actuarial instruments were designed specifically to assess risk or estimate the probability or likelihood of an outcome using ‘predictor variables’ (Hart, Douglas and Guy, 2016).

be highlighted, however, that concerns have been raised about an overreliance on the findings of risk instruments in proceedings that have a bearing on sentencing and/or restriction of liberty (Singh, Grann and Fazel, 2011). The implications from this are that all types of risk instruments share a common weakness if relied upon solely to evaluate ‘individual’ risk.

RATED page updated: September 2019 © Risk Management Authority 2019

A further suite of risk assessment instruments, commonly known as ‘structured professional judgement (SPJ) instruments, are distinct from actuarial tools in that they are guidelines designed to reflect the discipline with respect to scientific knowledge and practice (Hart and Logan, 2011).

It2016).must

The development of actuarial instruments has evolved over thirty years. Initially, instruments were based on static risk factors that may contribute to broad classifications based on longer term risk. These have since progressed to more sophisticated approaches that contain a range of dynamic factors and are designed to assist in an understanding of an individual and his/her behaviour, with the aim of reducing the likelihood of that behaviour through appropriate interventions (Hanson, 2007).

Whilst they incorporate the empirical evidence, they also include consideration of other clinical factors and do not lead to a quantified ‘score.’ In this sense, instruments can assist with the identification of risk factors and strengths of an individual (Baird 2017). There are a range of SPJ approaches, ranging from those requiring scenario planning and formulation to others that require an appraisal of the risk and protective factors of individual cases. Several instruments are primarily, although not exclusively, developed for application with mentally disordered individuals who offend; these are most commonly used by mental health professionals, psychologists and psychiatrists. Dynamic instruments allow for reassessments to be carried out to evaluate the effectiveness of treatment in order to guide future decisions. Each SPJ tool focuses on a particular offence type, such as sexual or adolescent violence, and, as such, these instruments are predominantly utilised in specialist settings for the purposes of detailed and individualised risk management planning (Abbiati et al., 2014; Lloyd, 2019; Logan

RATED page updated: September 2019 © Risk Management Authority 2019

Running in parallel with this development has been an on going academic debate concerning the applicability of group data to the individual and the relative superiority of actuarial and clinical approaches (Quinsey et al., 2006). A useful guide for practitioners is an approach that bridges both perspectives, drawing upon research work highlighting the limitations of the actuarial approach (Cooke and Michie, 2010; Hart, Michie and Cooke, 2007); while endorsing the counsel of others that abandonment of the actuarial tradition is unwise (Craig and Beech, 2010).

In line with the conclusions of researchers (Boer, 2006; Rettenberger et al., 2009), the RMA Standards and Guidelines encourage a ‘convergent’ approach to risk assessment which incorporates the use of both actuarial and SPJ tools. Furthermore, structured professional judgement and decision making are necessary components of all risk assessment practice, regardless of the type of instruments employed or the professional background of the practitioner. It is within this context that the RATED has been developed.

2.1 FRAME Approach

The framework promotes risk assessment practice that makes meaningful use of risk assessment tools without being over-reliant on them, by ensuring that the valuable contribution of such instruments is located within a structured approach to risk assessment which recognises the strengths and limitations of such tools and the importance of professional/clinical judgement. Risk assessment tools do not necessarily provide the comprehensive range of risk, responsivity and protective factors that inform an understanding of individual risk. Extrapolating from this, these tools should be used to inform rather than replace professional/clinical judgement.

The RMA is tasked with promoting effective risk assessment and management practice. In July 2011, the RMA, along with Scottish criminal justice agencies and the Scottish Government, published the FRAME policy paper. This established the policy direction in which a shared consistent framework to promote proportionate, purposeful and defensible risk practice could be achieved. FRAME proposes an approach that is grounded in the principles of risk practice to ensure all professionals are aware of the processes and skills associated with effective risk assessment and management. It also helps professionals understand the contribution that different kinds of instruments make to this practice. Moreover, it firmly promotes the practitioner as the ‘assessor’ who skilfully, knowledgably and within the parameters of his/her competencies, applies appropriate tools in a structured, meaningful process. Logan (2016) recommends that risk judgment can be substantially structured by utilising formulation, which organises the information into an explanation for why it happened and the circumstances in which it could possibly transpire again.

This is the fourth edition of RATED, a guide intended to assist practitioners in applying risk assessment tools as part of the FRAME approach to risk assessment to assist with the identification of risk factors and strengths of an individual. RATED also intends to keep researchers updated about the most recent developments.

RATED page updated: September 2019 © Risk Management Authority 2019

2.2 Purpose of RATED

RATED AIMS AND OBJECTIVES

• Youth Violence: instruments for measuring risk of violence in adolescents.

of the current publication are as follows:

•To provide a guide for assessors to consider when applying a tool as part of a holistic risk assessment process.

RATED page updated: September 2019 © Risk Management Authority 2019

•To provide relevant research information on each assessment tool included in the directory, which includes (but is not limited to) validation evidence.

• Responsivity Issues: this category consists of instruments for individuals with personality disorders or learning disabilities.

• General Risk Assessment: those tools that can be used for general application for risk.

• Violence Risk: instruments to establish the risk of violence.

• Intimate Partner Violence and Stalking: instruments that relate to intimate partner violence and/or stalking.

• Youth Sexual Violence: tools designed for measuring sexual violence risk in adolescents.

• Sexual Offending: tools relating to sexual violence risk or internet offending.

• Youth General Assessment: similar to the General Risk Assessment category, but for use with adolescents.

•To provide an impartial and factual account of the strengths and limitations of each instrument.

Risk assessment instruments may focus on specific types of risk developed and/or particular groups. This directory is structured by type of tools. This introductory section provides further background to the purpose of RATED and the criterion used to rate tools. The categories of tools are the following:

The publication of RATED aims to provide a summary of the empirical evidence to inform a balanced and individualised approach to assessment, as well as to contribute to effective and ethical practice (Andrews and Bonta, 2010). The RATED directory is intended as a resource for Accredited Risk Assessors undertaking assessments for the High Court in Scotland when the court is considering imposing an OLR, in addition to other practitioners involved in risk assessment and risk Themanagement.objectives

 Ethnicity  Age  Mental disability  Religion  Gender  Gender reassignment  Sexual orientation

The ‘risk need responsivity principles,’ suggest that effective interventions are proportionate to the likelihood of re offending, address the needs related to offending, and are responsive to a range of issues that may influence a person’s ability to respond to interventions (Andrews and Bonta, 2010; Taxman et al., 2006).

2.3 Responsivity Considerations

Hubbard and Pealer (2009) found that the more ‘issues’ an individual has, the less likely a programme will be successful in achieving what it aims to do. To that end, strategies that are informed by responsivity issues are more likely to be effective and reduce recidivism (Beaver and Schwartz, 2016).

Whilst every effort has been made to ensure the accuracy of the information presented in this publication, given the evolving nature of research relating to risk assessment and risk management, the evidence base is subject to change.

RATED page updated: September 2019 © Risk Management Authority 2019

Practitioners are, therefore, encouraged to keep abreast of the emerging evidence when administering the assessment tools. Additionally, practitioners are invited to contribute to the development of the evidence base and subsequent editions of this publication by providing feedback on the contribution of specific instruments to their practice by emailing RATED@rma.gov.scot.

The RMA’ s (2018a) revised Standards and Guidelines highlights the importance to practitioners of ensuring that risk assessment and management practices do not disadvantage those with ‘protected characteristics.’ This is entrenched within the Equalities Act (2010), which attributes ‘equalities duty’ upon public sector organisations. The purpose of this is to recognise differences based on a number of protected characteristics and tailor services accordingly to ensure that people are not disadvantaged on the basis of those characteristics. The protected characteristics are:



1 The other protected characteristics within the UK Equalities Act (2010) are: religion, sexual orientation, marriage/civil partnership, pregnancy, gender reassignment (both biological changes or social functioning ones) and physical disabilities.



As part of its equalities duty, the RMA (2015) seeks to “inform practitioners of relevant equalities considerations and help them apply appropriate risk assessment tools to specific populations.” Since RATED is a means by which the RMA advances this objective, the equality and diversity guidance laid out in the Standards and Guidelines (RMA, 2018a) informs the content of RATED. An Equality Impact Assessment (EQIA) was also carried out to consider the role of protected characteristics in the publication of RATED.

Physical disabilities

When compiling a new edition of RATED, the available published literature relating to instruments was reviewed, including studies that have examined their applicability to specific groups. Where the empirical information is available, the applicability of risk assessments to the relevant protected characteristics is considered. Of the ten protected characteristics, a number have limited if any research evidence. Those that have attracted most attention in the literature are: people with a mental impairment/learning disability (mental disability) people from an ethnic minority (ethnic group) females (gender) adolescents (age)



RATED page updated: September 2019 © Risk Management Authority 2019 

The research evidence in relation to the remainder of protected characteristics1 is lacking, with only a handful of studies available. For instance, transgender issues are explored in relation to the youth sexual violence tool MEGA♪ (Miccio Fonseca, 2018); whilst a study by Bhutta and Wormith (2016) incorporated a measure of religiosity and spirituality into the LS/CMI to test its effectiveness in a religious country.

Marriage/civil partnership

Pregnancy



Regrouping of reference lists:

A number of significant changes have been made to the fourth version of the directory. These include a range of amendments to the format, layout and content of the directory, a summary of which is provided below.

•Learning Disabilities and Diagnostic/Personality Assessments were previously in separate categories. These four categories have now been amalgamated into a new category entitled ‘Responsivity Issues,’ documenting those tools specific to disorders that could influence offending.

The existing research base is limited to the extent that many tools do not have published studies pertaining to protected characteristics. Moreover, when protected characteristics are reviewed in relation to risk assessment tools, the focus tends to be broad, e.g. mental ability broadly is reviewed; yet specific mental disorders receive less attention. If further research emerges then future editions of RATED may create new categories for these protected characteristics.

2.4 Developments in RATED Fourth Edition

In the application of tools, practitioners should be aware of the available research, or lack thereof, in relation to applicability to persons with a protected characteristic.

Rearrangement of the directory entries:

RATED page updated: September 2019 © Risk Management Authority 2019

•The Stalking category has been combined with the ‘Domestic Violence’ category, since these types of offences can often be interlinked (see Kerr, 2018). Additionally, the term ‘Domestic Violence’ has been replaced with the more commonly used term within the field ‘Intimate Partner Violence’ to give the new category of ‘Intimate Partner Violence and Stalking.’

•‘Internet Offending’ (offences related to indecent images of children) has been grouped with ‘Sexual Violence Risk’ and the category has been renamed as ‘Sexual Offending.’

Reference lists are included as a separate document for each of the eight sections of tools. To facilitate accessibility for users, references within the lists are grouped by tools.

Structured Assessment of Positive Factors for Violence Risk: Youth Version (SAPROF:YV) and Short Term Assessment of Risk and Treatability: Adolescent Version (START:AV) in the Youth Vioelnce Assessment section

RATED page updated: September 2019 © Risk Management Authority 2019



Assessment, Intervention and Moving On Project Version 3 (AIM3), The AIM Project Assessment Models for Children under the age of 12 years old (AIM Under 12s), Protective and Risk Observations for Eliminating Sexual Offence Recidivism (PROFESOR) and Technology Assisted Harmful Sexual Behaviour: Practice (TA HSB Guidance), all in Youth Sexual Violence Risk.



Brief Spousal Assault Form for the Evaluation of RISK (B-SAFER) in the Intimate Partner Violence and Stalking section.

Oxford Mental Illness and Violence Tool (OxMIV) in the Responsivity Issues section.



Upon reviewing the literature for the current risk assessment tools, a number of new tools were identified. Following a review of their usefulness and applicability, the decision was made to include the majority of them in this revised edition. The additional entries within the directory reflect both the extent of the review for this edition and the speed of development in the field of risk assessment. The new additions to this edition of RATED are:

Comprehensive Assessment of Psychopathic Personality (CAPP) in the Responsivity Issues section.

Risk for Sexual Violence Protocol (RSVP) in the Sexual Offending section.



Terrorist Radicalization Assessment Protocol 18 (TRAP 18) and Workplace Assessment of Violence Risk (WAVR-21) in the Violence Risk section.



Oxford Risk of Recidivism (OxRec) tool in the General Risk Section.

Barr 2002R, Child Pornography Offender Risk Tool (CPORT), Sexual Offender Treatment Intervention and Progress Scale (SOTIPS) and Vermont Assessment of Sex Offender Risk 2 (VASOR 2) in the Sexual Offending section.

AssetPlus in Youth General Risk Assessment Section.

A number of tools have been upgraded from awaiting validation status in the third edition of RATED (RMA, 2015) to validated in this edition:

Inclusion of new tools:

•For three bars, the word ‘little’ has been replaced with ‘preliminary.’

Rewording of rating criteria:

To improve the accessibility to research, where available hyperlinks have been included to the studies cited in tool entries. The hyperlink will generally take the user to the journal’s page, which contains the article’s abstract. Users may then check whether they have access to the full article through their institution or they may have the option to purchase it on the journal’s landing page.

•For nine bars, the word ‘high’ has been replaced with ‘sufficient.’

RATED page updated: September 2019 © Risk Management Authority 2019 

The rating criteria refers to the amount of evidence available for a particular element (e.g. inter-rater reliability). The terminology has been changed in this version of RATED:

•For zero bars, the wording of ‘no or poor’ has been changed to ‘insufficient.’



•For six bars, the term ‘some’ has been substituted with ‘intermediate.’

Hyperlinks to research:

Hyperlinks are available within individual tool entries and in the list of references for each of the eight sections.

The Youth Assessment and Screening Instrument (YASI) in the Youth General Risk section.

The Multiplex Guided Inventory of Ecological Aggregates for Assessing Sexually Abusive Adolescent and Children (MEGA) in the Youth Sexual Violence section.

In cases where the full article is already freely available on the internet (for instance, through ResearchGate) the hyperlink to this is included to facilitate users’ access to research. In a small number of cases, the research cited is not available online and, hence, no hyperlink is included.

4 Sensitivity refers to the extent to which an instrument can correctly classify when an individual possesses a particular characteristic. Sensitivity is inversely proportional with specificity, whereby one increases as the other decreases.

2.5 Using RATED

Where they have the necessary competencies and training, practitioners can use with confidence validated tools that possess a robust validation history and empirical grounding. These tend to be tools that have also evidenced sufficient inter rater reliability2, specificity3, sensitivity4 and predictive accuracy5 in identifying individuals at risk of re offending. Practitioners should use tools with caution that possess many of the essential attributes and have the potential to demonstrate all of these in the future (with further studies and/or evidence). Reports detailing findings based on the use of tools that are awaiting validation should outline the limitations of their use in this respect.

5 The predictive accuracy of a tool refers to its ability to distinguish between certain populations, such as those who reoffend and those who do not.

•Evaluating the validation literature available;

Using RATED to select a risk assessment tool

•The performance of tools with respect to the criteria outlined which includes validation history, empirical grounding, inter rater reliability and ability to identify targets for intervention.

RATED page updated: September 2019 © Risk Management Authority 2019

•Providing information on the evidence base for the tool design;

The use of suitable risk assessment tools is an important concern for practitioners and agencies involved in the management of offenders. RATED is designed to inform and support decisions regarding which tools to use by:

•The type of risk that they wish to evaluate (general risk of reoffending, risk of violence, sexual offences, intimate partner violence or stalking);

•Setting out the appraisal against the assessment framework.

•The age, gender, ethnicity, mental state and cognitive abilities of the individual;

When using RATED, practitioners should consider the following:

2 Inter rater reliability refers to the degree in which two or more assessors consistently rate items within a tool.

•Describing the purpose and design;

3 Specificity relates to the ability of a test to classify when an individual does not possess a particular characteristic.

Practitioners are advised to review the range of guidance documents dealing with risk assessment. These include the RMA Standards and Guidelines, FRAME and the RMA accredited manner for risk assessment for Risk Assessment Orders (RAO) or Interim Compulsion Orders (ICO) when the High Court is considering an

RATED page updated: September 2019 © Risk Management Authority 2019

In cases involving the 17 21 year old age group, practitioners should use their judgement as to whether to apply instruments designed for young people, or those designed for an adult population. To inform this judgement, specific consideration should be given to the extent to which the tool incorporates evidence of its validity with young people and the impact of child developmental processes of the young person being assessed. Similarly, one should give careful consideration as to whether the selection of a given instrument is appropriate for certain minority groups, e.g. those with mental health problems, cognitive impairments and learning disabilities. A good guide to an instrument’s suitability for a particular group is whether a sample from the target group has been included in the original research sample or been the subject of a successful pilot study.

Practitioners should use tools with extreme caution that possess few of the essential attributes, but appear to have the potential to meet all of these in the future (with further studies and/or evidence). Reports detailing the findings based on the use of such tools must include a justification for selecting them and should outline the limitations of their use.

In all cases, practitioners should have the necessary training for using the tool(s) of their choice; be aware of their limitations and the caveats of their use; be in a position to discuss these and to evidence their assessment. Importantly, these limitations should be clearly outlined in the assessor’s report to inform future decision making. Above all, risk assessment tools should be used as part of a risk assessment process to inform rather than replace professional judgement.

TheOLR.Standards and Guidelines states that practitioners should select risk assessment instruments appropriate to the risk needs and characteristics of the individual being assessed, as well as in accordance with their own competencies, experience and training. Practitioners should also be aware of the validity and

•Seek the views of other experts on the approach to be taken in the case;

The directory also highlights that there may be cases (perhaps with unusual types of offending) where available instruments have little or no validity, wherein the following activities are recommended:

•Use the structured professional judgement framework, as set out in this manner, to guide the process of case formulation and reaching a risk judgement;

•Be explicit in the report about the limitations of the approach used.

•Seek supervision from a sufficiently experienced and expert colleague;

limitations of any tool that they use (RMA, 2018a). RATED sets out the RMA’ s evaluation of the evidence available for each instrument and highlights any limitations or caveats that professionals should take into account when using a tool as part of a holistic risk assessment process.

•Gather as much information as possible to support the formal risk assessment;

RATED page updated: September 2019 © Risk Management Authority 2019

•Review the relevant publications to look at what the latest academic and clinical literature describes in relation to the type of offender and the approach to the risk under consideration;

•Validation History

3.1 The evaluation process

The adopted framework comprises a number of criteria considered essential for the evaluation of risk assessment tools, building upon earlier work by McIvor, Kemshall and Levy (2002) and the findings of other researchers (Heilbrun, Rogers and Otto, 2002; Rogers 2000). Criteria considered as essential are:

The following criteria were used to select research studies for inclusion in the directory:

1 In this document, the word ‘evaluation’ is used to refer to the assessment of the tools’ performance with respect to the criteria framework outlined.

•Empirical Grounding

The RMA has developed a process for evaluating1 risk assessment tools and this has been followed for each tool presented in this document. This evaluation process will also be applied in future to assess newly developed tools. The RMA researcher working on this update to RATED used a standard pro-forma to describe each tool based on the summarised evidence and the adopted evaluation framework. This provides individual assessments of the tool against each of the framework criteria and comments to support them.

Adopting an evaluation framework

The RMA welcomes applications for inclusion in the directory, as well as any comments its users.

•Inter Rater Reliability

Compilation of evidence and evaluation

Evidence collated for the current audit, comprised of publications (including ‘grey literature,’ such as doctoral theses and dissertations for Master’s Degree and other unpublished material) and evidence of ongoing research.

3 EVALUATION PROCESS

RATED page updated: September 2019 © Risk Management Authority 2019

•Size of study population, with greater emphasis on evidence drawn from large sample populations;

•Date of publication, with greater weight given to more recent publications;

•Relevance to UK population; i.e. whether the study had employed a sample of individuals from the nations within the UK;

•The type of risk addressed, i.e. the risk of general, sexual and/or violent offending, or any combination of these;

•An outline of its validation history, considering studies carried out in the United Kingdom and internationally; this includes details of the study focus and setting (community, secure unit, prison or mental health institution), the size of the sample and type of population used;

•The publisher(s)/developer(s) and the year of publication (where known);

•Focus of study, taking into account whether specific issues have been considered, such as predictive validity, inter rater reliability, significance/ usefulness,2 sensitivity and so on.

•A brief description of the risk assessment tool;

RATED page updated: September 2019 © Risk Management Authority 2019

•Target population, i.e. male/female offenders, forensic mental clients, and children and young offenders;

•Sponsorship, i.e. whether the study has been commissioned and/or funded by a government or statutory agency, with greater weight given to independent studies;

•Evidence of inter rater reliability, significance/usefulness;

•Strengths, practice considerations, and future research directions.

•Authorship, i.e. whether the author(s) were, or were not, involved in the validation of the tool under consideration, with greater weight given to studies conducted by independent researchers rather than the author(s) of the risk assessment tool;

2 Significance/Usefulness of a tool refers to its evidential basis, predictive power and clinical utility in enabling the practitioner to undertake risk formulation and risk management planning.

•Place of publication, with greater weight given to publications in peer-reviewed journals;

•The title and any acronym by which it is known;

The following information was provided for each tool listed within the directory:

The following criteria have been used to identify the strengths and weaknesses of each risk assessment tool listed within the current directory. It is important to note that these criteria have not been applied to risk assessment tools awaiting further validation. This is due to the limited research available relating to the psychometric properties of these risk assessment instruments.

Tools are divided up into ‘Validated’ and ‘Awaiting Validation.’ Validated tools are those which have sufficient research studies evidencing their suitability to be used within the field. Due to the scope of research available on the, validated tools have more extensive entries than those that are ‘awaiting validation.’ A further distinction between these categories is that to rate a tool in as validated, the RMA considers the existence and quality of the validation evaluation studies, assessed on the basis of the availability of two or more independent papers written by different authors and published in peer reviewed journals.

3.2.1 Empirical Grounding

3.2.2 Inter-Rater Reliability

3.2 Criteria definitions and performance measurement scales

The RMA liaised directly with tool authors and publishers wherever possible to compile this material accurately.

RATED page updated: September 2019 © Risk Management Authority 2019

Inter-rater reliability refers to the degree to which two or more assessors are consistent in their ratings when using the same risk assessment tool. There are two types of statistics used to estimate this: the Intra Class Correlation (ICC) and Kappa

Empirical grounding examines the scientific and theoretical underpinnings of the risk assessment tools presented in this document. For example, a risk assessment tool based on sound theoretical evidence and / or other extensive scientific findings observed in prior research would be considered to have a high level of empirical grounding. Higher levels of empirical grounding may, therefore, increase the utility of the instrument in assessing the risk posed by an individual.

Tools that are awaiting validation currently do not have the sufficient research evidence to be considered validated; although these have the potential to become validated in future.

The validation history criterion is split into four subsections which are as follows:

d.Applicability: Mental Disabilities. This item refers to the validity of the risk assessment tool for those with mental disorders and/or learning disabilities.

b.Applicability: Female Offenders This item refers to the validity of the risk assessment tool for women and girls.

3.2.3 Validation History

To rate a tool in this area, the RMA considers the existence and quality of the validation evaluation studies, assessed on the basis of the availability of two or more independent papers written by different authors and published in peer reviewed journals. The papers are required to have examined the predictive validity of the tool and/or its practical usefulness for the assessment and management of risk of harm to others. This approach accommodates concerns that have been raised in the literature that different research designs may be appropriate for identifying the properties, strengths and limitations of various types of instruments.

c.Applicability: Ethnic Minorities This item refers to the validity of the risk assessment tool for ethnic minority populations.

(K)Coefficients (see section 5 for further details). Researchers must be able to demonstrate that the risk assessment instruments are reliable; without reliability results, the risk assessment’s performance cannot be replicated and validation cannot be attained. Even if a tool had predictive validity, its usefulness will be undermined if it is applied inconsistently by different practitioners (Taxman, 2018). It is thus of utmost importance for a risk assessment tool to have high inter rater reliability whereby assessors using the same tools score the items similarly.

a.General Predictive Validity This item refers to the capability of the risk assessment tool to discern between the recidivist and non recidivist populations. The recognised statistical measure for this is the receiver operating characteristic curve (probability that a recidivist would receive a higher rating than a non recidivist) (Taxman, 2018).

RATED page updated: September 2019 © Risk Management Authority 2019

Tools are rated on these criteria using the following performance scale:

evidence of sufficient empirical grounding, inter rater reliability or validation.

3.2.4. Contribution to Risk Practice

0 Bars No evidence of or evidence indicating insufficient empirical grounding, inter-rater reliability or validation.

evidence of empirical grounding, inter rater reliability or 9validation.BarsSufficient

For tools that are ‘awaiting validation,’ this section is entitled ‘Tool Development’ and covers findings of studies in relation to predictive accuracy. ‘Tool Development’ will also cover the empirical grounding of a tool and studies relating to inter rater reliability where these are available.

This section in tool entries supports the tiered approach to risk practice promoted within the RMA’s Framework for Risk Assessment, Management and Evaluation (FRAME), in particular the first three standards relating to: (1) Risk Assessment; (2) Planning and Responding to Change; (3) Risk Management Measures.

3 Bars Preliminary evidence of empirical grounding, inter rater reliability or 6validation.BarsIntermediate

This final section documents other elements to be considered before using the tool, e.g. whether it has a clinical over ride feature to allow the risk rating to be readjusted at the assessor’s discretion. Also noted in this section are any other research findings about the tool that do not relate to its inter-rater reliability or predictive accuracy. In tools that are ‘awaiting validation,’ this section is entitled ‘General Notes’ and covers similar areas.

RATED page updated: September 2019 © Risk Management Authority 2019

3.2.5 Other Considerations

This section does not incorporate the ratings scales observed in the previous criterion; rather, it provides a qualitative evaluation of the properties of the instrument and its contribution to risk practice.

The Brier score is defined as the average quadratic difference between the predicted probability and the binary outcome. Its purpose is to measure the ‘calibration’ of a set of predictions.probabilistic

ICC values range from 0 to 1 and are typically reported with two decimal points, e.g. .75. Cicchetti followingrecommends(1994)thethresholds: <.40 = ‘poor’ .40 to .75 = ‘moderate’



Brier Scores

An ICC correlationestimationrepresentsscoretheofthebetween two scores. It measures the magnitude of agreement for inter rater reliability.



STATISTICAL TERMINOLOGY

Intra Class Correlation Coefficient poor/moderate/excellentdescriptors;(ICC)

The Brier score ranges from 0 to 1. The best possible score is 0, indicating total accuracy. The lowest possible score is 1, meaning that the forecast was inadequate.wholly

Statistical Analyses

Authors of reports and studies on risk assessment tools adopt a range of descriptions of statistical outcomes. For the sake of consistency in the tables below, the RMA has adopted the following descriptors drawing from the most prevalently used terminology in the literature.

Definition Interpretation

There is a useful explanation that uses an analogy of the bull’s eye on a dartboard: actually hitting the bull’s-eye represents accuracy; landing shots together indicates good reliability. Considering this, hitting the bull’s eye and landing all the shots together would convey both accuracy and precision (Viera and Garrett, 2005).



Kappa (.) poor/average/excellentdescriptors;Coefficient

–

 <

Effect sizes

The effect size quantifies the size of the difference between two groups. It is calculated as the standardised mean difference between the two groups: mean of Group A minus mean of Group B; the total of which is divided



The Kappa Coefficient (.) measures the agreement between two individuals.

Effect sizes can be interpreted in terms of the percentiles or ranks at which two Interpretationsoverlap.distributions of effect sizes are dependent on the assumptions that the two groups are



A Kappa is always less than or equal to a value of ‘1.’ A value of ‘1’ implies perfect agreement and a value of ‘ 1’ implies thresholdsRecommendeddisagreement.perfectare: 0 Less than agreementchance 0.01 0.20 poor agreement 0.21 agreementaverage0.60 0.61 agreementexcellent0.99

association between a predictor variable and the outcome.

Sensitivity and Specificity

The higher the value of sensitivity, the greater the ability of the measure being

Sensitivity is the ability of a test to correctly classify an individual as possessing a particular

normally distributed and have the same standard deviations.

The values of r can range from ‘-1’ to ‘1,’ with ‘0’ indicating that there is no .40.25.10follows:be(1988)Accordingrecidivism.decreasedassociatedscoresindicatenegativereincreasedassociatedhighindicatePositiveandpredictorbetweenrelationshipthevariabletheoutcome.valuesthatthescoresarewithcidivism;whereasvaluesthathigharewithtoCohen,rvaluesmayinterpretedasaresmall,aremoderate,arelarge.

by the deviation.standard

Pearson small/moderate/largeCoefficientCorrelationdescriptors;

Pearson coefficientcorrelationmeasures the

In the context of risk assessment, the ROC curve is a plot that shows the probability

tested to correctly identify individuals.

Z+ small/moderate/largedescriptors; Measures associationthebetween the predictor variable and the outcome. These two groups usually comprise of the (1) recidivists and (2)the non recidivists, separated by the difference in their scores obtained on assessments.risk

Several ROCcalculatedindicatorsdifferentcanbefromcurves.The

Receiver CharacteristicOperating(ROC) Curve low/moderate/highdescriptors;

For instance, a sensitivity of 62% on a risk assessment tool indicates that it has the ability to correctly classify just under two thirds of individuals who will reoffend. For specificity, higher values indicate the ability of a measure to correctly identify who will not possess characteristicscertain (e.g. who will not go on to reoffend).

According to Cohen (1988), d values may be interpreted as .20follows:isconsidered ‘small’; .50 is considered ‘moderate’; .80 is considered ‘large’.

characteristic (e.g. offending). Specificity is the ability of a test to classify an individual as not possessing a particular characteristic (e.g. not decreases.increases,asspecificity,proportionalSensitivityoffending).isinverselywithmeaningthatthesensitivitythespecificity

Area Under the Curve (AUC)

most commonly used indicator is the ‘area under the curve’ (AUC) (see below).

low/moderate/highdescriptors;

AUC values can range from ‘0’ to ‘1’. They can interpretedbeas the probability that a randomly selected recidivist has a worse score than a randomly selected Valuesnon-recidivist.between ‘.51’ and ‘1.0’ indicate recidivism;associationspositivewithwhilst

values between ‘0’ and ‘.50’ indicate that predictions are no better than chance.

The area under the curve (AUC) for the ROC curve is a useful summary statistic for the extent to which a measure discriminates between recidivists and non recidivists.

that a measure will correctly identify persons as recidivists or non recidivists (Mossman, 1994; Rice and Harris, 1995). It is a plot of the ‘hits’ (the proportion of recidivists correctly identified as recidivists) against the ‘false alarms’ (the proportion of identifiednon-recidivistsasrecidivists).

Using Cohen’s (1988) d values as a guide, AUC values may be interpreted as .556follows:isconsidered ‘low’; .639 is considered ‘moderate’; .714 is considered ‘high.’

Cohen, J (1988) Statistical Power Analysis for the Behavioral Sciences. New York: Routledge. Access Here.

Andrews, D., Bonta, J. and Wormith, S. (2006) ‘The Recent Past and Near Future of Risk and/or Need Assessment.’ Crime & Delinquency 52(1), 7 27. Access Here

Reference List

Bhutta, M. H. and Wormith, J. S. (2016) ‘An examination of a risk/needs assessment instrument and its relation to religiosity and recidivism among probationers in a Muslim culture.’ Criminal Justice and Behavior 43(2), 204 229. Access Here.

Boer, D. P. (2006) ‘Sexual offender risk assessment strategies: is there a convergence of opinion yet?’ Sexual Offender Treatment 1(2), 1 4. Access Here

Cooke, D. J. and Michie, C. (2010) ‘Limitations of diagnostic precision and predictive utility in the individual case: A challenge for forensic practice.’ Law and Human Behavior 34(4), 259 274. Access Here.

Abbiati, M., Laix, J., Gasser, J. and Malin, V. (2014) ‘Predicting physically violent misconduct in prison: a comparison of four risk assessment instruments.’ Behavioral Science and the Law 37(1), 61 77. Access Here

Andrews, D. A. and Bonta, J. (2010) The Psychology of Criminal Conduct (5th edition). Cincinnati, Ohio: Anderson Publishing. Access Here.

Craig, L. A. and Beech, A. R. (2010) ‘Towards a best practice in conducting actuarial risk assessments with adult sexual offenders.’ Aggression and Violet Behavior 15(4), 278 293. Access Here.

Baird, C. (2017). A question of evidence part two. Summary and conclusion. Madison, WI: National Council on Crime and Delinquency. Access Here.

Fazel S. and Wolf, A. (2018) ‘Selecting a risk assessment tool to use in practice: a 10 point guide.’ Evidence Based Mental Health 21(2),41 43. Access Here.

Hart, S. D., Douglas, K. S. and Guy, L. S. (2016) ‘The Structured Professional Judgment Approach to Violence Risk Assessment: Origins, Nature and Advances.’ In L. Craig and M. Rettenberger, M. (eds.) The Wiley Handbook on the Theories, Assessment and Treatment of Sexual Offending. Hoboken, New Jersey: John Wiley and Sons, 643 666. Access Here

Hubbard, D. J. and J. Pealer. (2009) ‘The Importance of Reponsivity Factors in Predicting Reductions in Antisocial attitudes and cognitive distortion among adult male offenders.’ The Prison Journal 89 (1), 79 98. Access Here

Hanson, R. K. (2007, March) ‘How should risk assessments for sexual offenders be conducted?’ Paper presented at the Fourth Annual Forensic Psychiatry Conference, Victoria, British Columbia, Canada. [Not accessible]

Lloyd, M. (2019) Extremism Risk Assessment: A Directory. Lancaster, UK: Centre for Research and Evidence on Security Threats. Access Here

Heilbrun, K., Rogers, R., Otto, R.K. (2002).’ Forensic assessment: Current status and future directions.’ In Ogloff, J.R.P. (Ed.). Taking psychology and law into the twenty first century. New York: Kluwer/Plenum, 119 146. Access Here.

Hart, S., Michie, C. and Cooke, D. (2007) ‘Precision of actuarial risk assessment instruments: Evaluating the ‘margins of error’ of group v. individual predictions of violence.’ British Journal of Psychiatry 190(S49), S60 S65. Access Here.

Hart, S. D. and Logan, C. (2011) ‘Formulation of violent risk using evidence based assessment: the structured professional judgment approach.’ In D. Sturmey and M. McMurran (eds.) Forensic case formulation. Chichester, UK: Wiley Blackwell, 83 106. Access Here.

Logan, C. (2016) ‘Structured professional judgment: applications to sexual offender risk assessment and management.’ In Phenix, A. and Hoberman, H. (eds.) Sexual Offenders: Diagnosis, Risk Assessment and Management. New York: Springer, 571 588. Access Here

McIvor, G., Kemshall, H. and Levy, G. (2002) Serious Violent and Sexual Offenders: The Use of Risk Assessment Tools in Scotland. Edinburgh: Scottish Executive Social Research. Access Here.

Risk Management Authority. (2018b) Literature Review a review of the risk posed by internet offenders. Paisley: RMA. Access Here.

Risk Management Authority. (2016) Standards & Guidelines for Risk Management. Paisley: RMA. Access Here

Risk Management Authority. (2018a) Standards & Guidelines Risk Assessment Report Writing. Paisley: RMA. Access Here.

Risk Management Authority. (2015) Equality Duty Progress Report. Paisley: RMA. Access Here

Miccio Fonseca, L. C. (2018, Spring). Sexually abusive youth who are transgender. Perspectives: California Coalition on Sexual Offending (CCOSO) Quarterly Newsletter, 1, 14 17. Access Here.

V. L., Harris, G. T., Rice, M. E. and Cormier, C. A. (2006) Violent Offenders: Appraising and Managing Risk (2nd Edition). Washington, DC: American Psychological Association. Access Here.

Rice. M. and Harris, A. (1995) ‘Psychopathy, schizophrenia, alcohol abuse and violent recidivism. ‘International Journal of Law and Psychiatry 18(3), 333 342. Access Here.

Mossman, D. (1994) ‘Assessing predictions of violence :being accurate about accuracy.’ Journal of Consulting and Clinical Psychology 62(4), 783 792. Access Quinsey,Here.

Rettenberger, M., Matthes, A., Boer, D. P. and Eher, R. (2009) ‘Prospective Actuarial Risk Assessment: A Comparison of Five Risk Assessment Instruments in Different Sexual Offender Subtypes.’ International Journal of Offender Therapy and Comparative Criminology 54(2), 169 186. Access Here

Rogers, R. (2000) ‘The uncritical acceptance of risk assessment in forensic practice.’ Law and Human Behavior 24(5), 595 605. Access Here.

Taxman, F. Thanner, S. M. and Weisburd, D. (2006) ‘Risk, Need and Responsivity (RNR): It All Depends.’ Crime & Delinquency 52(1), 28 51. Access Here.

Taxman, F. S (2018) ‘Risk Assessment: where do we go from here?’ In J. P. Singh, D. G.Kroner, J. S. Wormith, S. L. Desmarais and Z. Hamilton (eds.) Handbook of Recidivism Risk/Needs Assessment Tools. John Wiley & Sons, Ltd: New Jersey, 271 284. Access Here.

Singh, J. P., Grann, M. and Fazel, S. (2011) ‘A comparative study of violence risk assessment tools: a systematic review and meta regression analysis of 68 studies involving 25, 980 participants.’ Clinical Psychology Review 31(3), 499 513. Access Here.

Viera, A. J. and Garrett, J. M. (2005) ‘Understanding Interobserver Agreement: The Kappa Statistic.’ Family Medicine 37(5), 360 363. Access Here.