Appropriate Evaluation of Standardized Assessment Tools
Currently, human service assessment tools do not lend themselves to meaningful evaluation. We believe that the performance criteria of "validity" and "repeatability" do not function as intended in adult protection services (APS). They do not show how and to what extent resources are efficiently utilized with their use. On the other hand, these measures fit well as goals in developing assessment tools. For an assessment tool, validity and repeatability measure the degree by which this tool mimics the assessment of an adult protection social worker.
The recent study “Evaluation Report of Adult Protective Services Standardized Intake Decision Tool” commissioned by a state's department of human services (1), failed to achieve its primary evaluation objective. The report attributed the lack of data and sample size limitations as primary reasons. The availability of APS transactions data in the state’s databases (3) and county data collected by agencies (4) appear to contradict this conclusion.
The study described how the standardized tool cited was used to screen out "reports" from those accepted for further consideration in the APS process. This step gave an "initial screen-out" of 16,540 cases (41%) of the reported cases in the repository. Another assessment of those accepted cases resulted in an "override to screen-out" of 14,155 or 35% of the reported cases. The "final screen-in" came to 9,815 or 24% of reported cases. The study also suggested that 11% of the “final screened-in” was substantiated.
Proposed Evaluation Methodology. We believe that standardized intake or assessment tools are developed to assign social worker resources away from the "non-value-added" steps of the APS process. These workers will then be redirected to where direct contact with clients occurs. It follows that each FTE saved means increased client moments (5). We propose replacing the performance measures, Validity and Repeatability with Selectivity and Reach. Selectivity defines the ability of screening tools to select cases from reported maltreatment cases. Reach, its complement, indicates the extent to which substantiated cases in the population are reported and screened. It is calculated as the ratio of the number of substantiated maltreatment cases included in cases accepted with the theoretical amount of substantiated cases in the population (2). With the information from the study cited above, the two performance measures are calculated as:
Selectivity = Initial Screen-Out/Total Reports from Counties = 16,540 / 40,510 = 41%.
Reach = Final Substantiated Cases/Theoretical Substantiated Cases in Counties =1,080/6,832 = 16%.
Theoretical Substantiated Maltreatment Cases. The estimation of potential maltreatment cases substantiated within the APS process follows the procedure in a previous article (2). In this article, cases labeled "Accepted for Investigation and Services" for 2019 work as a proxy for the number of theoretical substantiated cases for all 87 counties. This choice is based on the availability of public information. The table below lists this dependent variable, the independent variables, and their respective sources.
Multiple regression analysis yielded below a model of theoretical substantiated cases in Minnesota counties. The improvement of the estimation methodology is another path to the future development of standardized screening tools.
Adult Protective Services. A report of an incident or occurrence of adult maltreatment may come from mandated or non-mandated sources. This report receives an initial eligibility examination and is kept in a federal repository. The county, as the lead investigating agency, takes a case from the repository, clarifies the report, applies screening criteria, and accepts the case for investigation and services. The investigative phase evaluates the level of risk by assessing the environment and interviewing the vulnerable adult, the perpetrator, and associated parties. Once substantiated, the case is assigned to collaborated agencies to mitigate the abusive condition.
The administrator seeks to manage limited county resources well so that all instances and occurrences of maltreatment are reported, investigated, and mitigated. Resource planning depends upon cycle time measurements, process capability, and forecast of service needs.
Ideas for Further Research. Our research that went into this article suggested to us ideas on how the APS or, for that matter, other social service standardized assessment tools can be enhanced. These potential research areas in this article assumed the clinical aspect of social services out of scope.
Theoretical Adult Maltreatment Cases. A statistical model estimated follows the methodology in a previous study (2). Variables for future work may be inferred from social work professionals, from the collection of primary data from service delivery, and from client feedback.
Public and Voluntary Reporting. Report receiving centers resembling customer service centers may be established to encourage the reporting of potential abuse or abusive situations. Community education on the healing benefits of APS services on individual welfare and personal relationships.
Standardized Screening Tool. There is always room for continuous improvement towards goals of selectivity and reach. The crucial benefits of quantitative measurement and scoring are now a timely consideration. Any development here will enable the migrations of standardized tools to artificially intelligent computerized assessments (7).
Investigative Process. Investigations rely on information surrounding abuse or abusive situations. Research on the estimation of maltreatment in the general population is needed. Its findings will find use in measurements of tool performance and in the delivery of APS.
References:
Evaluation Report of Adult Protective Services Standardized Intake Decision Tool. Adult protection / Minnesota Department of Human Services (mn.gov). 2021.
Predicting Vulnerable Adult Maltreatment Cases in Counties. Www.mgmtlaboratory.com, Jan 2022.
Minnesota Department of Human Services. https://mn.gov/dhs/assets/2017-2020-VAP-county-dashboard_tcm1053-488092.pdf
Association of Minnesota Counties. County Data (mncounties.org)
Client Moments: A Better Social Service Workload Measure. Www.mgmtlaboratory.com, Jun 2019.
Continuous Improvement and the Administrator. Www.mgmtlaboratory.com, Mar 2019.
Evaluating Human Services Performance across Counties: Multiple Regression Analysis (MRA) Demonstration. Www.mgmtlaboratory.com, Jul 2018.
AI Needn’t Be Complex. Www.mgmtlaboratory.com, Sep 2017.
By Staff. Www.mgmtlaboratory.com 2022
_________________________________________
Mgmtlaboratory.com staff and affiliated management consultants offer their experience in management methods and managerial tools to private and public organizations. Non-profit organizations and government entities may inquire at contact@mgmtlaboratory.com about free online consulting services. Www.mgmtlaboratory.com is operated by Service Administration Laboratory Corporation, a Minnesota non-profit organization.
Comments