Testing for Racial Bias in Standardized Screening Tool Results

This article, the third in a series (2) (3), relates our findings from information and data in a recently published “Evaluation Report of Adult Protective Services Standardized Intake Decision Tool (1).” The study observed, “. . . a significant correlation between the race of a person (specifically whether the person is Caucasian or Black/African American) and rate of screen-out by the override process).” While we confirmed the significance of the difference between override rates, our study also showed lower screen-out for Black/African American cases by the SDM, the standardized decision tool used to select vulnerable adult maltreatment cases from those reported.

The study cited (1) featured the processes by which incidences of vulnerable adult maltreatment or maltreatment situations reported in Minnesota Counties flow through the adult protection system with the SDM tool and the override assessment as a single screening process. Our study separated these processes. Of the 40,510 reported cases flowing through the SDM tool, 23,970 cases or 59% were screened-in and 16,540 cases or 41% were screened out, calculating to a selectivity rating of 41%. We assumed that any case flowing through the override process was assessed by an “expert” worker. For the override process, the total screen-out combines 16,540 cases and 14,155 cases for a much higher selectivity rating of 76%. The comparison of selectivities indicated that the SDM tool requires an extensive revamp to increase its selectivity rating and be a meaningful part of the APS.

Our study traced the reported cases through the SDM screen and the APS worker override. The SDM screened-in 80.3% of reported cases in Hennepin and Ramsey counties. Those screened in from reported cases in other Minnesota counties were at 46.9%. The analysis of screening by the SDM tool between two racial case categories exhibited these findings: 58.0% of Caucasian cases screened-in from those reported compared to 73.9% with which Black/African American cases were screened-in.

The opposite resulted in the APS worker override. From the cases screened in by the SDM, the override process let a Final screen-in of 18.0% from Hennepin and Ramsey counties which compares to the Final screen-in of 63.7% from all other Minnesota counties. The reversal of case flows through the screening processes was also observed for the two racial categories.

The SDM screened in fewer Caucasian cases at 58.0% than Black/African American cases that were screened in at 73.9%. The Final screened-in rates through the APS override were opposite at 45.4% and 20.1%, respectively. The effect of interaction between the two screening processes in series may have a role in these results.

Our statistical analyses highlight the dissimilarity of the flow of cases from Reports to Final screen-in. The distribution of cases screened in and screened out in larger counties, i.e. Hennepin County and Ramsey County, significantly differs from that in smaller Minnesota counties. These differences in case distribution were also observed in our comparison between the two racial groups.

Recommendation. We believe that the SDM tool should be replaced by a more discriminating screening tool with a "Selectivity" rating approaching that of the APS override process. Complemented by an improved vulnerable adult maltreatment reporting system, the new decision tool should aim to “Reach” every potential case of maltreatment in a county. The parental assessment tool in the Northstar Care program for the welfare of children and youth developed by the Minnesota Department of Human Services (5) is an example of such a screening system.

Testing for Similarity of Racial Distribution among Case Flows. Tracing how cases flow through assessment processes or standardized screening tools requires the adoption of a concept of a screen. Cases flow into a screening tool where a portion are screened-in and screened-out. Cases screened-in has attributes that closely conform to a set that defines a vulnerable adult subject to maltreatment or in a maltreatment environment. Screened-out cases possess non-conforming attributes. Statistical analyses in our article relied on chi-square testing (4) (5) (6).

The table below compares reported Black/African American cases in larger counties represented by Hennepin and Ramsey counties to cases in smaller counties comprising the rest of Minnesota. The distribution of cases and corresponding expected values were calculated. The chi-square test analysis (7) (8) yielded, p = 0.00, which indicated that the alternative hypothesis held. The distribution of Black/African American cases between SDM Screen-in and Final Screen-in from larger and smaller counties are not similar. In larger counties, the SDM tool tended to screen in more Black/African American cases than expected while the tool tended to screen in less than expected in smaller counties. The opposite is true with the override process. The larger counties had screened in fewer Black/African American cases than expected, while smaller counties screened-in more than expected.

In the table below, reported Caucasian cases in larger counties represented by Hennepin and Ramsey counties were displayed with cases from the remaining counties in Minnesota. The distribution of cases and their respective expected values were calculated. The chi-square test analysis performed yielded, p = 0.00, indicating that the alternative hypothesis held. The distribution of Caucasian cases between SDM Screen-in and Final Screen-in from larger and smaller counties is not similar. In larger counties, the SDM tool tended to screen in more Caucasian cases than expected, while in smaller counties the tool tended to screen in fewer than expected. The override process produced opposite screening results. Larger counties screened in fewer Caucasian cases than expected, while smaller counties screened-in more.

The last table shows the flow cases from those reported from the counties, through the SDM screen through the Override process. Expected distributions of cases were calculated from the observed distribution along the flow of cases. A Chi-Square test with two degrees of freedom yielded, p = 0.00, indicating that the alternative hypothesis held. The differences between the observed distribution of cases and their respected expected values are statistically significant.

This test suggests the following: (1) the SDM tool screened-in fewer Caucasian cases than expected while screening more Black/African American cases than expected, and (2) the override process screened-in more Caucasian cases than expected while screening-in fewer Black/African American cases than expected.

References:

1. Evaluation Report of Adult Protective Services Standardized Intake Decision Tool. Adult protection / Minnesota Department of Human Services (mn.gov). 2021.

2. Appropriate Evaluation of Standardized Assessment Tools. Www.MgmtLaboratory.com, Apr 2022.

3. Predicting Vulnerable Adult Maltreatment Cases in Counties. Www.MgmtLaboratory.com, Jan 2022.

4. Minnesota Department of Human Services. https://mn.gov/dhs/assets/2017-2020-VAP-county-dashboard_tcm1053-488092.pdf

5. Minnesota Assessment of Parenting for Children and Youth Practice Guide. https://edocs.dhs.state.mn.us/lfserver/Public/DHS-7060-ENG

6. Association of Minnesota Counties. County Data (mncounties.org)

7. Continuous Improvement and the Administrator. Www.MgmtLaboratory.com, Mar 2019.

8. County Performance Comparisons: Chi-Square Testing for Similarity. www.mgmtlaboratory.com, Aug 2019.

By Staff. Www.MgmtLaboratory.com 2022

_________________________________________

Mgmtlaboratory.com staff and affiliated management consultants offer their experience in management methods and managerial tools to private and public organizations. Non-profit organizations and government entities may inquire at contact@mgmtlaboratory.com about free online consulting services. Www.mgmtlaboratory.com is operated by Service Administration Laboratory Corporation, a Minnesota non-profit organization.