NLM logo
National Information Center on Health Services Research and Health Care Technology (NICHSR)

HSRProj (Health Services Research Projects in Progress)

Information about ongoing health services research and public health projects


Comparative effectiveness of clinical decision making processes required by public health systems
Investigator (PI): Mark, Tami
Performing Organization (PO): (Current): RTI International / (919) 541-6000
Supporting Agency (SA): Patient-Centered Outcomes Research Institute (PCORI)
Initial Year: 2018
Final Year: 2022
Record Source/Award ID: PCORI/IHS-2017C1-6371
Funding: Total Award Amount: $1,936,744
Award Type: Contract
Award Information: PCORI: More information and project results (when completed)
Abstract: Background and significance: Substance use disorders (SUD) are a major cause of premature mortality and morbidity, contributing to 135,000 deaths in the United States each year. Over the last decade, the increasing prevalence of SUD has led to a reversal of gains in life expectancy for segments of the United States. Opioid overdose deaths have reached epidemic levels. Effective addiction treatment can reduce morbidity and mortality and improve functioning in key life domains, such as employment, social relationships, and criminal behavior. However, SUD treatment delivery systems in the United States are frequently not delivering high-quality, effective treatment. On average, only 10% of those with SUD receive any treatment, a third of patients in specialty treatment facilities discontinue treatment prematurely, and only a minority of patients receive optimal treatment such as medication-assisted therapy for opioid addiction. One way to improve addiction treatment systems in the United States is to require the use of evidence-based, structured, and transparent methods of patient assessment and addiction treatment matching. Meta-analyses conclude that matching addiction treatment settings, interventions, and services to an individual's particular problems and needs is critical to effective addiction treatment. The National Institute on Drug Abuse (NIDA) identifies treatment matching as a core principle of effective addiction treatment. Yet, implementing this principle within addiction systems has proven challenging. Until the late 1980s, providers had little financial incentive to systematically consider patient matching. The standard for addiction treatment referral was to refer patients to the most intensive services that the patient would accept, typically a 28-day inpatient rehabilitation program followed by discharge to self-help group participation or weekly group counseling. In the 1980s, however, researchers demonstrated that, for many patients, intensive outpatient treatment was as effective as rehabilitation. In the early 1990s, managed care companies began to push back on paying for long stays in residential treatment and developed their own (often propriety) criteria to determine when more intensive, costly treatment was medically necessary. The American Society of Addiction Medicine (ASAM) patient placement criteria were developed to provide a systematic, evidence-based, and transparent approach to addiction treatment matching. The ASAM patient placement criteria base recommendations for the most appropriate setting on an assessment of each patient's biopsychosocial severity and function across six dimensions: (1) acute intoxication and/or withdrawal potential, (2) biomedical conditions and complications, (3) emotional/ behavioral/cognitive conditions and complications, (4) readiness to change, (5) relapse/continued use/continued problem potential, and (6) recovery environment. A hierarchical decision-tree algorithm then recommends a level of care that is designed to maximize the possibility of successful treatment, which is broadly defined as Level I (outpatient services), Level II (intensive outpatient/partial hospitalization services), Level III (residential/inpatient services), and Level IV (medically managed intensive inpatient services). Studies have demonstrated the reliability and validity of the ASAM criteria and find that treatment matching based on ASAM criteria leads to better treatment retention and outcomes relative to care as usual. An electronic version of the ASAM criteria offers a structured, computer-guided interview, based on validated instruments, that results in a biopsychosocial assessment of patients with SUD and an algorithm-based level of care recommendation. The ASAM criteria may also address the significant need for greater patient information about addiction treatment options and decisions. There is an increased recognition of the importance of creating a patient-centered health care system generally, and an addiction health care system specifically. Research with patients reveals that patients and their families desire more information about options for substance use treatment. For example, in a recent PCORI-funded project that engaged patients in discussions of how to make SUD treatment more responsive to patients' and family members' needs, one of the three areas that emerged as a priority was creating ways to improve patient and caregiver knowledge about SUD treatment options, and the development of tools to ensure that initial patient placements into SUD care were appropriately matched to their clinical, emotional, psychosocial, and spiritual needs. The recent implementation of Centers for Medicare and Medicaid Services Medicaid 1115 SUD demonstration waivers offers an opportunity to address the long-standing problem of the mismatch between patient needs and treatment approaches. A key requirement of the 1115 SUD waivers is that addiction providers systematically use ASAM criteria to determine optimal addiction treatment among addiction treatment options. As of writing, several states are beginning to implement 1115 Medicaid SUD demonstrations, and many others states are in the process of submitting a waiver or have waivers under CMS review. These waivers offer a unique opportunity to test the ability of ASAM criteria to improve SUD treatment delivery systems and addiction treatment outcomes. Although ASAM decision criteria have been shown to be a reliable and valid tool for patient placement decisions, the ASAM criteria have never been evaluated when implemented systemwide, under real-work conditions. Furthermore, there has been no research on whether the ASAM software leads to improved outcomes relative to the non-computerized implementation of the ASAM criteria. Finally, it is not known if the ASAM criteria help patients understand their addiction treatment options. The California 1115 SUD demonstration waivers provide natural experiments in which to test the systemwide implementation of ASAM criteria. California was approved for an 1115 waiver in 2015. Because California's public SUD system is county based, some counties have decided to participate in the 1115 waiver and some have decided not to participate. Counties that are not participating will continue to base treatment decision making on the judgment of SUD professionals using non-standardized, non-transparent assessments and treatment placement criteria. Among counties that are participating in the 1115 wavier, some will be implementing a computerized version of the ASAM criteria and some will implement the non-computerized version. The California 1115 SUD waiver rollout establishes a strong naturalistic research design, whereby changes in treatment outcomes before and after ASAM implementation in 1115-waived counties can be compared to non-waived "control" counties using interrupted time-series approaches. State and local governments, the federal government, and private payers are searching for opportunities to improve their addiction treatment systems to better serve the thousands of Americans with opioid and other addictions. A critical question facing these systems is how best to ensure that patients receive the most appropriate treatment that maximizes their chances for recovery. Today, only about 55% of substance abuse treatment clinics use the ASAM criteria, and many of those facilities may not be using it optimally. Should the ASAM criteria, as applied through computerized or non-computerized approaches, prove effective, other SUD systems will have the evidence necessary to support investment in the ASAM implementation. Patients also have great interest in identifying approaches to determine the best addiction treatment options, given their clinical and social circumstances. Study aims: This study will compare three approaches to addiction treatment decision making as implemented in large public substance use disorder systems: (1) use of ASAM by SUD providers who have received extensive training in ASAM criteria implementation, (2) use of a computerized ASAM-based decision support tool with standardized assessments, or (3) use of non-standardized patient assessment and treatment matching. Our specific aims are (aim 1) to test whether health system implementation of ASAM criteria results in better treatment retention and reduced substance use relative to decision making based on clinical judgment alone, (aim 2) to test whether health system implementation of the computerized version of the ASAM results in better treatment retention and reduced substance use relative to the non-computerized version of ASAM, and (aim 3) to test whether ASAM criteria implementation leads to better patient information about treatment options and subsequently to better treatment engagement and substance use outcomes. Study description: California has a total population of 37 million people, of which 12.5 million are Medicaid enrollees. Approximate 14% of Medicaid beneficiaries in California have a substance use disorder. Approximately 4% of Medicaid beneficiaries access SUD treatment in any given year. California's public system for treatment of substance abuse is administered by 58 county drug and alcohol treatment programs. Counties contract with SUD providers and establish regulations regarding admission criteria and reporting requirements. This study will determine whether county-wide roll out of the ASAM criteria, or the computerized ASAM criteria, improves addiction treatment effectiveness. The within-county change in treatment effectiveness before and after the ASAM criteria roll-out will be compared to counties that opted not to implement ASAM. The study will focus on five counties: Two of which are implementing the standard ASAM criteria, two of which are implementing the computerized ASAM, and one which is not implementing ASAM. ASAM counties: The two ASAM counties in the study are Santa Clara and Alameda counties. These counties are requiring their providers to use ASAM criteria to assess patients and to determine level of care. These counties also use ASAM criteria to determine the ongoing need for high-intensity services, such as residential, and approve those services for Medicaid payment. Computerized ASAM counties: Los Angeles and Marin counties have decided to implement the ASAM criteria using an electronic version of ASAM criteria. The electronic version of the ASAM criteria is a web-based, clinical decision support software program. The ASAM software launches a computer-guided patient assessment interview based on validated tools such as the Addiction Severity Index. Using the information about the patient's needs and history collected through the interview, the software conducts a number of calculations to recommend a level of care. Clinical judgment in non-ASAM counties: In San Diego county, ASAM is not required. County providers have discretion in how they assess patients and determine the most appropriate treatment settings. Placement decisions are retrospectively reviewed by other community-based organizations under contract with the county. Outcomes: Primary outcomes. We will evaluate whether the implementation of the ASAM criteria improves treatment retention and substance use reduction. Retention in treatment for at least 14 days is a validated outcome measure and has been shown to be predictive of reduced substance use, 2-year mortality, criminal justice involvement, and detox episodes. Abstinence and reduction in substance use is necessary for patients to achieve recovery and improved psychosocial functioning. Therefore, substance use is commonly used as a key outcome in comparative effectiveness studies of alcohol and drug treatment. Intermediate outcomes: The ASAM criteria are anticipated to improve treatment outcomes (i.e., treatment retention and substance use) by increasing the use of more appropriate treatment settings for each patient, such as shifting the number of patients using residential, intensive outpatient, and opioid treatment programs. We will examine whether the distribution of the treatment setting used at treatment initiation changes after ASAM implementation. We also hypothesize that ASAM implementation will improve communication with patients about treatment options. Data sources and data collection: The main quantitative data sources are data from the California Outcomes Management System and data collected with a patient survey. Qualitative information will also be collected through interviews with patients and providers. California Outcomes Management System (CalOMS) data. CalOMS has a large, high-quality longitudinal data set that is well-suited to this study because it captures information on all Medicaid-funded treatment provided by addiction facilities in California. All publicly funded substance abuse treatment agencies in California must report admission and discharge data to the state via CalOMS, regardless of whether treatment is successfully completed or not. Providers are required to submit data on each patient every 30 days. CalOMS data elements include amount of alcohol and drugs used at discharge, primary substance of abuse at admission, payer source, age, gender, treatment modality, wait-time for treatment, time in treatment, and other variables. These data are collected through patient interviews using standardized questions. The state and counties inspect and verify the CalOMS data completeness and quality through automated summary reports. The state and counties require that missing or erroneous data be corrected. They also require that providers obtain training in data collection and submission protocols. The CalOMS database contains all SUD admissions starting in 2007. The data are available with a 6-month lag. Numerous peer-reviewed studies have been published using the CalOMS data. Patient survey. We will evaluate whether the implementation of the ASAM criteria results in improved patient information about treatment options by surveying patients using a reliable and valid measure of patient-provider communication. Patients receiving treatment at addiction clinics within each of the five counties will be surveyed at the beginning of their treatment episode regarding communication with their providers about treatment options. They will be surveyed 3 months later regarding their engagement in treatment and substance use. Patients will be recruited from SUD clinics that are under contract with the counties. The SUD clinics will represent the main forms of treatment settings, including SUD outpatient, SUD residential, and narcotics treatment maintenance. County behavioral health departments and the state department of health will reach out to the SUD providers to ask for their assistance with the research. Onsite staff at the SUD clinics will assist with patient recruitment, enrollment, and consent. Clinics will be compensated for their time in conducting the research study. We will train the staff in the study protocol and research protocols. This will include, but not be limited to, good research practices, human subject protections, data safety and data integrity, protocol review, and privacy regulations. Research teams at clinics will recruit patients age 18+ with Medicaid coverage who have been in treatment for 30 days or less. Patients will be selected using a stratified approach so that the proportions of patients with respect to age, gender, primary drug of abuse, and prior treatment episodes are equivalent across the study arms and reflective of the patient SUD population in the counties. Patients will be introduced to the study and provided a brochure. If the patient agrees to participate, the provider will review the IRB-approved informed consent form with the subject and obtain consent. Contact information for consented patients will be provided to RTI. Patients will complete the baseline survey using a smart phone, laptop, or telephone. A reminder will be sent out 3 months later inviting the patient to conduct a follow-up survey that will take about 10 minutes. To enhance response rates, each patient will receive a $10 incentive for completing the baseline survey and a $15 incentive for completing the follow-up. The survey will be available 24/7 over the telephone using interactive voice recognition software (i.e., a prerecorded voice administers the questions verbally and the software can accept verbal replies or keyed numbers) and through an online link that can be accessed via a web-browser. All survey modes will be available in English and Spanish. The survey will be hosted on a secured server. Patient-provider communication instrument development. We will develop an instrument to measure the degree to which patients being treated for addictions were provided information about their treatment options and information as to why the provider recommended the treatment. We will also measure whether the patient feels that the choice of treatment was in their best interest. We will develop the initial draft survey by drawing on existing measure constructs and source instruments. After finalizing the measure constructs, we will map existing measures and items to those constructs, and identify any gaps for de novo measurement development. Our team will use a comprehensive approach to developing and testing the survey instrument. The process will ensure that the final instrument assesses the constructs of interest, is reliable and valid, and is easy for patients to use. The multidisciplinary Study Advisory Committee will provide input throughout the survey development process, including advice on measures, review of iterative drafts, and advice on survey data collection. We will review the instrument using the Question Appraisal System (QAS), a structured, standardized methodology developed by RTI to assess common problems with question wording, tasks required of respondents, formatting, and other elements of survey items that may contribute to error. The QAS is both a coding system (i.e., item taxonomy) that describes the cognitive demands of the questionnaire and a method of documenting question features that are likely to lead to response error. These potential errors include errors related to comprehension, task definition, information retrieval, judgment, and response generation. We will use a cognitive interviewing approach to assess respondents' understanding of the questions, instructions, and response options. Cognitive interviewing employs a "think-aloud" approach, which allows the interviewer to observe reactions to questions and hear the respondent's comments and questions, leading to quick identification of unclear terminology, ambiguous phrasing, or inappropriate or missing response options. We will conduct two iterative rounds of cognitive interviews with 6-9 patients per round. We will recruit patients from the SUD clinics to participate in the cognitive testing. We will also conduct usability testing of web-based and telephonic surveys to ensure that respondents can easily navigate the survey, designing features that enhance the user experience and encourage response. We will evaluate how effectively and efficiently participants navigated the web survey. Qualitative data collection: Patient and provider interviews. We will collect qualitative, in-depth information about the treatment decision making processes within the three study arms through interviews with patients and providers. The research team will purposively select patients from the survey sample to participate in qualitative semi-structured interviews. Patients will be selected for interviews based on demographics (age, gender, race/ethnicity) and which substance(s) are the focus of their substance use disorder (e.g., stimulants, alcohol, opioids, cannabis). The goal of sampling will be to develop a qualitative sample that reflects the diversity of patients and patient experiences in California's Medicaid SUD programs. Six patients will be selected from each participating treatment county (30 patients in total), and semi-structured interviews (approximately 30 minutes each) will be conducted over the phone. Patients who participate in interviews will be compensated for their time with a $75 gift card. The semi-structured interviews will be designed to elicit patient perspectives on how they were linked to specific treatment programs, their beliefs about how well their level of care matches their treatment preferences and needs, their experiences in care, their treatment satisfaction, and other issues pertinent to key study outcomes. The patient semi-structured interviews will be recorded and transcribed. The research team will also conduct interviews with treatment staff at county-contracted SUD clinics (including residential, outpatient, and methadone maintenance clinics). Staff who are central to the treatment placement decisions will be invited to participate. Six treatment providers will be interviewed in each county for a total of 30 interviews. The semi-structured interviews will last approximately 30 minutes and be conducted over the phone. Providers who participate in semi-structured interviews will be compensated for their time with a $75 gift card. The semi-structured interviews will be designed to elicit provider perspectives on their county's patient placement procedures, their beliefs about how well patients' levels of care match their treatment needs and preferences, and information on other issues concerning the relationship between patient placement, facilitators in administering the ASAM criteria with fidelity, and barriers to use of the ASAM criteria. The provider semi-structured interviews will be recorded and transcribed. Quantitative data analytic plan: Our mixed-methods design will include quantitative analysis of the CalOMS and patient survey and with qualitative data from patient and provider interviews. Aim 1 is to test whether health system implementation of ASAM criteria results in better treatment retention and reduced substance use. Our data analyses will take advantage of natural experiments created by the 1115 Medicaid demonstration waivers in California to quantitatively determine the impact of implementing the ASAM criteria. Counties implement the ASAM criteria at different times, with some counties not implementing at all, providing within-county and between-county variation that enables us to assess the comparative effectiveness of the three approaches to treatment decision making. We will quantitatively determine the comparative effectiveness of the ASAM placement decision making approach using CalOMS data and a comparative interrupted time series (CITS). With a CITS design, program impacts are evaluated by looking at whether, in the follow-up period, the treatment group deviates from its baseline trend (baseline mean and slope) by a greater amount than the comparison group. The CITS design implicitly controls for differences between the treatment and comparison group with respect to their baseline outcome levels and growth. CITS is a strong design when randomization is not possible, as is almost always the case with health system interventions. The CITS model's strengths include the ability to control for secular trends, clear graphical presentation of results, ease of conducting stratified analyses, and ability to evaluate both intended and unintended consequences of interventions. We will use pre-period propensity score matching to ensure that the counties are similar across key characteristics that may influence the effect of the ASAM implementation. Importantly, we will control for pre-period engagement and substance use disorder rates by provider (which will be calculated by aggregating individual data from the CalOMS to the provider level using the provider identifier). We also include patient-, provider-, and system-level characteristics to control for these differences and include county fixed effects to capture time-invariant unobserved characteristics at the county level. For example, we include patient variables such as primary drug of abuse, age, and gender; facility measures such as percentage of patients who are covered by Medicaid and average time in treatment of patients; and system-level variables such as the number of residential SUD beds in facilities that accept Medicaid. We will assess the pre-implementation balance between implementing and non-implementing counties and the overlap in their propensity scores. The CITS model will include 4 years of pre-implementation observations and 2 years of post-implementation observations. We will model the outcomes by month; thus we will have 48 months pre-intervention and 24 months post-intervention. Because Alameda County's implementation pre-dates the study period, their post-indicator is always turned on and they are treated as a pseudo-control group. Given the exogenous variation is driven by county-level differences and that the main model includes only five counties, standard errors will be calculated using a wild bootstrap procedure that corrects for a small number of clusters. Aim 2 is to test whether health system implementation of the computerized version of the ASAM results in better treatment retention and reduced substance use relative to the non-computerized version of ASAM. We will use the CITS model to estimate the effect of the two different ASAM implementation approaches: the traditional ASAM approach and the computerized ASAM approach. In addition to conducting analyses on these five counties, we will conduct sensitivity analyses using the larger sample of all counties. All of the counties that decide to implement ASAM will do so as of the end of 2017. By using all 58 counties, we will have a large sample of counties on which to test the impact of the ASAM criteria, thus allowing for a sensitivity test of the results in the study population counties. Any differences between the results from the two populations will likely point to the need for further understanding of the variation in how the ASAM criteria are implemented and the extent to which they are guiding treatment placement. Aim 3 is to test whether the ASAM criteria implementation leads to improved communication with patients regarding treatment options, patient engagement, and subsequently to better substance use outcomes. We will conduct analyses on the data collected through the patient surveys to test for difference between the three arms in patient communication, engagement, and substance use. We will first analyze the effect of the ASAM criteria implementation on patient engagement, using regression and propensity score techniques to control for confounding factors. Then we will use multivariate regression with propensity weighting to analyze the effect of patient communication at treatment entry on substance use outcomes 3 months later. Qualitative data analysis: The interview transcripts will be analyzed in Dedoose analytic software using template analysis--a group of techniques for thematically organizing and analyzing textual data. Transcripts will be analyzed using a mix of thematic codes that are defined a priori as critical for study questions (e.g., patients' perspectives on program appropriateness for their needs, patient satisfaction), and codes that will be added to the coding scheme as the project's qualitative researchers read and interpret the transcripts. Through an iterative process, the researchers will edit and supplement a priori codes until consensus is reached on a codebook that lists and defines all codes to be used in qualitative analysis. The qualitative researchers will then use the codebook to code all transcripts using Dedoose. After coding all transcripts, the qualitative researchers will conduct a code reconciliation process in which they will discuss discrepancies in coding until they reach 100% consensus on all codes. They will then use coding results to define major themes that emerged from patient and provider interviews and advise the study PI on ways that coding results can complement quantitative findings and inform the interpretation of quantitative results.
MeSH Terms:
  • Algorithms
  • California
  • Centers for Medicare and Medicaid Services (U.S.)
  • Comparative Effectiveness Research
  • * Decision Making
  • Decision Support Systems, Clinical
  • Evidence-Based Medicine
  • Health Services Research
  • Humans
  • Life Expectancy
  • Medicaid
  • Mental Health Services /*organization & administration
  • National Institute on Drug Abuse (U.S.)
  • Opioid-Related Disorders /*rehabilitation
  • Outcome Assessment (Health Care)
  • Patient Education as Topic /methods
  • Public Health
  • Self-Help Groups
  • Software
  • Substance-Related Disorders /*rehabilitation
  • United States
Country: United States
State: North Carolina
Zip Code: 27709
UI: 20183351
Project Status: Ongoing