Current Issues & Findings in Mental Health Services Research
Although not an exhaustive review of current issues, this section discusses some of the salient issues in mental health services research. Current issues include management, use, accessibility, cost, effectiveness, and quality of mental health services.
Managed Care
Although earlier events contributing to the rise of managed care can be singled out, the passage of the Health Maintenance Organization (HMO) Act in 1973 led to the rapid growth of managed health care systems, particularly HMOs. One of the primary aims of managed mental health care has been the containment of costs due in large part to high inpatient utilization rates, much of which is asserted to be inappropriately restrictive and costly. Another primary goal of managed care is to increase access to needed care, particularly for those who are underinsured. By law, HMOs were to provide eight basic services, including emergency and outpatient crisis intervention services and treatment and referral for substance abuse in addition to general health services: consultation and referral, emergency health, diagnostic laboratory and radiologic, home health, and preventive health services.
Managed mental health care, or behavioral health care, generally refers to the third-party oversight of the cost and quality of mental health services. Providers and insurers define a specific target population and benefits package aimed at providing integrated, comprehensive, and coordinated care, often at a capitated rate. Although managed care is now the primary model for the majority of those privately insured, development in the public sector has been much slower. However, state mental health authorities more frequently are increasing controls on expenditures, restructuring the financing and delivery of health care under Medicaid, and opting to contract out for managed mental health care services. Between 1991 and 1995, Medicaid managed care increased threefold, from covering 9.5% of enrollees to a total of 32.1% in 45 states.
Managed mental health care may be provided by a number of mechanisms: (1)"carve-out" programs, whereby a specialty behavioral group contracts with a net-work of individual clinicians, sets up its own clinic, or does a combination of these things; (2) employee-assistance programs; (3) integrated delivery systems (ie, a “carve-in” system), in which individual treatment facilities expand to take on mental health care; (4) HMOs, in which a primary care physician serves as a gatekeeper to additional, more specialized services; and (5) Medicaid carve-out experiments, whereby CMHCs collaborate with managed care companies. Managing mental health care typically entails many of the following elements: prepayment of care; prior authorization; utilization review or case management; preferred providers; negotiated rates; practice guidelines; or the development of comprehensive managed mental health programs for employers, insurers, or HMOs.
The HMO Act expanded mental health coverage, and the prepayment mechanism allowed more flexibility in service delivery than traditional indemnity plans. However, in the 1980s, indemnity plans expanded their mental health benefit packages. Criticism of HMOs subsequently arose, contributing to the rise of carve-out programs. Traditional indemnity insurance is generally more restrictive than managed care approaches, limiting reimbursement to specific diagnoses and particular treatment processes and settings, whereas prepayment options tend to encourage more flexibility and innovation such as the use of paraprofessionals in service delivery.
Health care reform has not been without concern. The 1973 mandate for mental health services was imprecise, leaving considerable flexibility and latitude to administrative decision-making regarding the nature and extent of services provided by HMOs. There has been a general lack of awareness about services delivered within HMOs, particularly regarding the delivery of mental health services. Further, several legal and ethical issues have been raised in relation to managed mental health care.
Research Findings
Little research has addressed the effects of health care reform, particularly in publicly funded populations. As states increasingly incorporate a managed care model in service delivery, research is needed that examines the impact of such broad-based reform on the quality of care received by individuals with severe emotional disturbance and by other vulnerable populations. The original focus of the HMO movement was the delivery of high-quality, accessible services at an affordable cost, but emphasis has shifted to containing costs, providing investment opportunities, and extracting profits. Although the emphasis of managed care has diverged from its original focus, the effects of managed care that have been debated are those original issues—quality, access, and cost.
Capitation alone does not appear to result in reductions in access, adequacy, and appropriateness of services for seriously emotionally disturbed individuals. Studies of access to general health and mental health services have demonstrated equal or superior access for managed care as compared to fee-for-service arrangements for both publicly funded and privately insured populations. However, managed care appears to limit access to specialists, including mental health specialists. Client, family, and system outcomes need to be maintained to see whether system and financing changes negatively affect quality of mental health services and outcomes.
Some research has reported that managed mental health care has been successful in achieving its goal of reducing inpatient hospitalization for both publicly funded and privately insured populations. An evaluation of a prepaid plan, carve-out approach implemented in Utah also has demonstrated reduced utilization of inpatient care. The Fort Bragg Demonstration Project resulted in significant shifts from inpatient to less restrictive outpatient mental health care with no decrements in mental health outcomes. Efforts in New York to develop a community-based, family-centered system of care cut inpatient utilization rates for children with serious emotional disturbance.
Further, managed care has been shown to reduce costs of mental health as well as general health care. Results from an evaluation of Massachusetts’ carve-out approach yielded reduced costs, as did Utah’s approach. A Government Accounting Office (1993) review of several states’ managed Medicare approaches found reductions in cost. Additional evidence of cost savings has been collected by an evaluation of service system change in California. However, not all results have been positive.
There is general agreement that managed care has the potential to reduce and contain costs of care. Although some positive evidence indicates better access for managed health and mental health care, and scant results in general health care indicate comparable or improved quality in managed care, debate and suspicion regarding these assertions continues. For example, Miller (1996) claimed that evidence indicates the quality of outpatient mental health care suffers under managed care because of efforts to reduce costs by denying needed services and providing inadequate treatment (eg, by cutting long-term treatment for individuals with moderate to severe illness that may benefit from such treatment). Moreover, surveys of clinicians indicate that many are frustrated and dissatisfied with managed behavioral health care and that they feel that it negatively affects the quality of delivered care.
The uncertainty of the impact of managed care approaches on access to and quality of care is especially true in the public sector, where fewer evaluations have been conducted than for managed care approaches in the private sector. Investigations that have been conducted have demonstrated some success as well as areas of concern. For example, the evaluation of the Massachusetts approach highlighted that although costs were reduced, readmission rates and lengths of stays were higher, specifically for children and adolescents.
Clearly, more research is needed in order to examine how broad-based system reform affects the utilization, access, cost, quality, and outcome of mental health care. Attention should be given to conducting implementation studies to determine the nature and extent of reform efforts before impact assessments are conducted; to evaluating services and systems using a systems perspective; and to collecting both process and outcome data. In studies of publicly funded populations, additional investigation must be aimed at vulnerable populations such as children and adolescents. Although variability exists across state models of care (thereby making inferences more difficult), the differences in findings between studies of publicly funded and privately insured populations needs further investigation.
Finally, in prelude to the remainder of this chapter, the changing structure of the health care system can be expected to influence utilization, access, and cost, but there is strong doubt that system change alone can influence the effectiveness of mental health services.
Utilization
Introduction
It has been estimated that 12–22% of children and adolescents have a diagnosable mental disorder, and some researchers suggest an increase in prevalence over recent years. The ECA study and the NCS estimated that 30% of adults will have a diagnosable mental illness in any given year and that as many as one out of every two adults will experience one or more disorders in their lifetime. Findings suggest that mental illness is widespread, often chronic, and highly comorbid; however, it is estimated that only a small proportion of those in need of services receive them. Studies of service utilization aim to describe what types of services are used, by whom, in what amount, and in what context.
Research Findings
Research has demonstrated substantial unmet mental health service needs. The Great Smoky Mountains Study of Youth (GSMS), a population-based community survey of youth, found that 20.3% of the surveyed youth had a diagnosable mental disorder, although this result did not include those adolescents with substantial impairment but no diagnosis. Generally, less than one third of those in need of services received them, and specifically, only 40% of adolescents with severe emotional disorder received services. Findings from the ECA study and the NCS illustrate an even gloomier picture for adults in terms of unmet mental health needs. According to these two studies, 70% to 79% (respectively by study) of adults with a disorder in the past 12 months reported that they had not received services. Barriers to service utilization that contribute to unmet needs are considered later in this chapter, in the section titled “Access.”
Utilization research also has examined the service sectors, or settings in which services are obtained (eg, juvenile justice, education, child welfare, specialty mental health, and general medical care settings). Although not drawing from a nationally representative sample, the GSMS found that of the 19% of adolescents who received services in one or more than one sector, approximately 18% utilized services in specialty mental health settings, 85% in education settings, 11% in primary care, 2.5% in child welfare, and 7.6% in juvenile justice settings. Children with serious emotional disturbance tended to be served in multiple sectors, including specialty mental health settings, underscoring the importance of coordination across multiple sectors. A sad footnote to these findings is the relative dearth of research in settings outside specialty mental health, particularly in educational settings.
In contrast, research on adult utilization has demonstrated relatively more mental health service use in the general medical sector as compared to child and adolescent utilization findings. Nonpsychiatric physicians provide close to half of all mental health services. In fact, the trend is growing, attributed partially to increasing partnerships between primary care physicians and mental health professionals. Finally, in addition to general medical settings, the importance of informal voluntary support networks (eg, self-help groups, friends, relatives), which provide a significant proportion of services, is beginning to be acknowledged.
Utilization research also has investigated demographic variables as they relate to service use. Service utilization has been linked to education, marital status, race, distress, and insurance coverage. Additionally, education, income, and insurance coverage have predicted the length of outpatient treatment. Because of the importance of demographics, utilization research should incorporate a minimum standard data set including these variables.
System reform efforts of states and communities is ongoing. For example, CASSP principles that aim at improving access, limiting barriers, providing a continuum of care, and increasing coordination among service sectors motivate reform efforts for providing mental health services to children, adolescents, and their families. These and other broad-based reform efforts, such as state Medicaid reform, can be expected to change patterns of mental health service utilization. Utilization and de facto policy are influenced by insurance mechanisms that often undermine intended or de jure policy. Thus further research is needed to inform public policy and planning because inasmuch as aims of reform better address unmet mental health needs, they may increase service utilization. It is likely that these efforts will put further strain on limited financial and human resources.
Access
Introduction
Although there may be substantial disagreement about the way in which care is delivered, there is more agreement that everyone, regardless of financial status, should be entitled to some health care. Access to general health care as a public policy objective has been debated since the early 1900s. The enactment of Medicare and Medicaid in the mid-1960s greatly expanded health insurance coverage for certain populations. Limited federal initiatives addressed the provision of mental health services and indirectly improved access to mental health care. Such initiatives included the HMO Act and the Community Mental Health Center Act, amended in 1971 to specifically mandate the provision of mental health services for children and adolescents. Additionally, among the goals of managed care is to improve access to health and mental health care.
The findings reported in the “Utilization” section indicate that a substantial proportion of those individuals with a diagnosis or functional impairment do not report receiving mental health services. This discrepancy may be due to individual barriers such as personal economics, attitudes and beliefs, knowledge about services, family history of mental health problems, family support, stigma, or perceived need for services. Howard et al (1996) illustrated the potential pathway to mental health services by distinguishing between (1) intrapersonal decisions about the presence of a mental health problem and what to do about it and (2) interpersonal and external behaviors needed in order to overcome treatment barriers and enter into services. Further, the failure to receive services could be due to system barriers, such as inadequate transportation, limited service hours, unavailability of appropriate services, poor coordination of services, unhelpful staff, and restrictive eligibility criteria.
Access to mental health services for children and adolescents is more complicated. Parents often serve as additional gatekeepers by making decisions about the initiation or termination of services. A child’s receipt of services depends not only on the child’s willingness but also in part on the parent’s perceived need for and ability to obtain services. Access can be increased by utilizing standardized screening for emotional, behavior, and substance abuse in children and adolescents, particularly those at high risk, such as children in foster care. For example, Halfon et al (1995) reported that only about 41% of foster parents of children demonstrating emotional, behavior, or developmental problems reported or were aware of the problems. Because children depend almost solely on their parents or primary caregivers to seek needed services, when access to needed services is not facilitated through alternative routes (eg, screening in primary care, schools, courts), children and adolescents will remain underserved.
Research Findings
Research has indicated that individual barriers to service utilization are more problematic than are system barriers. In the ECA study, 80% of the participants were very receptive to mental health services, but 83% perceived at least one barrier to receiving those services. Child and adolescent perceptions of services were critical as two of the most frequently perceived barriers to services. Both were related to the child’s refusal to seek services. However, parents who previously had sought services reported more barriers than did those who had not recently used services. In contrast, families who subsequently had utilized school-based services reported fewer barriers. Ultimately, parent-reported barriers to access have been linked to lower utilization of services.
Issues of access have been particularly salient with the rapid growth of managed care and health care reform in general. A substantial body of literature claims positive findings. For example, the Fort Bragg Evaluation Project found that a continuum of care in North Carolina greatly increased access to mental health services for children and adolescents compared to more traditional services at the two control sites. In fact, access at the demonstration site increased more than threefold over a 5-year period, whereas access remained stable at the control sites.
Although some research has suggested fairly good availability of psychological services, particularly in larger HMOs, service utilization patterns and treatment patterns are likely to be influenced by capitation. For example, Glied (1997) found that women receiving mental health care within an HMO were more likely to see a primary care provider than a specialist. Moreover, when compared to fee-for-service providers, HMO providers were more likely to prescribe medication and less likely to recommend psychotherapy.
Concerns about inappropriate barriers to mental health services delivered within HMOs and managed behavioral health care have been raised partly because of the model of care in which the primary care physician is a gatekeeper. Usually, the approval of the patient’s primary care physician is required for mental health services even though general practitioners and pediatricians often, and unfortunately, are not well-trained in mental health issues. Although roughly half of existing HMOs, for example, have some self-referral mechanism, patients often are not aware of these or how to access services. Problems limiting access include delays in scheduling nonemergency services, caps or limits on mental health coverage, patient copayments, and procedural difficulties in applying for psychological services.
Research has implied that managed behavioral health care may indeed enable better access to services, at least initially. However, mechanisms of managed care affect mainly system barriers to care, which may not be as important as individual barriers in determining access and service utilization.
Cost
Introduction
Research regarding the costs of mental health services is linked directly to service utilization. Programs designed to alter utilization patterns are typically designed so that costs of services can be reduced while maintaining quality of services. This endeavor intrinsically examines the efficiency of services or delivery systems (ie, how costs relate to benefits). The estimation of costs and benefits depends on the accounting perspective chosen. These perspectives include buyers (ie, individual patients), insurers (ie, insurance companies, Medicaid, Medicare), and society at large. For example, from the patients’ perspective, only resources that the individual pays out for treatment should be included in a tabulation of cost. These resources include out-of-pocket payments, lost wages due to treatment participation, and travel expenses. From the insurers’ perspective, only direct reimbursements for services or expenditures should be included. From a societal perspective, all of the resources mentioned above would be included as well as communal costs such as the lost productivity of family members or the ultimate costs of not providing treatment to those in need.
Hence, when examining the costs, one’s accounting perspective will determine the goods or services included as costs. Typically, in the analysis of social programs, it is difficult to identify and estimate costs from a societal perspective. Moreover, cost analyses seldom are conducted for the benefit of individual patients or families but rather for the agencies delivering services. This is especially true of publicly funded programs, where the cost to families is rather low. In light of these multiple perspectives, researchers have defined costs in two ways: the cost of mental illness and the cost of mental health treatment. The cost of treatment reflects a combination of the patient and insurer perspectives and considers only monies spent for mental health care. It includes costs of direct services (eg, the cost of an inpatient hospital stay or outpatient session) and indirect services (eg, family travel costs, missing work). The cost of mental illness reflects the societal perspective and considers the costs to society when an individual has a mental illness. These costs include lost productivity, the consumption of public resources, and the destruction of community property. Although both types of cost analyses are complex, cost of illness studies are much more difficult to conduct than are cost of treatment studies. As a result, cost of illness studies are rare and very expensive to execute.
Most studies on the costs of mental health services report the aggregate costs of mental illness, but they leave important questions unanswered. The primary problem is that because the estimates are pieced together from different data sources, there is no complete picture of the costs of mental illness for any given person. For example, in the children’s field, juvenile justice costs are derived from one source and costs of inpatient psychiatric care from another. These different costs are combined only at the aggregate level.
These cost considerations have several important implications. First, aggregate estimates do not allow one to look at how total costs vary across individuals of different ages, diagnoses, or family background. Second, aggregate estimates do not allow one to look at the relationships among the different types of costs. For instance, one cannot examine the relationship between the direct and indirect costs of mental illness and how that relationship varies across settings. The receipt of mental health services may reduce costs in other areas; thus the costs of mental health services would actually be much lower than the direct expenditures associated with them. A narrow focus on the direct costs can produce a vastly different picture about the costs of mental health services and may lead to very different public policy options. Third, the fact that aggregate estimates are not based on a single sample of individuals further obscures information about the variation that exists across individuals. Costs may vary substantially over the course of an illness, and different interventions may produce different cost trajectories over time; however, yearly aggregate cost estimates will obscure these observations. If mental health services represent an investment in children, spending now may reduce costs later. In contrast, foregoing current expenditures may only shift costs into the future, where they will be borne by other members of society. The best information will be provided with data that follows individuals over time and offers information on diagnoses, family environment, and other characteristics.
Research Findings
The increase in the cost of health care from the 1970s has been astronomical. Health care costs rose from $90 billion in 1973 to $735 billion in 1991, and it is predicted that health care expenditures will total $1.5 trillion in the year 2000. Mental health and substance abuse treatment costs have risen at a rate exceeding those for any other insured conditions.
A broader understanding of the full costs of mental illness includes consideration of direct and indirect costs, including mortality and morbidity, beyond specialty mental health services alone. Family members of those with mental illness bear still other costs, for example, limiting a parent’s ability to maintain an income.
Given the national environment of managed care, most current cost analyses concern the effects of financing of mental health services. A key determinant of who bears the costs of mental illness is the financial arrangement through which a family pays for mental health services. Unfortunately, there are gaps in our knowledge about the ways in which mental health services for children are financed and about the effect of financing on service utilization. Children and adolescents covered under more generous private insurance are reportedly more likely to use services and to use more services than are children and adolescents with less generous coverage. This was evident in the Fort Bragg Demonstration Project, in which access, utilization, and costs of services rose dramatically with the implementation of a no-copayment financing structure.
These studies, however, are rather limited. They have examined only families with private insurance and thus provide no means of comparing service utilization in uninsured families or in families with Medicaid. In addition, because prior studies do not examine services in all settings, they offer a narrow view of how families respond to financial incentives. These questions involve difficult methodological problems, most deriving from the fact that individuals are not assigned to insurance randomly. Because there is self-selection (ie, people choose insurance), factors other than insurance may determine their behavior.
Managed care is promoted as a powerful tool to increase cost-effectiveness, preventive care, and service quality. However, little research has examined the impact of managed care on actual costs, and the studies that have been undertaken have provided inconsistent findings. Whereas some researchers have reported little to no reductions in costs associated with managed care strategies, several evaluations have found evidence of cost savings as a result of incorporating a managed care approach. For instance, the California System of Care Model Evaluation Research Project, a longitudinal study initiated in 1989, sought, among other goals, to reduce California’s group home utilization rate. Researchers (Attkinson et al, 1997) found that time-series analyses of data provided evidence of real cost savings attributable to the System of Care implementation for the demonstration counties in comparison to aggregate state expenditures. Another study (Goldman et al, 1998) found that an introduction of managed mental health care with a carve-out model resulted in a decrease in costs of over 40% that was maintained over a 6-year follow-up period. The cost reductions were not attributed to reductions in access. Rather, lower costs were associated with fewer outpatient sessions per patient, lower inpatient utilization, reduced inpatient lengths-of-stay, and lower costs per unit of service. Researchers also found evidence in Colorado of cost savings for managed care. Preliminary findings indicated decreased costs and probability of outpatient use under capitation as compared to fee-for-service. The researchers found no adverse changes in patient outcomes. Although inconsistent results have been observed, many researchers believe that managed care has the potential to reduce and contain health care costs.
Effectiveness
Introduction
In the mental health services research field, a basic distinction has developed between the efficacy of an intervention and its effectiveness. Studies of the efficacy of an intervention examine the relationship between intervention processes and outcome under tightly controlled or optimal conditions. The investigator usually has substantial control over sample selection, the intervention, its delivery mechanisms, and the environmental setting. For example, randomized clinical trials (RCTs), often conducted within a university-based laboratory, are typically efficacy studies. In contrast, studies of effectiveness examine the impact of interventions in naturalistic settings or under average conditions of use. The researcher loses much of the control over factors such as sample selection, fidelity of services delivery, and choice of clinicians. Studies of effectiveness usually are conducted in real-world clinical settings (ie, schools, general medical settings, CMHCs, or HMOs).
One of the critical differences between the two types of studies is the trade-off between internal and external validity. Internal validity is the extent to which a causal linkage can be drawn between the intervention and the desired outcomes by ruling out alternative explanations of the observed effect (eg, maturation). The tighter the control held by the experimenter, the more likely he or she will be to rule out other explanations of observed change other than the studied intervention. External validity refers to the likelihood that the results observed in one investigation would be repeated, or would generalize, to other settings, samples, or times. Real-world delivery of interventions tends to include heterogeneity in the patient population, the services, and the environmental context in which they are delivered. According to Cronbach et al (1980), evaluation has placed too high an emphasis on internal validity. Furthermore, threats to external validity always are present, and attempts to control them lead to the evaluation of the intervention in an artificial setting, nothing like the setting for which the intervention was intended.
As mentioned earlier in this chapter, effectiveness research is the ultimate goal of services research, although both types of research are ultimately important. The following five-phase model (Haywood et al, 1995) serves as a basic, yet ideal, framework for conducting research: (1) hypothesis development; (2) methods/instrumentation development and refinement; (3) controlled intervention trials such as RCTs; (4) defined population studies in which subgroups of the population are studied; and (5) demonstration and implementation studies that examine the transfer of efficacious interventions to more applied settings. Studies that follow this model are studies of effectiveness.
Research Findings
A plethora of efficacy studies of psychological treatments and services (primarily psychotherapy) have been carried out in both the adult and the children’s mental health fields. Many of the individual studies have failed to demonstrate significant effects because of a number of possible causes (including methodological problems or the possibility that the treatment simply was not effective); however, meta-analytic reviews have produced “stark, dramatic patterns of evidence for the general efficacy of such treatment” (Lipsey and Wilson 1993, p. 1182). Several meta-analyses have indicated that children and adolescents who receive some form of mental health intervention function better than those in control conditions. Lipsey and Wilson concluded that “well developed psychological, educational, and behavioral treatments generally have meaningful effects on the intended outcome variables” (1993, p. 1199). Lambert and Bergin (1994) concluded that psychotherapy was effective for adults, more so than placebo interventions, and that gains were maintained for extended periods. Finally, many psychopharmacologic studies have demonstrated efficacy under controlled conditions.
Efficacy research of psychological interventions compares favorably to similar research in general health care. In 1985, the Office of Technology Assessment estimated that only 10–20% of medical practices had demonstrated medical efficacy by RCTs. Moreover, treatment effect sizes for psychological interventions are generally as large as many found in general health care.
Relative to the number of efficacy studies, few studies have examined the effectiveness of mental health interventions or services, particularly in regard to children. More investigations of this type are greatly needed. The effectiveness research that has been conducted has focused primarily on system-level variables (eg, administrative and financing structures, service coordination, access, utilization) with little investigation of clinical outcome and specific mental health treatments. According to Henggeler et al (1994, p. 230)
the dearth of clinical outcome data in the services research literature raises the frightening possibility that evidence of increased access and variety of services may be construed as a proxy for quality and effectiveness of clinical services rendered.
The majority of studies that have examined the effectiveness of mental health services have not demonstrated a general link between the intervention studied and improved clinical outcome for children or for adults. For example, the Fort Bragg Evaluation Project found that children in a continuum of care had increased access to the appropriate services compared to children at the control sites who received traditional inpatient and outpatient services only. Both the experimental and control groups improved but equally as much: The clinical and family outcomes did not differ between the two groups. The findings of the Fort Bragg study were replicated in a randomized effectiveness study of publicly funded children and adolescents. A multisite Robert Wood Johnson Foundation project did not find a relationship between system changes and clinical outcome for the seriously mentally ill adults who were served. However, other researchers have obtained promising results demonstrating the potential effectiveness and cost impact of assertive community treatment for adults with severe emotional illness and multisystemic therapy for adolescents with serious emotional disturbance.
The 39 studies included in Speer and Newman’s (1996) review of outcome evaluations of community support services for adults with persistent and severe mental illness varied tremendously in research designs, patient groups, and outcome measures. According to the reviewers,"the findings of these studies should in no way be interpreted as reflecting the effectiveness of community-based mental health services in general” (p. 123). However, approximately half of the studies had positive or mixed findings (ie, positive findings on at least one measure, no or negative results on others). Seven studies reported nodifference effects, and four reported negative effects for the intervention. The authors asserted that (1) many of the studies had inadequate power to detect treatment effects due to insufficient sample sizes, and (2) many suffered from follow-up patient attrition that can bias outcome evaluation results, but (3) research design weakness did not seem to play a part in null findings. The reviewers emphasized the critical need for clinicians to collect outcome measures in order to develop knowledge regarding intervention effectiveness that will contribute to policy-making decisions in the future.
Some researchers have proposed potential explanations that might account for the differences in findings between efficacy and effectiveness studies. Similarly, several other researchers have discussed possible methodological reasons for failing to detect a statistically significant impact of services. One possible reason for the difference in clinical findings between efficacy and effectiveness studies is the yet untested belief that practitioners are not using treatments that have been found to be effective. Because of this strong possibility, professional societies are developing treatment guidelines that reflect the findings from these efficacy studies. However, it remains to be determined whether these techniques can be transferred to the real world and whether practitioners will adopt them.
More attention needs to be devoted to conducting well-designed effectiveness studies that have methodological advantages rather than obstacles to overcome (eg, small sample sizes, poor theoretical framework, inappropriate instrumentation) in order to bridge the gap between positive efficacy and null effectiveness findings. Research is needed that examines the components or specific processes within an intervention in order to determine those that are the most effective and under what circumstances.
Quality
Introduction
The quality of health and mental health care has received renewed attention since the mid-1980s. There has been increasing emphasis on accountability and routine monitoring of access to and the quality of health and mental health care delivered by clinicians and health plans. Many of the factors discussed earlier in this chapter—skyrocketing costs, changes in reimbursement strategies, including capitation and prospective payment systems, and recent health care reform efforts—have contributed to a renewed surge of concern about the quality of services.
The focus on quality of care is not new. Early emphasis on quality, beginning as early as the 1900s, was primarily self-imposed by professional organizations in regard to the education of physicians or hospital performance. More recently, emphasis on quality has buttressed the marketing of hospital services as well as the development and dissemination of practice guidelines in order to shape physician behavior. Current efforts have focused on making information about quality of health care more user friendly and accessible to patients. This latest trend emphasizes influencing the quality of services through market competition rather than by externally imposed standards and criteria (eg, practice guidelines, external peer or utilization review).
The advent and growth of managed mental health care has caused concern about the resulting quality of services, particularly if reduction in health care costs is a prime focus. Although most of the evidence for equal or improved quality in managed care versus fee-for-service care comes from general health care, there is reason to hypothesize that managed care will lead to better quality in mental health services. For example, managed care organizations usually offer a fuller array of services—including case management and wrap-around services—that could theoretically lead to benefits such as increased continuity of care. Further, it has been predicted that the cost of mental health services will eventually level off, thereby forcing managed care companies to compete on the quality of services alone. However, others have expressed much concern about the quality of managed mental health services.
Currently, several related, more specific issues have sparked the attention of researchers. These issues include: (1) the definition and conceptualization of quality, (2) stakeholder evaluation of quality, and (3) the validation of indicators of quality.
Definition & Conceptualization of Quality
The manner in which quality is defined has always been a source of debate. Yet the definition of quality is critical in that it leads directly to the assessment of and efforts to improve quality of care. Both the conceptualization and the measurement of quality should be clinically relevant as well as useful for stakeholders including patients, policy makers, researchers, and program evaluators.
Numerous definitions of quality have been offered. Much of the discussion of quality in the field today revolves around Donabedian’s tripartite model of quality—structure, process, and outcome—as well as his distinction between technical and interpersonal care. Similar in many respects to definitions offered by others, Menninger (1977, p. 476) defined quality as:
the goodness of fit between the problem requiring therapeutic attention; the desired outcome (the goal or purpose of treatment); the treatment used, as sensed or experienced by the patient, as judged by the physician and his colleagues, and as verified by outcome studies.
Definitions of quality generally portray it as consisting of multiple domains or components. Most definitions emphasize quality in terms of outcome. That is, the criterion for determining quality is the extent to which care improves desired outcomes. Moreover, many definitions address process, although indirectly. Process is the “black box,” so to speak, detailing the components of a specific intervention or service.
Typically, the inconsistencies between various definitions of quality arise primarily as a result of differences in level of concern (eg, patient-level versus systems-level) and perspective (eg, patient, clinician, insurer). Additionally, definitions of quality may be invoked for a variety of purposes (eg, research, evaluation of program effectiveness, accountability to funding agencies, internal process reviews) and thus may emphasize different aspects of structure, process, and outcome.
Stakeholder Evaluation of Quality
With the increasing push to control the quality of health services through market competition, more recent research has focused on assessing quality of care from the viewpoint of different stakeholders, particularly consumers of health services, rather than from the traditional clinician viewpoint. Stakeholders should be broadly defined to include all those whose lives are affected by programs, services, and the evaluation of those services. Thus individuals who have a stake in mental health services include (but are not limited to) the following: (1) policymakers, at federal, state, and local levels; (2) practitioners, acknowledging differences among professionals in educational training and contact with patients; (3) administrators, including managed care professionals, upper-level managers at care facilities, and purchasers; and (4) patients, again including and acknowledging potential differences in perceptions among patients, their family members, and members of the general public who are not currently receiving services. Following the lead of researchers in evaluation and systems theorists, the best picture of quality may be gleaned from the input of multiple stakeholders, each experiencing different parts of the service system. Input from multiple stakeholders in the evaluation of quality increases the chance that the ultimate goal of improving the quality of care will be realized by increasing the validity, fairness, range, and responsiveness of the information upon which program decisions are based.
Research typically has not included all relevant stakeholders in the study of quality of care and thus has not incorporated a true systems framework. The vast majority of efforts examine only one or two perspectives simultaneously, usually the views of patients and physicians, and exclude other groups such as policy makers, purchasers, administrators, and the general public (ie, non–service users). Most research has addressed quality of health and mental health care indirectly by exploring stakeholders’ views of physician or clinic performance or by investigating patient satisfaction.
Recent research has explored patients’ desire for, understanding of, and use of disseminated information on quality. In 1994, a multistakeholder conference including patients, families, clinicians, managed care representatives, and insurers among others was convened by the National Alliance for the Mentally Ill (NAMI), John Hopkins University, and NIMH. More theoretically based work is beginning to appear in the literature. This work promotes a multiperspective view of service quality, but much work remains to be done. The Fort Bragg Evaluation Project used a formalized brainstorming technique, called concept mapping, to allow various stakeholder groups to define the key elements of quality in that mental health service delivery system. Key elements were given operational definitions and measured by various stakeholders as part of the assessment of the quality of services delivered within the program.
Validation of Indications/ Indicators of Quality
In the history of quality assessment, most research has focused on the process of care, particularly the technical rather than interpersonal aspects of care. The pendulum has recently swung heavily toward outcome while generally remaining focused on technical aspects of care. Quality improvement techniques have included peer review, accreditation, patient- oriented report cards, and practice standards or guidelines. However, the indicators of quality have not, for the most part, been validated empirically. In other words, traditional research methodology has not been used to demonstrate the necessary linkages between structural or process indicators of quality and desired outcomes, thus demonstrating criterion-related validity. Furthermore, assessments of quality often have inadequate content validity because too few indicators have been used to cover the construct of quality. For example, mortality or readmission rates are taken to represent quality, even though these measures are quite limited and represent only a small piece of the quality picture. The field is in dire need of more research aimed at establishing the reliability and validity of indicators thought to measure quality of care.
