Running late with the deadline for your work? Then we are your reliable assistant in paper help.
Get ready to ask for our assistance when you need essays, research or course works, reports, case studies, etc. Our experts have seen it all and are ready to start working on your assignment right away. Go for it!
With over 6 years of experience in the custom writing service, our team of support agents, managers, editors and writers has got a lot of knowledge about everything that may be required by you. Here’s what you get for sure when cooperating with us:
Everyone needs some paper help from time to time, because we are only human.
Our prices start at $10 per page for works completed from scratch and from only $6 per page you need to be edited and proofread.
What factors influence the cost of our paper writing services? There are 5 of them:
You’re a lucky client! Why? Because you never pay for everything. You have lots of freebies to go with every single assignment. They are:
Asking for our paper writing help, you don’t only pay us. We also pay you! You can receive up to 15% bonuses back and even earn money with our referral program.
We understand that sometimes you may want your deeds to go unknown. That is why we guarantee your complete privacy and security with our paper help writing service. After registration, you receive a unique ID and that is the only thing along with your instructions visible to our experts. Only our support team will see all the details you provide to be able to contact you in case any questions arise and send you a happy birthday discount on your special day.
Our custom writing service is completely ethical and provides busy students with great resources for their assignments. In the modern world when we need to do a lot of things at the same time, it’s nice to know you can count on someone for back up. We are always here to create the needed sample or perfect your work through editing/proofreading or explain the solutions to any problems you may have. Find out how much more free time you can get with our writing help.
Globe Institute of Technology, Manhattan wiifm presentation topics lietuvos ryto televizija reporters notebook Ursulines Suffolk County, entertainment after ww1 essay reportaje padres hooligans definition 119th Street, West zip 10027 chiropractic report of findings software as a service Columbia New York State School of Industrial and Labor Relations (ILR) personal narrative powerpoint middle school Convent Avenue zip 10027. Putnam carbon dioxide global warming essay examples 97th Street, East zip 10029 essay komunikasi interpersonal John Jay College of Criminal Justice, Midtown Manhattan thEsiS TERminOLoGy looking for someone to make report on mother.
- Good morning, everyone. I'm Mark McClellan, the director of the Duke-Margolis Center for Health Policy and I'm very pleased to welcome all of you here today, those of you who are joining us in person and who are joining by webcast to this public meeting on strategies for promoting the safe use and appropriate prescribing of prescription opioids. This is a event being hosted by the Duke-Margolis Center for Health Policy and supported by cooperative agreement with the FDA. We are very pleased to have this timely and important discussion on a critical public health issue. As everyone here knows, increased prescribing of opioid analgesics over the last couple of decades has contributed to a growing crisis of opioid misuse, addiction, and overdose in the United States. By some estimates, there are more than 60,000 opioid overdose deaths last year alone and the economic cost in addition to the cost in lives has been rising at a rapid pace. Many patients experiencing opioid-related harms may have initially been exposed to opioids through a prescription for the treatment of acute or chronic pain. And in order to address this crisis while also preserving access to appropriate pain treatment for the millions of Americans suffering from pain, we are attempting to support the implementation of effective strategies to ensure that opioids are safely and appropriately prescribed. Some of these strategies include prescription guidelines, Prescription Drug Monitoring Programs, screening and risk assessment tools and other interventions designed to change prescriber behavior, to manage access to prescribed opioid analgesics, and to improve patient care. We've released in conjunction with this event an extensive landscape analysis that reviews some of the many steps being undertaken by policy makers, health system leaders, and healthcare payers, as well as others with deep concerns about this crisis to address this problem through new strategies. Early indications are that these strategies are contributing to reduced overall opioid prescribing as well as certain prescribing practices addressing prescribing practices that are associated with an increased risk of harm. These are important public health developments, they've been welcomed by many stakeholders as a necessary culture shift to stem the tide of new opioid users and to mitigate risks for patients who are currently on opioid therapy, but at the same time, there are also concerns that efforts to reduce prescribing will result in stigmatization, barriers to appropriate treatment for both acute and chronic pain, and other adverse consequences for all of those who are prescribed opioids for the treatment of pain or suffering with unmanaged chronic pain today. Assessing the impact of all of these strategies on patient health outcomes and on public health is critical to implementing these strategies effectively. And of course, strategies to support safe and appropriate prescribing are only one part of a broader public health response to the opioid crisis that includes evidence-based prevention, improved access to treatment for substance abuse disorders, and overdose prevention. To understand how safe prescribing practices and strategies are being adopted in this fast-moving environment as well as what is known about their impact on reducing opioid-related harms, the Duke-Margolis Center for Health Policy is pleased to hold this meeting today with conversation, with expert presentations, perspectives and discussion around understanding where we are and where we need to go from here on addressing the opioid crisis. Among the topics today are the landscape of health system and payer strategies to promote safe and appropriate prescribing of opioids as well as how data and health information technology tools are being harnessed combat unsafe prescribing. We're going to review how health system strategies are being implemented and barriers and challenges to their adoption, in addition to any potential unintended consequences, and finally, critical to success of all of these strategies is evidence, evidence on what's working. We wanna have a discussion of how to improve the evidence base for supporting and refining opioid strategies and how we can measure and define success and know when we're achieving it, and know what more we need to do going forward. So we've got a lot of ground to cover here today and we're looking forward to having a robust discussion on all of these key issues. So the overview for the day is that we're gonna start off, we're very honored to have Commissioner Scott Gottlieb with us who will kick off with some opening comments on this important topic. Following the Commissioner's keynote address, we'll have the FDA's Judy Staffa present an overview on targeting key outcomes for measuring the impact of opioid prescribing interventions. We'll then move into our first session moderated by Larry Greenblatt from Duke University School of Medicine, which will cover establishing guidelines and defining success for safe and effective and appropriate opioids prescribing. And after that, we'll take a short break. Coming back from the break, after a little bit after 11, we'll start session two, which will be moderated by Regina LaBelle, who's visiting fellow at the Duke-Margolis Center. We'll talk there about how to improve provider decision making tools to support safe and appropriate opioid prescribing. We'll then break for lunch. We'll reconvene for lunch at 1:30 for session three, which will be moderated by me to discuss provider and health system approaches to manage opioid access and improve patient safety. And then after another short break, Greg Daniel will moderate the last section to discuss payer and prescription benefit management approaches to managing opioid access and improving patient safety. Wanna remind everyone that this is a public meeting and as I mentioned, we're webcasting online. The record of the webcast will be available on the Duke-Margolis website after the event. For the speakers, Isha Sharma and Nick Fiore, in front, right here, will help us keep on schedule. They have signs that'll tell you how long you got to speak. And for everybody here and those of you online, we're planning to have a few minutes for questions and answers and brief comments at the ends of the sessions. We have microphones set up in the aisles. We ask you to come up to the mics and speak right into them, let us know who you are when you ask your question or make your brief comment. And you can also join the discussion online at #SafeRX and #OpioidCrisis. We hope to hear from many of you about these important issues today as well. There are coffee and beverages outside the room. Lunch will be on your own. If you have any questions about the many restaurants in the areas, the people at our desk outside can help give you some tips, and you should feel free to bring food back here if you don't have time to finish it while we're out for the lunch break. Okay, so that's an overview of the day and that's the logistics for the event. So to kick us off, I'm very pleased to introduce Dr. Scott Gottlieb, the Commissioner of the FDA. Dr. Gottlieb brings a unique combination of experiences and skills to lead the agency at this time, from both extensive experience at the FDA and in public service and in the private sector. Since taking over at FDA, it has definitely not been business as usual and that's particularly true when it comes to the opioid crisis. Commissioner Gottlieb has declared addressing this epidemic to be a highest order public health priority, and he's carried out a tremendous amount of initiatives already to address it. The FDA's engaged in a range of activities to support safe and appropriate prescribing. You'll hear about many of them today. These include promoting the use of abuse-deterrent formulations of drugs, improving safety labeling, strengthening requirements for post-market studies, risk evaluation and mitigation strategy efforts, and a range of other initiatives to develop better alternatives to the risks of opioids and to improve the risk-benefit framework for all the people in America who are experiencing pain. So we look forward to hearing from the Commissioner about how these efforts are going and perhaps some new or further steps to improve the safety and appropriateness of opioid prescribing and to kick us off for the day. Commissioner, over to you. (audience applauds) - Think you you took my comments, too. (laughs) No. (audience laughs) Thanks a lot, I appreciate the introduction, Mark. Thanks for having me here today, and thanks for hosting this meeting. I know a lot of the leadership from FDA's gonna be here today. Doug Throckmorton who's led these efforts for many years at FDA, many others at FDA who work hard on these issues for many years and have been dedicated to trying to address this crisis in a multitude of ways. I wanna start today by reflecting on one of the great triumphs of the modern, or the battles we won in the wars against pathogens. These victories were achieved through the introduction of antibiotics and vaccines, as many of you know. And our triumphs against pathogens were so swift and complete that by 1969, the Surgeon General said it was time to close the book on infectious disease. But the cunning of the pathogens and the power of natural selection was underestimated. The bugs adapted to our innovations and the nature of many diseases continued to evolve and evade our efforts. We're in the midst of a new epidemic of opioid addiction, and like the epidemics caused by pathogens, this one is also evolving, finding ways to evade our policy interventions and staying one step ahead of our efforts. And so what began as an epidemic by short-acting prescription opioids evolved as new and more powerful and longer-acting formulations of those drugs came onto the market. People went from swallowing pills to crushing them and snorting or injecting the ingredients. And the epidemic involves still further, as addicts moved on to low-cost alternatives, which are increasingly street drugs. So even as prescribing of opioid drugs has begun to fall, in part because doctors are taking heed of some of the risks of over-prescribing, opioid-related deaths are on the rise. And the addiction crisis has evolved to street drugs that are often laced with lethal doses of super-potent formulations of counterfeit fentanyl. And so like a deadly pathogen, just when we've advanced policy to address one aspect of this crisis, it mutates into a new and more deadly form. The latest data shows that drug overdose deaths in the U.S. climbed about 21% between 2015 and 2016 from a record high of more than 52,000 deaths to nearly 64,000. About two-thirds of those overdoses were linked to opioids, a crisis that began with prescription drugs has evolved into a much more insidious threat involving illicit fentanyl. In the early days of the crisis, before we had a clear picture of how bad it'd be, many people often characterized opioid addiction as a problem of non-medical abuse, but the evidence always suggested that addiction occurring in the both the medical and non-medical users, rather than abuse alone, was a key driver of the epidemic, we were just slow to see it. As a nation, we did eventually engage more vigorously, but we were late to treat opioid addiction like an epidemic, we were late to reach for more aggressive interventions, and now the addiction crisis is burgeoning out of control, and we find ourselves at a point where our interventions are being outmatched by its constant evolution. Addressing this crisis will involve the concerted efforts of many stakeholders, public and private, local and federal. And FDA's potential to impact this crisis has been regrettably diminished over time. Not because our tools are less relevant, rather because the scope of the epidemic has grown so much larger than the range of our purview. But at FDA, we still have a very important role to play. This is especially true when it comes to helping make sure that fewer new people become addicted through the medical use of these drugs. We have an important role to play in supporting the development of improved medication-assisted treatments for opioid dependents, which will expand treatment options for those currently addicted to opioids. We also have a role to play in helping support the development of new, safe, and effective treatments for pain that don't carry all the same risks as opioid medicines. And we have a role to play in interdicting illicit drugs that are pouring into our country, often through the mail and often through international mail facilities. I wanna focus my remarks today on the first challenge, the steps we're taking to reduce the incidents of new addiction that's derived from the medical use of prescription opioids. And some of the new steps we'll be taking to address this challenge. We know that a lot of the new addiction is still the result of medical use of opioid medicines. Some proportion of those who are addicted are patients who were first exposed to opioids through a lawful prescription. So a key to reducing the rate of new addiction is to rationalize prescribing. To make sure that people are prescribed opioids only when it's medically indicated, and when a prescription is written, it's for a dose and duration of use that comports closely with the clinical purpose of the prescription. There's a lot we're already doing to address this challenge. We recently expanded our risk management plan to incorporate all opioid drugs that are intended for use in the outpatient setting, including the immediate-release formulations for the first time. We've redrafted our blueprint for how providers should be educated about prescribing, and we're requiring this training to be extended to all providers who are likely to come into contact with patients who are prescribed these medicines, including nurses and pharmacists. FDA is also taking immediate action where needed, as we did with FDA's first-of-its-kind request to remove a marketed opioid pain drug from sale due to the public health consequences associated with the product's abuse and misuse. On the issue of illegal narcotics like fentanyl that are coming into the U.S. through mail facilities, we tripled the numbers of inspectors we have at international mail facilities, allowing us to quadruple the number of suspicious packages that we're able to open and inspect. We sharply increased the number of agents we have working on the dark web, looking for sources of fentanyl, and we're looking for ways to build out our footprint in IMS still further so we can stop and inspect and test more packages that contain suspicious drugs. We still have a relatively small footprint inside these facilities. And get more resources to trace illegal narcotics back to their source to bring action against their purveyors. Under leadership of the Opioid Policy Steering Committee, which I formed when I arrived at FDA last May, we're also reevaluating how marketed drugs are used, both for legitimate purposes and misuse and abuse. In particular, we're looking closely at certain opioids that may have higher abuse potential, like oxymorphone. And we find ourselves in some unlikely places as this crisis takes on a new dimension. Who would have thought and predicted just a short time ago that we'd have to take regulatory action to prevent the abuse of loperamide? And who would have thought that kratom would become a new gateway to addiction, abused for its opioid properties? The opioid crisis is taking many new and unpredictable turns, and our experience demonstrates that we need to be vigilant and aggressive at the outset as these new threats emerge. Our experience has shown that if new problems fester, they can quickly spiral out of control. And there are some new concerns that we have. One relates to the possible risk of misuse and abuse of gabapentinoids. Gabapentinoids are FDA-approved drugs that are not opioids and have been approved to treat a variety of conditions, including postherpetic neuralgia, fibromyalgia, partial onset seizures, neuropathic pain associated with diabetes and spinal cord injuries. Some literature suggests that clinicians may be prescribing these drugs off-label to patients with various types of pains as alternatives to opioids outside the approved indication for these drugs. Our preliminary findings show that abuse of gabapentinoids doesn't yet appear widespread, but use continues to increase, especially for gabapentin, and FDA's investing whether their abuse or misuse is also increasing, and if so, what should be done to address this problem. Although limited, the data suggests that gabapentinoid misuse and abuse may be growing, both when taken alone and taken in combination with opioids, benzodiazepines, or other central nervous system depressants. We're concerned that abuse and misuse of these drugs may result in serious adverse events such as respiratory depression and death, and we wanna understand changes in how patients are using these medicines. So among other steps that we're taking, we've looked at social media websites where opioid users share comments that describe methods and motivations for misusing or abusing gabapentinoids. And we've tasked our surveillance and epidemiology group inside FDA, who are focused on spotting early patterns of abuse of controlled substances, with investigating the use patterns of the gabapentinoids. We'll have more to say about our work on this challenge soon. Our swift attention to this matter is, in part, a consequence to the lesson of history. We know we need to investigate and respond to signs of abuse as soon as signals emerge, and we need to get ahead of these problems. And getting ahead of these new threats, or addressing those crises that are already fully manifest will require some forceful actions. The scope of this epidemic demands that we consider more potent steps. One action that we're examining is the blister packaging of the immediate release formulations of opioid drugs. I believe this merits consideration through a careful, science-based process. We need to ask: Would requiring the IR opioids to be packaged in unit-dosed blister packs with a number of pills contained in a pack comported with appropriate durations of use, say three or five or seven-day supplies, help ensure that providers dispense only the amount of drug needed for common, short-term uses for which these medicines are often intended? In December, the FDA hosted a public workshop on the role of packaging, storage, and disposal options within the larger landscape of activities aimed at addressing abuse, misuse, or inappropriate access to prescription opioids. New types of packaging, rather than a typical bottle with a 30-day supply, could remind prescribers to restrict the number of opioids dispensed to patients, in turn reducing the total amount of opioids. It could also reduce inappropriate access to opioids by kids. This would also help address the problem of excess supply in the U.S. Packaged in this fashion, the belief is more doctors may opt for for these shorter durations of use. This would reduce overall exposure and the rate of new addiction. Moreover, provided FDA concluded that there was sufficient scientific support for these shorter durations of use, this could provide the basis for further regulatory action to drive more appropriate prescribing. And so we're actively exploring our authorities in this area and also looking at how we can work voluntarily with the supply chain of manufacturers to get the IR opioids into blister packs. We've used packaging to address safety concerns in the past and most recently we requested that the manufacturers of loperamide make packaging changes, such as using blister packs, to address and the abuse and misuse of that OTC product. I'm certain there'll be more to come on this topic, and we're taking an all-of-the-above approach in our approach, in our efforts to combat this immense public health emergency. All reasonable steps are on the table. With millions of Americans misusing prescription opioids and more than 40 million people, 40 people dying every day from overdoses involving opioid medications, it's become abundantly clear that we need to do everything we can along with our partners to get ahead of this crisis. And so we appreciate your input here today. I appreciate the efforts of the many people from FDA who are here to share information with you and work with on you on these challenges. We'll continue to seek the feedback we can get from a broad group of stakeholders and explore a range of approaches to tackle this epidemic and ensure that patients with pain have access to proper treatment. Thanks a lot. (audience applauds) Um, I can take some. Do I have time? - Um, yeah. Thanks Commissioner, we have a few minutes for questions if anybody has one, please go to the microphone like I mentioned before. While that's happening, maybe I can start out with one: You framed this as an epidemic that we need to get ahead of, and I appreciate the analogies before, and you talk about some of the new steps that you're taking, there have also been new developments with financial support, budget processes and the like, to create new resources to address these challenges. What are the implications for the increase in the FDA budget that's expected, some of the recent Congressional legislation for what FDA can do on these issues? - So we've been having discussions with Capitol Hill. As most folks know, we've got a plus up in our budget this year, a lot of the money's targeted towards new initiatives to try to improve the landscape for medical product innovation, but there's also gonna be resources allocated to helping FDA, and other agencies for that matter, better support their efforts around the opioid crisis. We have a number of initiatives where we've been discussing, I will say that I think our highest priority, with respect to the opioid crisis right now, if I were to prioritize the allocation of additional resources with respect to FDA, would be putting additional resources into the international mail facilities. I mentioned that when I got to FDA, using resources I found within the agency I was able to triple the number of people we had in IMFs doing interdiction work, which allowed us to quadruple the number of packages we opened and inspected. But even in doing that, I increased the number of people in, I think it's six IMFs, from seven to 23, and increased the number of packages we're able to open and inspect and test and trace from 10,000 a year to 40,000. So it's a small fraction, a very small fraction, of what we're seeing. So we have a proposal that would increase our footprint in those facilities. Customs and Border Protection plays a primary role, along with the United States Postal Service, but we also play an important role. We have a lot of expertise in testing synthetic analogs that come through those facilities, figuring out what we're looking at. And doing some of the work on the dark web and some of the work internationally to try to trace these products back to their source. And so, we have a very critical investigative role to play and so having a slightly bigger footprint in those facilities, I think, could allow us to do much more interdiction work and stop the flow of drugs coming into the U.S. - Thank you. Over here. - [Linda] Good morning, Linda Streitfeld from CMS. Two questions about the blister packs. First, I wonder, you mention scientific evidence, and I wonder what evidence we have that blister packs, which reduce flexibility in prescribing, would actually reduce the number of opioids prescribed? And the second question, you talked about safety for kids, and I'm not sure what age children you're referring to. I don't know if we have a significant problem of young children accidentally ingesting opioids and that's what you're referring to, or somehow blister packs keeping older kids from trying them. - It was the former, it was inadvertent access by young children as ways to secure the blister packs as you know, and there's different kinds of blister packing. Some are more secure and tamper-proof than others. But I think the short answer is we still have work to do to try to establish the public health case for how having drugs dispensed in blister packs can drive more appropriate prescribing. What you'd wanna do is have evidence that there are breaks points in what the appropriate dose is for different indications, and if there are some common doses for some common indications for which the IR drugs are prescribed, then it might make sense to have them available in packs that would become an easier option for physicians rather than the sort of default being to write for a 30-day supply, like doctors write for drugs that are dispensed in packs like a prednisone pack, it becomes a default option, it can just become more, an easier option for physicians who commonly prescribe for two days, three days, five days, seven days, would pick your duration, but when you look at the data, and I've looked at some of the data that we have that we've derived from our Sentinel system, you do see breakpoints in the duration of use by patients by indication. I think we could work with the provider groups to try to establish some of these measures. If we do move eventually to a mandatory, a system of mandatory education, you can envision a framework in this country where doctors who wanted to prescribe longer duration of use because they treat a lot of patients with chronic pain or patients with metastatic cancer pain, might have to go through the mandatory education, other doctors would be defaulted into the shorter duration blister packs. You'd almost have a two-tier prescribing system where doctors could opt to prescribe for 30 days, but they might have to go through some additional steps. Other doctors who commonly prescribe for conditions for which a three-day dispense is appropriate would probably opt not to go through the hoops and default into the three-day pack. Who might that be? It might be dentists, it might be other doctors who practice general surgery and do minor procedures that require just a short duration of use. The reality is, when you look at the data, you still see 30-day doses being dispensed for those indications. I'm not sure anyone here can defend that. - And reviewing all that kind of evidence is definitely the focus for today. We have time for one more quick question. - [Cindy] Thank you, Cindy Demark from Delaware Medicaid, I also had question and concern regarding the blister pack. I'm worried about landfill, I'm worried about the increase in cost, so my question to you would be: Have you looked at using e-prescribing as a tool for dropdown for days' supply? Might be a little more expedient if that was-- - I think that there's a lot of, I think there's a lot of virtue in looking at a national e-prescribing system, there's a legislative proposal right now in the House that looks at that within the context of Part D, we might look at it outside of Part D. I think having intra-operatable state systems as well makes a lot of sense. These are discussions that I think have to happen with Congress, some of these would require new legislative authority. Congress can also help facilitate the issues around the blister packing as well and delineate a proper framework for doing that. But in the interim, we're gonna be looking at, looking at these options, trying to develop the evidence base to inform a proper discussion. - And you mentioned a national e-prescribing system built off of the e-prescribing capabilities now. We have a lot of health systems, health plans, others, represented here today who are talking about their ideas and the evidence to support them as well. Any advice for that part of the response to the opioid crisis, what health systems and health plans and the like can do? - I think if we had e-prescribing, and it would just be, you would talk about it in the context of controlled substances, and I think we're moving towards this anyway in this country, we're gonna have e-prescribing. It would also allow us to think differently about our REMS program 'cause we could, instead of trying to mandate interventions in the system, we could just require certain dropdowns and certain prompts associated with certain drugs, and I think we could partner much more effectively with the health system. I think from the standpoint of the health plans and the systems that are being created in the market, one of the challenges that I see, I think you probably share this, is lack of intra-operatable systems. And so, you talk about these PDMP programs in the states and you can't look across state lines, and so if you're sort of a border physician, you can't see what a patient might be doing within another state. And so I think it's very important that as we build these systems to try to address aspects of this crisis, we sort of build in upfront how we're gonna think about creating these systems in an intra-operatable fashion so we're not stuck where we have been with the EMRs where nothing can talk to each other. We're sort of creating a new system now, we should think about some of the lessons from the past right up front. - Commissioner, thanks very much for joining us this morning, thanks a lot. - Thank you. (audience applauds) - All right, and again, we have a very tight schedule today. To move us along, I'm very pleased now to introduce Judy Staffa, the Associate Director for Public Health Initiatives at the Offices of Surveillance and Epidemiology at the Center for Drug Evaluation Research at FDA. She will be discussing some of the work that has been undertaken by FDA to understand how we can think about assessing the impact of prescribing limits, many of the other kinds of interventions that you've already heard about from the Commissioner this morning, as well as how we can develop best practices for program evaluations to inform future efforts. So, this is gonna focus on outcomes for measuring the impact of interventions, what is going on now, and what else FDA might be looking for. Judy, very pleased to have you with us today. (audience applauds) - Thank you. Okay. Well I'd like to thank Duke-Margolis for giving me a chance to speak today. It's actually working out very well to come out after Dr. Gottlieb because his creation of the Opioid Steering Committee. Our group has a lot of talented scientists that have been working very hard to actually try to establish the evidence base behind strategies relating to things like prescribing limits, and things like packaging solutions, and so a lot of what we're doing lately has been focusing on those two areas, and so it dovetails very nicely with the work that Duke-Margolis has been doing, which is of course much broader, looking at the whole landscape of all different kinds of strategies. So, what I wanted to try to do, in 15 minutes, is to highlight some of the key issues that we've been running into as we look at these strategies and the evidence behind them. What I'm hoping is, by looking at this small piece around prescribing limits, that the kinds of things we've learned will help inform the discussions today because I think the things we're finding with prescribing limits will be very relevant to some of the other strategies as well. So I'm hoping this is gonna, there we go. So, to start out with, obviously everyone I think has probably memorized this graph, this is tragic and scary and this is why we're all here. So in terms of trying to move ahead with strategies to reduce prescribing, I don't think, even though we have no evidence, we can't wait for it. So clearly we've had to move ahead and many in the room today have championed those efforts, and the CDC has also put out guidelines around prescribing limits. Because without evidence, the best we can do is put together consensus of experts and guidelines, and CDC's very clear about that. So it's a place to start. But I think this is what drives us. So who's doing things out there? I think federal agencies have put forward prescribing limit guidelines, different insurers, third parties, localities like New York City, different states, and professional societies. CDC was among the first, they issued their guideline in March of 2016, so that one hasn't really been out there very long. So we went to the literature to try to understand what evidence is there around all of these different guidelines around prescribing limits that we could learn from? There's not much around the implementation of CDC guidelines yet because they're new, but that's being worked on. However, what we did find was a lot of these other efforts preceded CDC's guidelines, and so there is some evidence in the published literature. So what are people doing? Well we found it again, I know that the landscape analysis from Duke-Margolis goes much beyond the published literature, and that's why it adds so much value, and we'll hear more today. But in the literature what we found was a number of efforts, 29 published papers specifically, that looked at different ways to limit prescribing. Some of them were based on days' supply, some of them based on pill counts, and quite a few based on daily dose, relating to morphine milligram equivalents. Many of them used what Washington State started with, which was 120 morphine milliequivalents or less per day. So how do we evaluate these strategies, what are we looking for? What are the outcomes we should be focusing on? So what did these evaluations look at largely? Well, one outcome that's looked at a lot, and again it's probably the most straightforward thing to look at, is some measure of prescription volume. Whether it's prescription count, prescription rate, this is just an example of a graph from a publication from Blue Cross Blue Shield of Massachusets that looked at measuring prescribing volume, the Y axis here is actually prescriptions per member, and looked at that over time, if you can see the vertical line, that's when they instituted an intervention around prescribing them. And so you can see that things began to decline a little bit before that and then declined, visually you could probably say it declined a little faster after. But what's the issue with that? We think maybe that's kind of a blunt measure because which prescriptions are we actually trying to not have around anymore versus which prescriptions are we okay with having around? This is national data, which we watch all the time, that shows you that prescriptions in this country for opioids peaked around 2012 and then began to come down, probably for lots of reasons, lots of things going on, lots of awareness, lots of different programs. So it's not really clear that looking at this blunt measure without being able to determine which prescriptions are appropriate and which are not, we're not really sure that that's telling us the most that we need to know. And sadly, if we go back to the curve before that we looked at and we superimposed when prescribing began to drop, clearly overdose deaths are not going down. So that may not be the best measure at this point for looking and when we're trying to change prescribing practice. So what other things could we look at? As Dr. Gottlieb described, we had a public hearing recently, at the end of January, where we invited people to come and talk to us about different ways to think about prescribing limits and packaging solutions, and we learned a lot. And some of the things we learned about from some of the patient and provider groups were that there may be some unintended consequences to these kinds of strategies and we heard lots of anecdotal data or information from patients, very compelling stories, about how patients have been forced to taper or decrease their dose or do without completely, opioids, very abruptly, after many years of functioning quite well on them. And many of these patients found changes, unfavorable changes, in their disability, functioning, and tragically some of them actually commit suicide because of that. We have no data to quantify that, we're working on that, but maybe these are some outcomes we should be looking at or paying attention to in some of these programs. We're also hearing from prescribers and pharmacists that some of the extra steps and the burdens, the administrative burdens, on folks are actually causing them to turn away from prescribing opioids at all. And so maybe that's something we should also be thinking about and measuring. This is, I won't read this to you, but this is from our docket. We have dockets out there for the public hearing, for this meeting, for our packaging meeting. I would encourage folks to share experiences, share what they know. These are the kinds of anecdotes where we can learn more to actually structure our interventions better. But this tells the tragic story of somebody who was just completely shut off from opioids. So how well did these 29 published studies actually look at these outcomes? Well, looking at different patient outcomes, not very much, it really hasn't been the focus of many efforts up until now. The only one that's really been looked at is the one, the bar far to the right, that's overdoses, unintentional overdoses, which again, here's an example of one of the papers that actually did that. This was a program in Utah where they counted overdoses in patients and, well you can see, after the intervention was put in place, they went down a little bit. But then they kind of bounced a little bit back up. Again, making the point that maybe this is too distal to what we're doing, that maybe changing our prescribing practices for someone who's opioid naive in acute pain, maybe it's too far downstream to think that that could impact us. Maybe we need to be looking at the outcomes closer. We know that the epidemic has evolved, there's a lot of other things happening out there that could be impacting things. So we'd like to see in some of the discussions today, I just ask you all to think about as we think of how to evaluate some of the things we're doing in this space, prescribing limits as well as other things, can we be looking at more patient outcomes, things like quality of life and pain. I know these are gonna be very hard to measure, and they're gonna take us beyond big data and claims, and I get that, but I think the juice is gonna be worth the squeeze. I think we really need to think about and brainstorm up ways to get that, in addition to the things we already know we need to look at. What are we trying to prevent with these programs? Misuse, abuse, the onset of addiction, can we measure those things? And can we actually be very creative, and I'll point your attention to the bottom right box, can we define what an unnecessary prescription is, or an inappropriate dose, and can we actually measure that, and can we see how that metric performs over time? Not easy, but worth doing. So now I'm gonna just spend a couple minutes on a few methodology type issues we've run into that I'm hoping will come up in the discussions today. The graph on the upper left is the one I already showed you from Mass Blue Cross Blue Shield, and you can see that the prescriptions are going down over time, but if you insert a comparator group, which is what the lower right curve actually does, you can see that this is a study that actually took advantage of a prescribing limits placed on emergency room docs, that's the dark line that's going down over time, after the intervention. But you can see when they looked at orthopedic surgeons, who had no intervention, but were subject to all the other things that were going on in the health system, you can see that the prescribing went down for both, but we had a bigger drop in the group that had the intervention. So comparator groups and taking advantage of opportunities to look at them can be meaningful. Another issue is, and I think we'll talk about this today, many times there's multiple strategies that are included in one intervention. And if we can start to parse them out and determine through staggering or natural experimentation, like this study did from Staten Island where they looked at interventions in Staten Island, used other boroughs as comparators, and then looked over time at what the different pieces of that strategy were doing. That way we can target what's working, what's not working, and push our efforts forward toward the things that seem to be achieving the outcomes that we'd like. So, in conclusion, what are my takeaways? What I'm hoping we can think about today is how do we develop best practices here? And can we think about the way to, again, public health 101, as we're implementing strategies, can we build in evaluation plans up front? That's always the hallmark of a strong public health program. In addition to prescribing rates, which, I'm okay with looking at 'em, but can we build in some of these other, patient-centric, and more proximal measures to try to actually measure what we're doing and its immediate impact? And then again, including non-program comparators and taking advantages of some of the regional differences we see to try to understand different pieces of programs and how well they perform. I'd like to thank, again, Duke-Margolis for allowing me to speak and the folks behind me who are working on this stuff and doing all the real work on it, the talented folks in my office, a lot of them are here today, happy to connect you with them over the coffee urn if that's desired, to get more details. So, thank you so much. (audience applauds) - All right, thank you very much, Judy. We're now gonna transition straight into our first panel, which will be chaired by Larry Greenblatt from Duke University, I'll ask Larry and the panelists to introduce themselves and we'll get going with this discussion, thank you. (panel murmuring) - Okay. Everyone's set, well thank you, Judy, and thank you, Mark. I'm Larry Greenblatt, I'm a general internist at Duke Health, and I'm a professor in the Department of Medicine, also professor of community family medicine, and I co-chair the Opioid Safety Committee at Duke, and I've been working on this issue for years. I'm also deeply involved with the Opioid and Prescription Drug Abuse Advisory Committee, which is organized by our Department of Health and Human Services to try to tackle the opioid epidemic in our state. I'm very excited to be moderating the first panel of the day. In an effort to address the epidemic of opioid abuse and related deaths, the CDC issued their guidelines, the guideline for prescribing opioids for chronic pain in March of 2016 to offer recommendations for prescribing opioids to chronic pain patients. That guideline was initially directed at primary care physicians, although I understand it's really largely become a guideline used much more broadly, that includes my state. Building on these recommendations, many have instituted their own prescribing limits, or adapted these guidelines for local use. This has been done by states, health systems, and by payers. As Judy's noted, these guidelines have had a profound effect on prescribing behavior. We've seen this, certainly locally, in my community and across the state, based on some data that I've had access to. Through this session, we're hoping to delve a bit more into how these guidelines have been adopted across systems, what we know about how they're affecting prescriber behavior, and how they're affecting patient care and patient safety. We also want to examine how we might be able to better evaluate these outcomes and more broadly, to think about how we define success for safe prescribing interventions. So each of our panelists will be providing some brief remarks, followed by a moderated group discussion, and we're also gonna reserve some time at the end for your questions. Some of the questions that we like to dig into during our session this morning: How are the CDC guidelines and other prescribing guidelines being implemented by policy makers, health systems, and payers? How are prescribing guidelines affecting provider behavior? How is their impact on patient health and safety being measured? What are the barriers or unintended consequences of adoption of these guidelines? How can guidelines continue to be refined and improved? Especially as more data becomes available as you've heard from our other speakers this morning. And finally, how do we define success for implementing prescribing guidelines? So we're looking forward to a very interesting conversation. I'd like to introduce our panel, first is John Halpin. John is medical officer, Opioid Overdose, Epidemiology, and Surveillance Team at the National Center for Injury Prevention and Control at the CDC. Seated next to him is Gary Franklin who is our research professor in the Department of Environmental and Occupational Health Sciences at the Department of Medicine at the University of Washington. Erin Krebs is Associate Professor of Medicine at the University of Minnesota and health services researcher at the Minneapolis VA Center for Chronic Disease Outcomes Research. Lewis Nelson is Professor and Chair of the Department of Medicine and Chief of Medical Toxicology at the Rutgers New Jersey Medical School in Newark. And Myra Christopher is the director of the Pain Action Alliance to Implement a National Strategy Project, and that is a patient advocacy organization. So I'd like to turn it over to John, to kick things off and provide us with a few introductory comments. - Very good, thanks very much. Looks like my slides have been pulled up. On March 15, 2016, the CDC finalized an opioid prescribing guideline for primary care providers for chronic pain outside of active cancer, palliative, and end-of-life care. A guideline was needed to better align opioid prescribing practices in primary care settings with the best available evidence to ensure safe, effective pain management. The guideline is not a rule, regulation, or law. The guideline does not deny access to opioid pain medication as an option for pain management. Rather, the guideline is intended to inform clinicians discussions with patients and their prescribing decisions based upon the best available evidence about benefits and risks of opioid use. Among the 12 recommendations of the guideline, three principles are key to improving patient care. First, non-opioid therapy should be preferred for chronic pain outside of end-of-life care. Second, when opioids are used, the lowest possible effective dosage should be prescribed to reduce risks of opioid use disorder and overdose, and third, providers should exercise caution when prescribing opioids and monitor all patients closely using PDMPs and other means. CDC worked tirelessly to make sure we got the guideline right. However, the success of the guideline hinges on our effective dissemination of it. To this end, we identified four priority areas to maximize uptake and use. And this includes developing communication and translation materials to make the guideline more accessible, educating providers when they are first trained or through ongoing, continuing medical education, working with health systems to integrate the guideline into clinical decision support tools and practice protocols, and leveraging insurer or pharmacy benefit manager schemes, like drug utilization and review, or prior authorization for certain types of opioids or opioid refills. Improving insurance coverage for non-pharmacological treatments is important as well. So let me go a little deeper into these four aspects. In terms of our dissemination efforts, many of you might be familiar with Atul Gawande, a surgeon, writer, and public health researcher. He is best known for books like The Checklist Manifesto, were he compellingly argues that we can achieve greater efficiency, consistency, and safety in many areas of life, including medicine, by using checklists to guide action. We worked with Dr. Gawande and his staff to create this easy-to-follow checklist to the far left of the slide, or algorithm about the CDC guideline. We also recently released a mobile app for Apple and Android devices that includes prescribing guidance. That's depicted on the far right. It also has an MME calculator and motivational interviewing tips for prescribers when talking to patients about opioids and opioid-related risks. Within education and training, CDC has launched a series of interactive online trainings for healthcare providers, shown here. There will be 10 modules total, four of them are already available, and these trainings feature the recommendations in the guideline, provide sample scenarios, feedback, and resources like conversation scripts for discussing opioid use with patients. This, and all future trainings, will be available for continuing education credit. Also, CDC partnered with CDC's Clinician Outreach and Communication Activity, or COCA, and the University of Washington to present a webinar series about the CDC guideline. Seven one-hour webinars are available for free online, available for both CME and Maintenance of Certification credit via the ABIM. In addition, many health professional schools have made the voluntary commitment to integrate the guideline into their curriculum. To date, more than 60 medical schools have announced that they will align their existing curriculum with recommendations contained in the CDC guideline. Healthcare systems have the potential to improve pain management, including safer use of opioids through guideline-concordant care on a broad scale. CDC activities to assist health systems in this work are promoting the integration of PDMPs into the electronic health record systems to improve PDMP usage and effectiveness, creating the clinical quality improvement technical package to help health systems develop processes and deploy strategies to facilitate adoption and use of recommendations, updating clinical decision supports to integrate recommendations into electronic health records. For example, alerts for MME thresholds, defaults on prescribing amounts for initiation of opioids, and prompts to check the PDMP. Private and public insurers and pharmacy benefit plan managers can foster guideline implementation through an improvement in coverage and service delivery payment models and proactive use of claims interventions, claims information. Priority interventions that can facilitate guideline implementation are described here and they include coverage for non-pharmacologic treatments such as exercise therapy and cognitive behavioral therapy, removing barriers to prescribing non-opioid pain medications, reimbursement for pain management including patient counseling, coordination of care, and reviewing of PDMP data, promoting more judicious use of high dosages of opioids using drug utilization review and prior authorizations, and increasing access to evidence-based treatments for opioid use disorder. And finally, it is critical to insurers to implement payment models that improve access to interdisciplinary, multi-modal care. So I'll end there, and I'll be open to questions at the end of the session. - [Larry] Thank you, all right, Gary. - [Gary] Do you mind if I stand up here, I can't see that slide. (Larry laughs) - [Larry] Help yourself. - Thanks very much. Most people don't realize that most healthcare delivery is regulated at the state level. The boards and commissions in every state regulate, in our state I think we have 40 or 50 different health professions. So the folks that were sort of pushing opioids back in the late '90s, got model guidelines and regulations passed in more than 20 states. And in Washington state, for example, the new regulations said no doctor shall be sanctioned for any amount of opioid written, and that was based on the false teachings that there was no ceiling on dose, that the way to treat tolerance was to keep increasing the dose. There was pain as a vital sign issue, but the key is that, even if you were handing out bags of opioids and that case and complaint came to our board, that board couldn't do anything about it with that kind of regulatory language. So it's very important to realize that states have the largest role to play, probably, in reversing this worst of man-made epidemics. 20% to 30% of patients on chronic opioids are either severely dependent or addicted, and the other new thing to realize is that there's been a huge increase in people becoming disabled on opioids and moving to the Social Security Disability system. This is a slide showing the use and expenditures on opioids and the biggest area of expenditure for the federal government on opioids is in the SSDI system and people under the age of 65 who are disabled. It's very important to realize that opioids have probably contributed to the initiation and perpetuation of disability in worker's compensation systems with the spillover effect to SSDI and other permanent disability systems. This was a study that we published in 2008 based on tracking 2,000 low back injury workers and looking at their relationship between their early opioid use and disability one year later. And we found that just getting seven days or more of an opioid or two prescriptions in the first six weeks after the injury was associated with a doubling of the risk of being disabled one year later, even if you adjusted for everything, including the severity of the injury. I reported the first deaths at Washington State in the country, and these were injured workers, people that came into our system with a low backache and then died from an unintentional overdose of opioids. They went to sleep and did not wake up. Their spouse would find them dead in bed in the morning and we would end up, of course, paying death benefits from the worker's comp system. So we began to work on guidelines in Washington state in 2006, and had done three successive guidelines in 2007, 2010, and 2015. The last guideline that we did brought in from just focusing on chronic use of opioids to preventing inappropriate acute use, sub-acute use, coming up with a case definition of what is clinically meaningful improvement in pain and function, you shouldn't keep using opioids if you're not getting at least a 30% improvement in pain and function, not just in pain, and we came up with also new guidance on tapering opioids and algorithms for how to taper, a case definition of who should be tapered. The result of all these efforts is that Washington state has had a nearly 50% decline in the deaths associated specifically with prescription opioids. But then you gotta start thinking about who's prescribing most of the opioids. If you look here, for people that are 24 and under, it's actually largely dentists. Extractions, there are 56 million tabs of hydrocodone per year prescribed. This data is from Washington state's PDMP. You can see that the average number or the median number of pills from a dentist to a 17-year-old for example, for an extraction is 20 pills, but the 75th percentiles and the 95th percentiles are up to 30 and 40 pills, it's much higher for orthopedic surgeons. So it's gonna be very important to get a hand on this acute use, on post-operative use, and how many people are becoming dependent from that kind of use. This is a very important study from the CDC that showed that the risk of long-term opioid use goes up by 1% per day, starting with the third day of the initial prescription. So Washington state has done a dental prescribing guideline that we think is the first one in the country that has been rolled out for the entire state. Two important things here is that 85% of people who have an extraction get an opioid. This says, prescribe non-opioid analgesics, this first line because that is what the randomized trial evidence shows. In addition to that, especially in people 24 and under, limit the dose of the number of pills to eight to 12 tablets. If you say three days, you can get up to two pills every four hours. That's actually a lot of pills, you have to deal with the number of pills as well as the number of days. This is an interesting study that was just published in JAMA Pediatrics that looked at the number pills left over in teenagers who had a surgical procedure. So for a T&A, 52 pills dispensed, 44 pills left over, and these other things, the majority of pills that are given to kids are left over for subsequent misuse. This slide shows that about 6% to 8% of people who have surgery go into chronic opioid use. And this is probably the most important study that has just came out a few weeks ago that looked at the incidence of dependence, abuse, or overdose in a large, commercially insured population. And what they showed is that with each refill, and with each week of opioids after a surgical procedure, there's a 44% increase in subsequent chronic opioid use and probably dependence. So Washington state has come up with a preliminary post-operative guidance that will probably end up in regulations in Washington state. It's very important to focus on this stuff. It's important also to come up with metrics. We're sending out these report cards to doctors. This is a helping hand, this is not a sledgehammer. And Washington state has also come up with specific metrics that should be used by health plans and by the state. You can use the PDMP to track these metrics, it's very important, including looking at the chances of somebody on new opioids transitioning to chronic opioids. And then finally, I think Erin's gonna say a little more about this, but collaborative care and coordinative care for pain is the most important thing that we could be encouraging and incentivizing at the state level. Thank you. - Thank you. Okay. (audience applauds) Gonna ask you told your applause till we're all done. Okay, Erin. - Hi. So first I just need to say, I'm a full-time VA employee, but I'm not here representing VA today. I'm here as an academic, general internist and health services researcher. And I'm gonna play against my normal type here, I don't have any slides, I'm not gonna talk about my research. I'm gonna provide a little perspective from primary care. And some experience from both my clinical practice and my research patients. So first I have to say that I think the CDC guideline is an incredibly good thing for primary care. It provides really useful guidance, and the recommendations, especially the ones about what we shouldn't start are incredibly important. For example, for most patients with chronic pain, just don't go there, don't start opioids. And if you are starting opioids, don't go above a certain dose. Keep the dose low, and if that's not working, try something different instead of continuing to increase that dose. What I'm gonna talk about more, though, is really unintended consequences. In terms of how that guideline and the recommendations are interpreted in a less nuanced way than they are actually presented in the guideline itself. So first I mentioned primary care, and sometimes if we talk about a U.S. crisis, we also talk about a crisis in primary care, right? Because not very many physicians are choosing this specialty anymore. And those of us who did, a lot of us are burning out early and leaving. And the truth is that often, primary care can just feel like being on a hamster wheel. You're trying to see always more patients in less time and you have more administrative burden it seems every day. So that is fundamentally a challenge that we face. Primary care is responsible for most chronic pain management. And that's actually how it should be, in my opinion. Because chronic pain management, frankly, does not require high-tech solutions, or high-paid subspecialists in most cases. As Gary mentioned, usually it's teamwork among frontline clinicians that is the best approach to chronic pain. Talking about not just primary care physicians, but physical therapists, psychologists, pharmacists, nurses, all of those folks working together as a team to provide the kind of longitudinal, patient-centered care that those of us who went into primary care, that's what we wanted, that's why we chose this specialty. But opioids have really added to our problems in primary care. I think obviously opioid management takes a lot of time, at least if it's done well. But more than that, opioid management is actually a huge source of professional dissatisfaction. Because we in primary care have really learned the hard way that opioids don't make people better. That's just the truth. And they create a lot of new problems that are incredibly to difficult to manage, and that's what I'm gonna talk about more. So some estimates are that about 5% of patients prescribed opioids are receiving long-term, high-dose opioids. Maybe that 5% is getting 50% of the total milligrams of opioids prescribed. So it's a very intense group. These patients are at huge risk of serious harms including death if they continue high-dose opioids, and they're also especially vulnerable to unintended injury. The CDC guideline recommendation is to work with these patients to taper them, but that is not at all straightforward. So as an example, I'm going to briefly tell you about a patient who came to me after taking opioids for more than 10 years following back pain. She was on about 200 morphine-equivalent milligrams a day, and stable by all accounts, adherent to therapy. But her quality of life was really poor. She's profoundly fatigued, couldn't keep up with her home responsibilities. Yet she was totally certain she would be worse off without the opioids. A lot of patients in her situation, who have no demonstrable benefit from opioids, believe they cannot live without them. They're afraid of worse pain, they've often experienced withdrawal symptoms that can involve pain, and faced with the prospect of tapering, they feel desperate and hopeless. This is what it feels like to have opioid dependence, I'm not just talking about addiction, I'm talking about the expected physiologic opioid dependence that develops. I've found getting patients on board with tapering can be harder than the tapering process itself. For this patient, it took several visits before she agreed to start tapering her dose. One small step at a time. And it took us two years to reduce her dose by 90%. But ultimately she had a big improvement in her quality of life and her pain was not worse. Okay, we didn't fix it, but it wasn't worse. And we both agreed the taper was a success. This patient is fairly typical for my practice. Most patients I see have uncomplicated opioid dependence and tapering goes well. But it frankly takes a lot of time and effort even in the most simple cases. Of course situations can be much more difficult. Another patient similar to her in terms of demographics who had been started by a pain specialist on very high doses, about 600 morphine-equivalent milligrams, she thought she needed even more because her pain was not controlled. She ran through her opioids too fast nearly every month. And no matter how short a supply I gave her, the same problem kept happening. So it became obvious to me that this was someone with a more complex opioid dependence, an opioid use disorder. But that was really not a diagnosis she could accept. Because she never bought drugs on the street, she was getting these medications for her pain, and she was only getting them from me. So no matter what I did, she really did not want to go and see my colleagues in addiction medicine. So just to summarize, these patients had at least two serious problems that are difficult to manage in primary care. The first problem was that chronic pain that they came to me with. The second problem is the opioid dependence that we gave them as an iatrogenic second problem. So my message this morning is that as we consider strategies to help prevent new patients from getting into this boat, from developing this iatrogenic opioid problems, that we remember that top 5% of patients who are already on high-dose opioids, opioid-dependent patients who really need very careful help, and make sure that for each proposed implementation strategy we think, does it make it easier or harder for a primary care practice to take responsibility for the management of that high-risk patient, and help that patient get to where they need to be. Thanks. (audience applauds) - [Larry] Okay. (laughs) Lewis. - Good morning everybody, and thanks for having me. I'm Lewis Nelson as you heard. I'm actually the chairman of Emergency Medicine at Rutgers New Jersey Medical School. - Oh, sorry. - Which is fine. And I'm here to provide with a little bit of an Emergency Department perspective on things that we're doing to address some of the aspects of the opioid crisis that affect emergency medicine and acute pain management. The ED has long been the focus of sort of aberrant prescribing on the sort of somewhat unfounded but partially true belief that we're an easy target for people to come in and receive an opioid prescription. I'm not sure that we are any more guilty of that than any other physician or prescriber, but we certainly sit in a place that makes it a little more accessible to the average person than a doctor's office or another medical venue. But based on that, we certainly have tried to address some of those issues and we've done this through various different efforts that we've put in place. Now we sit in a very interesting place in the Emergency Department because we're often the first opioid prescription that a patient might ever get. And it's usually for an injury and other painful episode that brings them to a hospital before they can get to see their primary care doctor or maybe before they get to see one of their surgeons or something like that. So because of that, we have historically prescribed opioids to very large numbers of patients, but typically in small numbers. But again, given that we don't know who's at risk of developing an opioid use disorder based on an opioid exposure, we likely have been, and probably will continue to be a major source of opioid prescriptions and potentially of opioid use disorders moving on until we can eliminate the use of opioids altogether. So what we've really done in the Emergency Department, or in emergency medicine, is try to do two things. We've really tried to limit opioid initiation, opioid use. We've spent a lot of time looking at alternatives to opioids. And some of these are very obvious. I mean, it could be using ice and an Ace wrap for an injury, it could be using ibuprofen, other nonsteroidal anti-inflammatory, indocedominothen, it could be using a gabapentin or some other opioid for somebody who comes in with what appears to be, say, neuropathic pain. We've also spent a lot of time trying to operationalize prescribing guidelines, often by diagnosis and often to the specific detail of how many pills for how long a duration of time. I worked on this in New York City when I was there, creating an ED network of prescribing guideline that worked within this network. And we all sort of agreed on what would be the right thing to do. We didn't generalize to every ED, and it's jurisdictional in other states and other areas have developed their own. As has been talked about, there's no national standard for how this is to be done. But when opioids are needed, and they are needed as we all know, we've spent a fair amount of time looking at how to use them safely. Our efforts have been really focused on using small numbers of pills for short durations of time, and both addressing the regulatory limits placed on us by states, in New Jersey where I'm from, now, we have a five day prescribing limit for acute pain in New York State, in Massachusets it's seven days, in whatever states you all come from might have your own limits, not all states have prescribing limits. But these limits have limits. They only apply to certain patient populations in certain settings, and they don't generalize very well. We've put together order sets often, again, by indication, ankle fractures should get this much opioid or back pain might get this much or maybe that's a bad example to be using. We've created metrics that allow us to nudge providers, we've created graphics that allow docs and other prescribers to see where they fit in the spectrum of other prescribers within the department. And hopefully nudge them towards more of a centralized standard, and so there's more of a group mindset and group practice. We spent a lot of time educating patients. I think probably the most important thing you need to do is set expectations among patients for the trajectory of their pain and how well they can expect that their pain medication might work. We have been, doctors in general, in prescription monitoring programs, although paradoxically, many states have excluded us from using them, or the requirement to use them. But we've also said, and I can't really leave a conference like this without talking about the other side of opioid prescribing is some of the opioid use disorders and some of the efforts we've made in the Emergency Department to sort of clean up the mess. So you'll turn off the spigot on one hand, but clean up the mess that remains on the other. And I'm not gonna spend much time on this other than to sort of break it down into our two big efforts. Which are really harm reduction and the things we've worked with other organizations, health departments, poison centers, and other groups to sort of, to organize things like naloxone distribution programs and bringing in recovery coaches for patients suffering acute overdose event, get the families involved in patients' addiction and abuse management as we're able to, and then really get involved with public health interventions, whether that's messaging for things like unsafe heroin-fentanyl combinations on the street or the availability of other public health or publicly available efforts like safe injection sites and things like that if they would be available. And then of course, addiction management for patients that have already demonstrated substance use disorder through screening and reducing barriers to treatment and in my and other departments, we've created sort of a warm handoff process to get people directly into therapy rather than giving them the proverbial piece of paper and say go find treatment on your own. I think the ED has come a long way in trying to get a grip on the problems that we see with sort of abundant prescribing and aberrant prescribing as well as trying to really clean up things on the other side, thank you. - [Larry] Thank you very much, Myra. - Good morning. Before I begin my remarks, I would like to congratulate the Duke-Margolis Center on their report. It is really outstanding. I believe it is the best tool we have today to bring together those of us advocating on behalf of those with chronic pain and those advocating on behalf of those who are struggling with substance use disorders, so thank you for this good work. I'm here today to talk about patients. Jonathan and Mariah who live with sickle cell anemia as do four generations of others in their family. Kalind, who is a paramedic, or she was, who was severely beaten, pushed out of the back of her ambulance, and head injured by a person high on PCP. Mike is a environmental attorney who was injured in a bicycle accident after graduating from Westpoint and serving as a lieutenant colonel in the U.S. Army before his accident. And Angela who lives with pain associated with cancer treatment and now co-chairs a group of chronic pain patients who serve as advisors to pain researchers across the country. And more than 30 million other Americans who live with what we've now defined as high-impact chronic pain. Chronic pain is a complex neurological disease which when improperly managed consumes the lives of those who live with it. People lose their jobs, they lose families, they lose the ability to engage in things that bring meaning and joy to their life, sometimes they even lose the notion of self. Our healthcare delivery system has failed this population of patients. Furthermore, in recent years, their lives have become worse due to unintended consequences associated with efforts to contain the opioid epidemic. Many have referred to people like Jonathan, Mariah, Kalind, Mike, and Angela as collateral damage in the war on opioids. These powerful medications have many inherent risks, including addiction and death. Even so, they have been categorized by the World Health Organization as essential medications to population health and are critical to some of those living with chronic pain. One of the key findings in the IOM report Relieving Pain in America is that pain must be tailored to each person's experience. This notion was foundational for those of us who worked on the IOM report and the subsequent National Pain Strategy. Although we know that formulaic approaches do not work when managing complex chronic diseases and that plans of care must be individualized, those of us who worked on these reports acknowledge the value of and need for guidelines for those providing comprehensive chronic pain care, especially for primary care providers, Erin. We recommended that leaders from various professional associations and federal agencies come together to evaluate existing guidelines to look at the evidence that supported them and come to consensus about best practices and develop an educational plan to support their product. That did not happen. Instead, days before the National Pain Strategy report was released, the CDC published the guideline for opioid prescribing for chronic pain. Although those of us advocating for comprehensive chronic pain care agreed with the vast majority of these recommendations, the guideline caused a firestorm in the pain advocacy community. Recommendations regarding dosage limits and caps on duration of opioids prescribed, which were admittedly based on weak evidence, unintendedly intensified the burden for those who live with this disease and must bear it day after day after day. The response to concerns we expressed was that the guides lines were only voluntary and they encourage physicians to use their professional judgment. Many of us express concerns, however, that the CD brand bring such gravitas, and rightly so, that the guidelines would be seen by clinicians as mandatory and state medical boards and legislators were likely to codify them. These things have happened. Within weeks, clinicians began telling chronic pain sufferers that they could no longer prescribe medications that had been successful in helping pain patients manage their lives, sometimes for years. Patients dependent upon opioid therapy without any history of abuse or misuse were told that they would have to seek specialty care or were abandoned by their primary care providers. Comprehensive chronic pain care, like most chronic disease management, is front-end loaded. That is, it is really expensive. Until an individualistic plan of care is developed. So I have completely understood why private insurance companies have not quickly gotten on board with this model. But for the life of me I cannot understand why, within the seven years that have passed since the IOM report was published, CMS has not embraced their recommendations to move towards this model. Once CMS gets one of us because of age or disability, most often we are theirs until death do us parts. Guidelines are surely necessary for changing the way chronic pain is provided, but they are not sufficient. And, they can be terribly harmful to individuals when they lack an evidence base and lead to a heavy-handed approach. Jonathan's physician who was new to the pain management clinic where he had received care for more than six years when the CDC guideline was published, told him that she was no longer able to prescribe opioids, but she would be glad to talk to him about amputating his foot, which is the major cause of his pain. We can do better than that, we must do better than that. Comprehensive chronic pain care will improve the lives of millions of Americans, it will save billions of dollars, and it will reduce opioid prescribing. But until there is funding to support this care model, it will not happen. And people like Angela, Mike, Kalind, Jonathan, Mariah, and millions more will consider to suffer needlessly. - Thank you. Thank you for your terrific remarks for everyone on the panel. I think you can see here we have a very, very well-informed, very highly successful panel, and everyone here has embraced in some way interacting with the guidelines and supports guideline-based therapy. In my institution, when I've gone to our physicians and tried to encourage folks to implement these guidelines in their everyday practice, I get a lot of pushback, and I imagine all of you have had similar experiences. The CDC's asking us to do some risk assessment, patient education, to be identifying opioid use disorder and referring those individuals to be co-prescribing a lot, so routinely using our PDMP. And what I hear routinely from prescribers is, what are you gonna take away? I don't have any time for this because I'm maxed out, and I'm a primary care physician, we're busy, but so are the specialty physicians. And I'd like to hear from members of the panel, what strategies have you found helpful to allow folks to find the time, to make this a priority, to implement these, in day-to-day clinical practice? Go ahead, Erin? - Well, I always say, if you want something to happen on a routine basis, asking individual physicians to be the ones to remember to do it is not the way to go. You need to actually have health system support to make something routine. If we wanna do regular drug testing, for example, with urine tests that just needs to be something that the nurse checking the patient in, asks them to run to the restroom quick after they get their blood pressure checked while they're waiting for me to come into the room. That's how it becomes normal, standard practice. It also makes this clear that this is a normal safety approach. It's not something specific that we're singling individuals out for, but that this is a routine way that we do care. - [Larry] Okay, others? Gary. - I think we have to offer appropriate treatment for folks that we don't have any systematic way to screen all of our thousands of patients on chronic opioids as to whether they're severely dependent, or addicted, there's not even really a good, validated way to determine if somebody is actually addicted or not. Jane Ballantyne has a slide of a spectrum of chronic opioid users. On one side there's a very small, white, definitely addicted, on the other side it's a very small black definitely not addicted, but everything in between is gray. So we just need a lot more work and research on making these determinations, and I think the most promising thing that I've seen has been sort of the Vermont Spoke-and-Hub model where's there's been some funding, regional funding, of assistance to primary care people to help them identify and treat patients with medication-assisted treatment. And SAMHSA has recently given millions of dollars to, I think, almost every state to help kickstart the state systems, especially in Medicaid, but we're interested in worker's comp also in this spoke-and-hub model because this problem is crushing our safety net clinics. It is crushing our federally-qualified health centers, they don't have the time to deal with it. - Go ahead, Lewis. - If I understand the question, it's sort of how to operationalize the guidelines and get them into our workflow, and I think that given our workflow is nearly completely involving an EHR, it has to really be, as I spoke about it a few minutes ago. You can't have to go look up a guideline every time you wanna treat a patient. It has to be in your workflow. So you go on, you have a certain disease or a certain approach you wanna take with a patient, and it's out there for you. There's an order set, and that can include that it's this person's time to their urine drug testing if it's a chronic pain patient. Or in my world it's if they have this fracture they should get this amount of drug. There could be a prescribing default that rather than have to think about each time how many pills I'm gonna give, it defaults to three days or five days or seven days, whatever it might be. But I agree that if you make people think too much, it's basically, it's an unpaid effort that we're never gonna be able to accomplish. But if you make it consistent and kind of syntonic with the work that they're doing, I think it'll go very easily with the flow. We've seen that in our ED and in EDs around the country. Again, I can't really speak to how it works in a primary care clinic, but we do have the ability to share an EMR. - [Larry] John. - I'd like to just comment about one of the major tenants of the CDC's prescribing guideline is to ensure that physicians are well aware of the various controlled substances that have been prescribed to their patients. And one of the greatest tools for that is the prescribing drug monitoring programs that are now in every state, but we recognize that it can be difficult to access that information, it can take time. And to better integrate that into workflow, to make it easier to use, we have a lot of efforts concentrated on integrating the PDMP into the electronic health record. And that has been one of our major efforts at the CDC, which I think speaks to this issue of improving workflow and making it easier for physicians to prescribe responsibly. - Larry, I wanna go back to the notion that what we need to be advocating for is comprehensive chronic pain care, which may include the biomedical approaches that we rely on now, including medication therapy, opioid therapy, interventional procedures, nerve blocks, spinal blocks and so forth, and also surgeries. But we have got to move away from this model, it doesn't work. We know it doesn't work. We have got to be able to offer people alternative therapies, alternatives to opioids. And frankly, most primary care providers don't have access to anything other than their prescription pad. So it goes back to funding. We have got to acknowledge that we need a comprehensive approach and that our frontline providers have to have access to those and reimbursement and time to manage this disease. And until we do it, it won't change. - [Larry] Mm-hmm. Okay. - So I actually couldn't agree more with Myra on that one. And what we're doing in Washington state, we have had the Centers for Occupational Health and Education for over 10 years that the federal government is actually trying to do some demonstration projects, and the focus of that is try to develop step care management for pain to try to prevent the transition from acute and sub-acute to chronic pain. Because that is, preventing that from happening is highly associated with folks going on to chronic opioid use. And collaborative care for pain is a whole 'nother model that we're trying to add to that, we had a whole day of conference. It's all available in videos if you wanna watch it. Dr. Krebs spoke there. The idea is to actually pay master's prepared therapists to help deliver brief interventions like cognitive behavioral therapy in a community and available on a multi-modal team with the physical therapist, with whoever that there are teams in these communities to help the primary care person who cannot do this stuff themselves, but they need, basically, a multi-modal team in the community. So that's gonna take funding. Big time funding from the payers and we're intending to do that in Washington state. - And I think Gary and I may hold hands and sing Kumbaya before this is over, but I'm so glad to hear you mention community-based approaches because we continue to think about this only from a medical model perspective. And we have got to think about whether or not our YMCA, our faith communities, our Weight Watcher programs, if they can't be helpful to us as well. - But one of the things I really enjoy about working on the opioid epidemic is it brings different kinds of people together. It really does. And that's politically, and people have different ideas about how to fix this, we all agree it's important. I'm gonna switch the subject here. A number of you have brought up this issue about opioid dependence and opioid use disorder, I'm gonna take just a brief moment and just define a little bit what we mean by that because there may be people in the audience who might think of those as kind of one and the same. And I think when we talk about opioid dependence, we're talking about a physical dependence. These are not people who have lost control, who are putting themselves at risk, but rather, if they don't get their medicine, they feel terrible. People will tell you it's one of the worst feelings there is, to be withdrawing from opioids. And people who are on chronic therapy, particularly high-dose therapy, they're gonna have physical dependence, pretty much that's universal. Opioid use disorder, which we used to call addiction, it's compulsive use, it's out of control, it's continued use despite trying to cut back, continued use despite harm. There are plenty of people who are getting prescription drugs who have opioid use disorder and we interact with those people. They come and they need prescriptions, they need to be getting them from us. However, there is a large and rapidly growing number of people who are getting their medications or their opioids illicitly. Sometimes they're buying pills on the street, which I understand are increasingly made from fentanyl which is illicitly sourced and stamped to look like prescription drugs that are already available. They'll make them look like a Percocet. And then of course it's injectable, heroin, fentanyl, fentanyl analogs. We sometimes interact with these people in the medical community, and we don't do a particularly good job at identifying these folks as having opioid use disorder and there are barriers to getting them into effective therapy. So for example, in order to prescribe Buprenorphine, prescriber has to have a waiver. So what you have is front-line clinicians and EDs and primary care, they can't initiate that therapy, they might recommend to a patient, hey, you need treatment. But if you don't initiate something there, that person may be lost at follow-up. They maybe be back out on the street, buying and using drugs and putting themself in harm's way. Lewis, you talked about a warm handoff and as we go around, I want you to define what you mean by that a little bit. And I think that is one strategy. There is something called the Three Day Rule, where the DEA allows three days of therapy for somebody who's withdrawing. That can be done in a hospital or in an ED, but I thought three days was you could write 'em a three-day prescription, but actually it's that person can get three days of therapy in your facility. Whether that be showing up to the ED three days in a row or in your hospital. If we had some way that we could allow non-waivered physicians to prescribe short courses, I think that would make a big difference. But I wanna see, what do you all think about how do we identify these folks? These are the folks who are often overdosing and dying, how do we get them into effective therapy? Anybody wanna start? Erin, you'll go first. - All right. Well I'll say, I do think we do need to do more of this in primary care, and all of the challenges that we've talked about apply here. I just got my waiver, which I have not used yet, but it is a little bit of a hassle. It's eight hours of training, not that big a deal, probably the harder part is figuring out how to make my clinic work and do this because it does require careful monitoring. You have to really have a little bit of a system set up for it, I can't have somebody come in and see only me all alone, repeatedly, to manage this, I just don't have that kinda access, and that's the case with most people in primary care. So it's figuring out the team, the collaborative care approach, and trying to learn from some of the other models. There's a Massachusets model out there that's been published that has worked very well, it's a collaborative care model in primary care, trying to coordinate these kinds of treatments. But I mentioned one of my patients who clearly met criteria for at least a moderate opioid use disorder and was personally not someone who would go out to purchase drugs on the street. But the consequences for her were just as severe. And she just simply could not go to a clinic where she would sit with people who were heroin users. She had a bias against heroin users and addicts and she did not view herself that way. She had lost control of her prescribed drugs. And I think honestly, that if at the time I had had my waiver and a clinic approach set up, that I could've taken better care for her. I think things would have turned out differently than they did, she did not have a good outcome. So that's one of my personal motivators for doing this. But I think the challenges are really important in terms of, it's not just individual prescribers getting a waiver, it's how does the system support that. - And it sounds like you're also advocating making this part of everyday practice. Any primary care system needs to have built into it some ability to treat these folks and not just refer to an addictions program. - I think so. I think if we're really gonna get a handle on it, we can't just rely on specialists to be the clean up guys. We need to have more of this in primary care, it might not be every primary care provider, but perhaps it should be every primary care practice has at least one person or one team that is doing this. - Okay. - You know, I have always found it sort of paradoxical that you need a waiver in order to prescribe Buprenorphine, a partial agonist that has a relatively low abuse potential and you'd need no training to prescribe a full agonist opioid with all of its abuse and overdose and death-related potentials. And I'm not sure why that is, but I was a little happy to hear when Dr. Gottlieb talked about perhaps a tiered approach to prescribing education certification which will sort of mimic a little bit of the certification needed to prescribe at least what sounds like the ERLAs and some of the other opioids. So that might be good. I had worked with the FDA on some of the REMS as they were being developed, and I always was an advocate for moving away from an education-only REMS and having a bit more of certification associated with it because if you don't have to get this education in order to prescribe, it seems like the bar to get the education is pretty high and there's really no incentive to do it. You asked about the warm handoff, and I think it's important because in the Emergency Department, it's often not hard to identify people that are having an acute substance use disorder moment. They come in after an overdose, they're brought in by their family or they're in withdrawal or something like that. It's a little hard to perhaps to identify the subtle patients who have an opioid use disorder that you might see in your clinic but we don't have a long-standing relationship with the patient. Often we'll identify things in the PDMP, which is wonderful, but it's very hard to know what to do with that information at times. If somebody's on long-term opioid use it might be quote-unquote appropriate or it might not be. And it's sometimes difficult to interpret, but for those that have what we identify as being sort of hit rock bottom or having a catastrophe in their life at the moment, we really don't in many places have a mechanism to deal with that. Honestly, I said it before, but we literally give them a piece of paper and say go find an addiction doctor. And it's a public health problem in many cities, they just don't have resources. And this is in big cities, in small cities all across the country. It's not any one particular jurisdiction, it's just a problem everywhere, we don't have the resources in most of our health departments or in our health systems. So what we've established and what many others have as well is this warm handoff process whereby we will literally have the addiction medicine person, whether it's a physician or a psychologist or an NP or PA come down and meet the patient in the Emergency Department and start a relationship with them there so that when they leave the ED, the likelihood they're gonna show up at the addiction treatment program is substantially higher. And this has been studied several times and it's been shown pretty clearly: While the show rate at clinic is still fairly low, unfortunately, it's substantially higher than it is without that warm handoff process. So, as you said, we can give you a Buprenorphine tablet and say go to the clinic, or we can give you a direct relationship as you leave the ED. And I think it's something that's very important that we develop across the country. - Larry, the thing that I have noticed over my career, in the late '70s, I'm a very old person, and in the late '70s, I attended a conference like this where we talked about should prescribing of opioids be criminalized? And then the pendulum swung and it was opioids for everybody. If you had a headache slightly, well take one of these, if you wrenched your ankle, take a whole bunch of them. And now the pendulum has swung back. The thing that excites me about this report, and about the discussion that we've had on this panel, is that there has to be some middle ground. My background's in philosophy, and I think about Aristotle all the time. He said, ethics is at least in part politics. This issue has been so highly politicized, that physicians, clinicians can hardly do their jobs. So we have got to let those in the healing professions take leadership here and figure out where the common ground is and do a better job for all those we're talking about today, whether they're dependent, addicted, in pain, suffering, we can and must do a better job. - Thank you. Okay. Next time I'll tell us again. - I'd like to emphasize the systematic approach to identifying all patients that need the help. And one sort of, I don't wanna call it low-hanging fruit 'cause it's kind of like, not good, but there is a report that most people who come to the hospital with an overdose go right back out into the community, back on their same dose, and most of the time their prescribing doctors don't even know that they were at the hospital with an overdose, so some of the state health officers, or the county health officers, using their legal authority in Washington have mandated that hospitals report the overdose to the health department where things are then set in motion to bring the primary care doc in and to get treatment for the patient if they can. This kind of systematic approach is what we really need to do for these patients. - Okay, thank you. We're gonna move on to another topic. So, we heard from Judy earlier that it's fairly straightforward to measure quantities of opioids being prescribed, and that's often being now used as a measure of success, but certainly we need much more than that. And I'd like to hear from each of you, how would you define success as we try to implement guidelines and change the way we practice and then how would you measure that success? - Well I will say that one measure ought to be whether or not individualistic plans of care are developed and whether or not they're evaluated over time and the outcomes lead to quality of life for the patients they serve. - You know, I'm a little concerned. So there are obvious markers, like the number of deaths and the number of overdoses that we see and the amount of naloxone prescribed and a lot of these other things that will tell us that we've certainly, we're more safely using opioids. But I'm a little bit concerned because, how do I wanna say this? Chronic pain to me is in part a function of having used an opioid. And so in a way, a marker for success would be a reduced incidence or prevalence of chronic pain, particularly patients using opioids for chronic pain. Now I recognize that that's a loaded statement. And fairly complicated, but again, when you look at the United States and compare it to the rest of the world, the incidence of chronic pain here is orders of magnitude higher and it's hard to ignore the fact that it's because we have so much opioid prescribing, and some would say it's associated and some would say it's causal. But I would like to see, and certainly in my practice in the Emergency Department, we deal a lot with patients with chronic pain. I would like to look at those outcomes, like functional outcomes and pain outcomes and those sorts of disease states. - John, yeah. - One thing I'd like to mention. An effort that's being done by the CDC to help measure progress is the development of an indicators checklist, about 20 different indicators all tied to the guideline, looking at things like what percentage of patients who have been prescribed opioids for the first time, who have been seen for pain for the first time were prescribed a non-opioid alternative to opioids? What percent of patients are now below 50 MME? What percent of patients are below 90 MME? Things like that, all tied to various aspects of the guideline that you can use as a sort of dashboard and monitor progress. So that's an effort we have going on, in six states, it'll begin next month and will give us a sense of whether or not we're making progress around the country in terms of implementing the guideline. - I don't think anybody's mentioned yet that if you're gonna be prescribing an opioid for almost any reason, especially chronically, the doctor should be using a two or three question validated questionnaire to track pain and pain interference with function. And then the also using a calculator to have to know what the ME, morphine-equivalent dose is and to check the PDMP. If you have all of those data points and you're looking at that over time, I think you can tell a lot, and most of that's not being done right now. - So just to clarify that. So pain, function, and impact on life-- - Two or three, right. - And then there's some straightforward, - You can add - validated tools. - an impact on life one, too, but right, two or three, these are very brief questionnaires that should just become part of the regular record. - And just to clarify your comment, the calculator isn't, you might be envisioning that the doctor takes out a calculator, but often that our PDMPs will calculate this for us, or our electronic health records, and it's simply matter of really taking a good look at, okay, what's the total quantity that this individual is taking per day? And sometimes, particularly if there's more than one drug, that figure can get lost on the prescriber. - And I just wanna mention, not much has been said yet about the combinations of opioids and benzodiazepines, and Z-drugs, and Soma, and a bunch of these other drugs, for whatever reason, they dramatically increase the risk and it's very important to keep track of that stuff, too. - Okay, go ahead, Erin. - So obviously there are a lot of metrics we can use that I won't go into more of those. I think there are two points. One is that we need to look at subpopulations when we look at these metrics. Both to understand how the guidelines are being implemented, the effectiveness of that and to identify the consequences. So I want to see new prescriptions for long-acting opioids going down. (laughs) And it was interesting, I saw one graph this morning where that looked flat. That should be starting to come down, those have limited usage and really should be, we shouldn't be using them for back pain. However, I would wanna look at patients with severe, life-threatening illness. Hospice patients, and I don't wanna see those numbers going down in that group. And if we see it going down, that's an indicator of an unintended consequence. Likewise, we want doses to be coming down. In general, we want the number of people on very high doses to come down. But if you look at the subgroup on really high doses, we don't want all those people to go under 50 in the next month, that means something way too abrupt and too blunt has happened to those folks. They need individual assessment and management. And a variety of different care paths. Again, just making sure we are really defining subpopulations affected and making sure that we're using the right measures for the right patients. And then I think there are a couple just more broad measures about our bigger picture, our culture change. In medical education, how much of the curriculum now is looking at substance use disorders and how much of the curriculum is looking at chronic pain? And if that is still an add-on hour here or there, we have not achieved what we need to achieve. We're not ever gonna fix it unless that's core curriculum. Something has to move. It might be neuroanatomy or something, I don't know, just a suggestion. (laughs) - [Larry] I'd vote neuroanatomy out. (mumbles) - And furthermore, I think we need a culture change that if someone has an acute episode of back pain, which is a horrible experience that 90% of us will have during life, we shouldn't think that the first thing to do is go to the ER to get Percocet. We don't need to do that. We should all recognize that's not indicated. And actually that other treatments are just as effective and much safer. If you surveyed the population, what's the right, what do I want for this pain problem? We want that number to come down, that wants opioids, or expects that that's appropriate care. - Erin, when we were - Just a few suggestions. - preparing the IOM report, we looked for public health models that were successful, and we really didn't find any in the United States. The best example we found and we cited in the report was in Australia, which was for low back pain. And it was a public health campaign that said if you have terrible back pain, you need to keep moving, which nobody wants to move when their back hurts, they just wanna (groans) crunch up. Don't go get an x-ray, 'cause it's not gonna show anybody anything, and be helpful. I can't remember the numbers now exactly, but the prescribing was reduced more than 20% for back pain in an 18-month period. So the cultural factors have to be considered as well as all the clinical. Because as we did say in the IOM report as well, and in the National Pain Strategy, chronic pain is an experience. It is not just a disease, it is a neurologic disease, but it impacts every aspect of people's lives. We are a culture that believes one pill will cure anything, and if that doesn't work, then wash it down with a martini, and that'll surely be better. So we've got to look at the cultural issues. - Okay, Gary, I'm gonna let you have the last comment, then I wanna take questions from the audience. - It's gonna be very important to focus here. And one of the areas we're focusing on is adolescents because their brains are particularly susceptible. So the stuff, the Monitoring the Future study at Michigan showed that if a kid in high school got one prescription for an opioid, there was a 33% increase in risk between 18 and 23 of misusing opioids. And that is a dotted line to misuse and heroin. The biggest increase in the heroin deaths is in the 18 to 30-year-old age group, and I'm pretty certain that interdicting the amount of acute opioids that are being used in teenagers up to maybe 24 even will have a big impact on reducing their heroin problem. - Thank you. All right, I'd like to take a few minutes and allow folks to come up to the microphones and if you could just line up, either side, we'll just alternate back and forth. And if you could introduce yourself and tell us your affiliation as well. - [Bob] Hi, I'm Bob Burns with the American Dental Association. Our guys basically manage acute pain by and large, and a lot of people don't realize we support the CDC guideline for chronic pain. We're trying to get the word out and educate our members. When they see the title that says chronic pain, they immediately, oh well, okay, it might be interesting reading but in the footnote here it says does not apply to dentistry necessarily. So I guess I'm wondering first, if there was a rationale for creating a chronic pain guideline without simultaneously creating the guideline for acute pain. And I guess second, if you're aware of any plans to develop a guideline for acute pain. - So we do have a new acute pain opioid guideline for dental in Washington state, and we've been working with some other states. I'd be happy to chat with you about working with your organization. This was fully sponsored also by the Washington State Dental Association. So this was all done with practicing clinicians, but it's focused on acute pain. I'd be happy to talk to you about that. - Bob, I'd like to ask you about another aspect of this. In our own community, we've looked at the issue of dental pain and how it plays into all of this. And a very high percentage of those who show up in ERs, particularly on the weekend, have dental pain. Which is excruciating, most of us have had some experience with that. So the ER gives three days and says see your dentist. If people do not have a relationship with a dentist in our community, on average it takes them six weeks to get an appointment. So the issue about access to dental care, and talking about the soft handoff, or the warm handoff I think you call it, Lewis, we've gotta figure that out, too. Because you can't just let these people struggle with pain and infection very often. - [Larry] Okay. Thank you, let's take another question. - Uh, yes, exactly-- - Paul Suchoski with Safe RX. - Oh, is that okay if we, we're gonna move on to another question. There's a bunch of folks lined up, go ahead. - [Paul] Yeah, I found the last discussion from the gentleman from the University of Washington very interesting about the impact on adolescents with these drugs. And one of the unintended consequences, I don't think it's getting enough attention, is how do you keep adolescents out of drugs? And it is a big issue right now, and there are discussions about blister packs earlier. Our company offers locking prescription vials. And if you look at the impact that child-resistant packaging had on adolescent poisonings back in the 1970s, FDA introduced those, adolescent poisonings went down by 45%. And we think the same potential is here with adolescents. How do you keep adolescents out of these medications that are not intended for them, they're intended for their parents? - Anyone wanna? - Could I just address that? I'm sorry, you said adolescent poisoning went down when they introduced child-resistant? - [Audience Member] Pediatric. - Went down. - I think. So I work at a poison center and we had a lot of effort put into that as well. I think toddler exposures went down with child-resistant containers, not adolescent exposures. 'Cause you know, adolescents can obviously open a child-resistant container. It's old people that can't. (audience laughs) But I would say that there are newer technologies available that function like child-resistant containers that have locks on them that have combinations and things like that, lock boxes, essentially, that could prevent you from getting it out of your parents' medicine cabinet and things like that. I imagine they're expensive and not widely available, but they do exist. So they're sort of a child-resistant lid on steroids. - [Paul] Actually, they're available today for less than a dollar. And they will keep adolescents out. - The locked ones? - Locked ones. - Yeah. Yeah, yeah. - One of my colleagues at Duke is working with some of the surgery groups to look at appropriate numbers of prescription or doses after various procedures, and one of the things that she's having her research assistants ask patients is, where are you keeping your medications? And it ends up being about 2% are keeping them stored safely. Almost none do. On the kitchen counter, in the bathroom, these sorts of things are the standard. And I think we really need to work on helping people to appreciate how dangerous these drugs are and how they really need to keep them locked. Okay, let's take another question. - [Joan] Hi, I'm Joan Baird, I'm with the American Society of Consultant and Senior Care Pharmacists where I work as director of pharmacy practice. Prior to that, though, I was a clinical pharmacist both in a couple state psych facilities in Maryland and I also was a senior care pharmacist. First of all, I'd like to say I've really enjoyed this. This has been very, very interesting and it's very interesting to hear from a physician's standpoint because I obviously look at things through a pharmacist's eyes. I want to thank you for differentiating between physical tolerance and addiction. I used to give pain treatment classes to the clinical staff and I had to differentiate those, I think there's a lot of confusion. And I'd like to thank you for acknowledging that physicians often don't have the breadth of training in pain management that they probably should have and I think that needs to be addressed. And I'd like to thank you for pointing out all the different challenges in appropriate pain treatment where we don't want to do harm. First, do no harm. My question is kind of stepping back from the patient who is not opioid naive, the ones that I know a lot of you are dealing with, perhaps it's the best question for Dr. Nelson because it sounds like he is dealing with some opioid naive patients in the ED. I would like to hear a little bit about how your pain assessment is handled. I feel like sometimes that is a weak link in the chain. We talk about scale of one to 10, or if it's a child or a senior with dementia, we might use Wong-Baker faces. But to me that's just step one. And I worry that we're not teasing out quality of pain. I think Ms. Myra talked about that a little bit. The quality of pain, is it sharp, is it intermittent, is it constant? One to 10 is just the start, and I'm wondering if in your programs you're including a differentiation of types of pain because obviously, that can dramatically impact prescribing. Or it should, at least. - I mean, I'll quickly address that. There is no answer. Right? We all know this, pain is a very subjective experience, and there is a certain degree of objectivity around it. But it is subjective and you can ask all the questions you want about quality of pain and all of that and will really never be able to treat it any better knowing all that information, in my opinion. I've always been in the school that believes opioid naive patients should remain opioid naive unless they absolutely cannot. So we really try to use alternatives to opioids for all of those reasons because that's the first taste of an opioid is often what leads to a lifetime of addiction. But the fact is, when Joint Commission put out pain assessment is sort of a fifth vital sign, sort of concept, it really never made sense because it's not objective, of course, it's very subjective. We do know that people who come in with, they slam their finger in the car door, they have 10 out of 10 pain, I believe it's the worst pain you can have when you slam your finger in a car door, but when somebody comes in with their femur sticking out of their thigh after a high-speed motor vehicle crash and they have 10 out of 10 pain, they can't be the same. Right? I totally buy they're both having 10 out of 10 for the syndrome, or for the finding. So it makes it really hard to assess a pain. And that's acute pain, I think chronic pain is an order of magnitude more difficult to assess. Because there's so much psycho-social catastrophizing, other things that are put on top of it. I think the most important thing that we need to do as I said before, is set expectations and to try to let people resolve some of the uncertainty they have around what the pain is going to be like. How long it's gonna last, how treatment's gonna be, things like that. I think setting that sort of understanding among patients does help a lot to take away the pain. - [Joan] I appreciate what you say. I do agree, obviously, that if your femur's hanging out or you've just been shot with a gun, you probably don't need a 10-minute pain workup. But if you come in with lower back pain or you come in with shooting pain, maybe secondary to your type 2 diabetes or... You see where I'm going with this. This is where I got frustrated, but especially in the long-term care setting, where that's where, you know, we didn't see the old people with their femurs hanging out or gunshot wounds, but we did see a lot of neuropathic pain that was being inappropriately treated because an appropriate pain workup had never been done. And that's kinda where I'm coming from. I think there are the obvious, where you probably don't need an MD or a PharmD to realize they're in pain, but going down from there is where I think maybe we're missing an opportunity. - I understand that most physicians in practice when asked, do you feel like you've had adequate preparation to evaluate and manage pain, that they say no. And increasingly, medical schools and graduate medical education programs, meaning residencies, are doing better with this, and there's a lot more CME out there. But, certainly, we can do a whole lot better, yeah. - Something to think about, - Certainly, yeah. - [Joan] you know, as we sort this out. Thank you very much for your time. - This is gonna be our last question, so thank you very much, and we can stick around afterwards during the break if you have individual questions. Go ahead, sir. - I'm gonna try to keep it as short as I can. I'm Alex Rich, I'm a health informatics nerd down at UNC. Before that, though, I was an Air Force pilot and crash investigator. So, when you look at a problem like an airplane crash, everybody initially has this instinct to blame a broken part, to blame a mechanic, or finally, to blame pilot error, which is this very broad term, and ultimately, it usually comes down to an information flow problem, a systems problem, a user interface problem. Getting the right information to the right person at the right time in the right format with the right context. So I've heard a little bit about behavioral nudges, I've heard a little bit about the desires for information flow to a clinician with a limited amount of time and attention. I'd like to hear more about what the state of the art is there, what trajectory we're on, and how we accelerate the process of getting where we need to be with information flow to the prescriber. - One piece of information that we're very interested in is using screening questionnaires at three or four or five weeks of pain, looking for psycho-social barriers to recovery that can be more effectively treated. And these are the most, fear-avoidance, catastrophizing, fear of not being able to work again, these are the kinds of things that affect, for example, injured workers the most. So having a systematic way to screen for those things, and then getting those patients the kind of help they need to improve their self-efficacy, to overcome those psycho-social barriers to recovery is what we think is really important. - [Larry] Okay. One more comment. - I'll just say, as a primary care physician, that the state of the art is not great in electronic health records and what they can do for us. Frankly, they've been built for billing purposes, not for care purposes. They kind of get retrofitted for care, but it's always tough. And so that's all I gotta say about that. (laughs) - We can do better. Okay. I would like to thank the panel very much for your participation and great, thoughtful responses. (audience applauds) - Thank you. I'd like to thank Larry and the panel as well. We are taking a break till 11:15 now. You've heard a lot already about efforts to develop and apply guidelines and measure success and progress in dealing with opioids and chronic pain. We're gonna have a lot more discussions during the rest of the day about system supports, other kinds of supports for doing that. So hold on to those questions, and we'll see you back at 11:15. I would like to ask our next moderator and panelists to come on up here, be up here at 11:15 so that we can get started again. - To begin. I'm Regina LaBelle, I'm a Visiting Fellow at the Duke-Margolis School, and I previously was the Chief of Staff at the Office of National Drug Control Policy in the Obama administration. In this session, we're gonna take a deeper dive into how health systems are leveraging health information technology tools, including Prescription Drug Monitoring Programs, and other health data, to improve provider decision making and support safe and appropriate opioid prescribing. The tools can provide real-time information providers on concurrent prescriptions, opioid utilization or over-utilization, and patient risk factors to support improved risk assessment and monitoring. The session will begin with some opening remarks from each of the panelists, followed by some moderated discussion, and then we'll have plenty of time for audience questions. Some of the issues we hope to address in this session are: What tools are currently being used to assist in clinical decision making? Have the tools been evaluated and validated? What are their strengths and weaknesses? How can the tools be better leveraged or integrated to improve clinical care and reduce risks related to aberrant drug behavior? Opioid use disorders and accidental overdose. What are barriers to further adoption or integration, and what additional data may be needed to assess the efficacy or impact on patient outcomes? So we're very glad to be joined by a really great panel today to bring this discussion. First we'll start, starting with Kelly Clark, who is the President of the American Society of Addiction Medicine. Then we have Peter Kreiner, who's a Senior Scientist, Institute for Behavioral Health and the Principal Investigator for the Prescription Drug Monitoring Program Training and Technical Assistance Center at Brandeis University. Jim Huizenga, the Chief Clinical Officer for Appriss Health. And Jaya Tripathi, principal scientist, advanced analytics for MITRE Corporation. So we'll begin with Kelly Clark first. - Thank you, Regina. I'm gonna start back at the issue of definitions. And I appreciate the earlier panelists speaking about this. I'm an addiction doctor, and I know how difficult it is to parse through a lot of the language we talk around here. But we really can't get to solutions until we know that we're all using the same language. So a few definitions to start today. Use. Well, taking a substance for a desired purpose. Many of you may have had a glass of wine or two with dinner last night. Having no problems with alcohol, that would be drug use. Misuse is the use of a medication for a purpose other than prescribed. So for example, if you gave one of your sleeping pills to your spouse, that would be misuse. It would also be diversion of a controlled substance. Risky dosages do not imply a disorder or an addiction is present. And that's something to keep in mind when we're talking about these morphine-equivalent units or doses for opioids. So for example, risky or unhealthy drinking is defined as a woman having more than one drink a day or seven a week, and a man having more than two drinks a day or 14 a week. Now that does not imply that they have a disorder. Or that they have addiction. But these are levels above which there are health consequences and are considered to be risky or unhealthy drinking. Substance use disorder is not addiction. So many people with mild substance use disorder with alcohol, for example, move through alcohol use in their lifetimes without the use of treatment or even use of peer groups. Substance use disorder involves having two things, one of two things present. A disorder means that either the person has some subjective discomfort with this issue or that they they have an impairment in functioning. That's what a disorder is. So one of those things must be present to be a disorder. But again, the presence of a mild disorder does not imply addiction. And very important in this space, physical dependence is not addiction. Physical dependence, again, is a state of the organism, it's a physical state where the organism, in our case humans taking opioids, need higher and higher doses to get the same response or there is a withdrawal condition that occurs when people stop using the substance. And drug addiction on the other hand, is a chronic brain disease. Addiction is a chronic brain disease, and there is, there are, there's a very technical definition that ASAM has. We are the addiction physician specialists. And we can see brains that have been changed because of addictive disease on scans, but from the outside what this really looks like is people lose control of their drug use and then they lose control of their lives because they've lost control of their drug use. And that's what addiction looks like from the outside. So opioid dependence. This is very important, okay, within about six to 36 hours of last opioid use, withdrawal can start and this can look like muscle and joint pains, gut cramping, diarrhea, vomiting, chills, runny nose, runny eyes, agitation. The risk here is very uncomfortable as someone said earlier. It's like a really bad flu feeling. People are sick and they can't function when they are in physical withdrawal from opioids. There are risks, particularly for pregnant women. And people who are medically frail or fragile with opioid withdrawal but it's not like alcohol withdrawal which can be far too often deadly. The way to stop opioid withdrawal very quickly is for a person to take an opioid. Within minutes or an hour. Those symptoms stop. They stop. So a person's who's dependent on opioids wakes up every day knowing they've got to get opioids, or they're going to be sick. They're not going to be able to take care of their kids, they're not gonna be able to go to work. This is what's so different about opioids than other drugs of abuse. Or the most prescribed drugs of abuse as well, is this withdrawal sickness process. And that can happen even when a person is taking medication exactly as prescribed. Exactly as prescribed. So we can see people who are engaged in what we would consider drug-seeking behavior. They may be going to emergency rooms, they may be even seeing different doctors. There is something we call pseudo-addiction, which means if their pain was actually treated, they would not be engaging in these behaviors. Many people don't understand that the chronic use of opioids actually can increase their pain. They only see that briefly after they take that opioid, their pain is better, but then actually it can increase that chronic level of pain, which people don't understand. So when we're talking here about decision support tools, when we're talking about treatment tools, we need to be clear when we're talking about treating pain, when we're talking about treating addiction, when we're talking about dependence, when we're talking about behaviors that even the patient doesn't understand that they're engaging in because they are trapped in their cycle of discomfort. So as we start off our piece of discussion today, I just wanted to take my minutes and lay down that framework for us all. - Thank you, Kelly, and good morning. We're going to jump from those very helpful definitions to the world of Prescription Drug Monitoring Programs. So, largely in response to the opioid epidemic, there's been a proliferation of PDMPs since the early to mid-2000s, and at least as early as 2011 as memorialized in the White House Prevention Plan, PDMPs have been recognized as a key tool in helping providers address the epidemic. At the same time, for a number of reasons that I think are pretty clear, and I'll say more in a minute, providers have been slow to adopt, to use PDMPs when they're not required to do so. PDMPs have been adapting, trying to address some of the barriers to use, and I'm gonna talk about some of those adaptations. I just wanna mention in passing that there's increasing evidence that provider use of PDMPs is associated with reductions in certain outcomes, rates of opioid overdose deaths, certain indicators of misuse. So in addition to informing individual clinical practice, there's a kind of population effect of PDMPs, of providers using the PDMP that's starting to be measured. Okay. One barrier that's mentioned, perhaps not the most important, is that providers complain that the PDMP data is not as comprehensive as they would like. And so most PDMPs have now moved to collecting data on all schedules, schedules II through V, rather than just through IV. Individual PDMPs are collecting data on other drugs of abuse. You heard a little bit this morning about gabapentin. Nearly all PDMPs at this point, and you'll hear more from my colleague Dr. Huizenga, are able to enable queries of data from other PDMPs, other state PDMPs, in particular, bordering state PDMPs. The vast majority, usually around 97, 98% of prescriptions that are filled and reported to a particular state PDMP, originate either in that state or in the immediate bordering state. So the most important inter-state data exchange occurs with the immediate border states and that's happening. Over the last few years, the Veterans' Health Administration and Indian Health Services have been working to submit their data, provide their data to state PDMPs, and that's happened at this point. We did a study looking at veterans who obtain care both from VHA facilities and from community facilities. About a third of veterans obtain care outside of VHA facilities. The various measures of risk are much higher for that group. Definitely behooves VHA providers to check the PDMP as well as the community providers. And some states are working to include new data fields. So you heard a little bit this morning, West Virginia is including, thank you, data on both fatal and non-fatal overdoses. It's really surprising that in many states, someone who dies of an overdose that the provider is not notified of that, there's no mechanism for that to happen. And several states are experimenting with including drug-related arrests and convictions in the PDMP data so the provider can see that information. Perhaps the biggest barrier is the timeliness of data, and I mean that in two ways. One is the timeliness of data submission by pharmacies. Which ranges all the way from essentially point-of-sale submission in Oklahoma to a number of states are still at one week, but many states, increasing numbers of states have moved moved to a next-business-day submission, which helps a great deal. Delegate accounts are enabled in many states at this point so that PDMP records can be looked up on patients ahead of the patient visit. And the biggest barriers is we've heard is the time it takes for a provider to log in to the system, query the PDMP, obtain the records, use that information in their clinical visit, and so states have been working to integrate PDMP reports into various health information technologies, health information exchanges, electronic medical records systems, pharmacy dispensing software, so that pharmacists can have immediate access to the data as well. It's an ongoing effort complicated by the fact that in most states, there are multiple systems. There's not, Washington state is the counterexample where there's a statewide HIE for the emergency departments and the PDMP has integrated with that. Thanks. In most states, the systems are fragmented. They may cover one medical system, a portion of the state, and so the PDMP has to integrate with each separate system separately. And in many PDMPs, and you'll hear more again from Dr. Huizenga, are improving the interface. So when you call up a PDMP report, you get a laundry list of prescriptions, which if there's a lot of prescriptions listed in a recent time, may not be extremely helpful, so there are ways that that data can be more concisely summarized while still allowing drilling down into details as desired. Additional tools that PDMPs are enabling. Many PDMPs issue alerts to prescribers when patients hit certain thresholds deemed to reflect patient risk. These are population-based measures of risk, they may or may not reflect risk in individual cases. So there's variation across states. There's a certain amount of commonality in what those risk indicators might be, but the precise measures vary across states. And New York State, for example, has mandated electronic prescribing, we heard a little bit about that earlier this morning. Electronic prescribing of controlled substances which, in theory, will solve several problems. Okay. I'm gonna conclude with just noting some limitations to PDMP data. 'Course it does not contain clinical information, it's strictly a report on outpatient prescriptions for controlled substances. The data quality, it's a recipient of data, so data quality depends on what the pharmacies submit. PDMPs can implement increasingly stringent filters in terms of which data submissions they allow and which they don't, which data fields are deemed as critical and which are not. But there's variation. And for the most part, PDMP data do not include a unique patient identifier. So determining which prescriptions belong to the same patient requires in most cases a probabilistic, sophisticated algorithm for determining that, for matching on variations of name, typos in variations in address, typos in birthdate and so on. And again, there's variation in how states do that. There's not, at this point, a gold standard. Thank you. - Can I have that please? - Sure. - Thank you. Okay, hey, thank you for being here today and allowing me to speak. Again, I'm Jim Huizenga, I'm the chief clinical officer for Appriss Health. I'm gonna show you a few slides on our approach to maximizing clinical effectiveness. A little background, our company supports 42 PDMPs in the country. That gives us the potential to move the needle and in fact the responsibility to move the needle on this problem. We get to hear from the stakeholders in each one of these states and we have for many years now. We hear from the PDMP administrators, we hear from the physicians in those states and the directors of health. And through those interactions, we have come up with an approach to help move the needle on this. So I'm going to, I'd like to just explain the direction that we're moving in and some of the success that we are having. If I were to summarize what we've heard from folks, and summarize the approach that we've developed when it comes to the PDMP and maximizing its effectiveness, I would say that the approach could be summarized as upfront, every patient, every time. And what that means is that the PDMP information is critical for effectively every patient encounter. I mean, to just look at it mathematically, every patient in the country is either opioid naive or they've had opioids. They fit in to one of those two categories. And that information is important to know as soon as possible while you're taking care of that encounter. Now this applies to prescribers and pharmacists alike. Every patient outside of infants and toddlers, but if we're talking about adults, this problem can and is effectively everywhere. And every time. Not just when you're thinking about prescribing a medication. This information again, it's important to know about the patient, it's a critical part of their history, and I would say that you really cannot take care of almost any problem without knowing this information. The way we're approaching this, to get to that upfront, every patient, every time is automated integrations, make the data usable, and expanding the scope and the functionality of the PDMP. And what that looks like is, is if you look at the time from patient arrival to the time that the provider is actually going into the room to see the patient, now I'm an emergency physician, so I've broken this down into triage, a registration triage and alike, but in the middle of that early on process, we provide technology that allows for an automated query to occur in line with all those other patient events. So that before the prescriber or before the pharmacist is actually interacting with the patient, this information has already been gathered and assessed and placed for them to be referenced. The way we've gotten there is we have worked with the manufacturers to build this technology into their OEM software. There's more than like 5,000 hospitals and untold numbers of pharmacies and individual offices, and if we wanna integrate the PDMP data into workflow, we can't do it one by one, we have to go to some other point in that process and that point we determined is with EHR manufacturers. So we've been working for over a year now with building this technology into these platforms so it can simply be turned on, and we're doing the same with the pharmacy manufacturers. So we've had some pretty good results. Since we've been working with integrating PDMPs into workflow for both clinicians and pharmacists, we have last year alone brought PDMP data automatically into workflow 288 million times. That's a good step forward, that's moving in the right direction. I would like to talk about, so now that we're getting data into workflow, or we're bringing the PDMP into workflow, we really do have to think about, well what are we bringing? How to make the data more usable in workflow is really the next most important step. A list of prescriptions is very important, and it's the foundation of all PDMPs. The individual elements of each prescription. But it's not very usable in clinical workflow. It's too much data, there's too much of a risk that people will make the wrong decisions and the wrong assumptions on it. So one of our approaches to how we improve the usability of the data is we have numerically represented the PDMP data. We have a series of use scores and risk scores that we bring into workflow. And with just a little bit of education, providers can use these numbers and incorporate that information into their decision. Now most often that decision should be that when they see something that's concerning or unexpected, that they look deeper. They don't use these numbers to make decisions on how they're gonna treat the patient, these numbers do not reflect, a high score represents more use of controlled substances, does not mean that the patient is abusing or otherwise is using them inappropriately, just raises awareness. Numerical representation is everywhere in the EHR, blood pressures, CVCs, hemoglobins, et cetera. So providers are used to looking at numbers and applying them to the patient, we are providing an avenue for them to do that. Sometimes, though, it's more than numbers, it's about patterns like with an EKG. An EKG has numbers on it, but does also have patterns, wave forms that can be critically important to determining what's going on with the patient. So we also represent the data in a graphical format so that those very subtle patterns of use, some of which can be very critical to understanding what's going on with a patient, can be made evident to the user of the system with just a glance. There we go. And lastly, automated integrations make the data usable and then expand the scope and functionality. So we are actively working right now on bringing holistic data into the PDMP viewpoint. So that means that we are looking at bringing claims data, continuity of care documents, we're looking at even at the role of criminal justice information, and we've heard this from many of the stakeholders in the states that criminal justice information can sometimes be incredibly relevant and important to taking care of the patient. So we're looking at bringing that in as well. We're opening up care coordination activities. So that prescribers and pharmacists can communicate through the platform, and lastly, we're starting to bring management tools in so that, as you heard earlier, if you have a waiver and you wanna begin prescribing Buprenorphine to take care of a patient, you need a system of support to be able to do that. And we're looking at building that support right into the PDMP. Thank you for your time. - I am actually unable to see that, so. I'll use the slides here. So hello everyone, I am Jaya Tripathi, and I'm a principal scientist for data analytics at MITRE. MITRE is a nonprofit that operates several FFRDCs, federally funded research and development centers, and we work in the public interest. So today we are here to talk about decision-making tools that support the safe and appropriate use of opioid prescribing. So what are some of the desirable features in decision making tools? Decision making tools should facilitate the practice of evidence-based medicine to substantially improve healthcare quality. They should prevent errors and follow clinical guidelines. They should be based on agnostic, standard data models, be dynamic versus static, be based on an amassment of expert knowledge, and lead to unambiguous decision paths. In addition, they should be pluggable, easily integrate-able into existing workflows, take no more than a few minutes of patient encounter time, and also have a validation loop that assesses the clinical effects and outcome evaluations and be adaptable and retrainable on new populations. So I started this research work in 2012, and very early on, I engaged in a landscape study with public and private stakeholders. As part of the landscape study, we did a survey of decision making tools for opioid prescribing. More recently, six years later, in 2018, we redid the survey with a different population. Here are the results, some of the results of that survey. The respondents were all physicians, and there were 63 of them. As you can see, their specialties varied from emergency medicine, pain management, oncology, psychiatry, and so on. 92% of the respondents, the physicians, had prescribed an opioid in the past 90 days. 22% however, had not logged in to their state's PDMP at all. 77% said they use an EHR system, however, only 16% use a clinical decision making tool, and 25% of them had said that their EHR was integrated into their state's PDMP. I will now share the respondent comments on decision making now and also their desired wish list. On the left, some of the decision making methods are traffic signal rules. You may have heard of the green, yellow, orange rules. The other one, ankle rule, PERC rule, personal experience, clinical judgment, use CDC guidelines, UDS drug screen, and Wisconsin PDMP. Wisconsin has a dashboard and the prescribers in Wisconsin are aware that if they are writing a controlled substance prescription for any more than three days, they must check in their PDMP. The wishlist: single sign-on for all PDMPs, differential diagnosis and data supporting decisions, automated best practice advisory in a popup, and clear, concise, decision making trees and bullet points. So we then set out to create our own decision making tool with a systematic approach in 2012. I mentioned the landscape study already where we involved public and private stakeholders. We then acquired a large, longitudinal PDMP data set from the State of Indiana, and applied descriptive analytics, then moved on to predictive analytics for the purpose of classifying risky patients, and finally built MeDSS, the Medication Decision Support Suite tool that is actionable, web-based and dynamically generated from an agnostic data model. So initially, after we built the tool in 2014, we had a usability and interpretability study with physicians and pharmacists from the state of Indiana. Here is a screen, we are now, we finished phase one of an outcome study, we were determining the prescribing behavior of physicians. They're in two groups, the people in control group get this tabular view which is the Massachusets PDMP screenshot from 2015 and thank you to Dr. Scott Weiner of Brigham Women's Hospital for partnering with me in this project. The test group gets MeDSS and with my PIs from Brigham Women's as well as Baltimore School of Medicine, we were also able to create a dashboard and add the patient's medical history which is the patient vignette popup. So I wanted to, I think I skipped a slide. Oh no, I didn't, okay. So as I mentioned, we worked with subject matter experts, physicians, pharmacists, analysts in the field and they helped us annotate patients on risky aberrant behavior. We then built a multi-class classifier of risk out of features derived from the raw attributes that we got from the INSPECT Indiana data set. Feature engineering is a complex, cyclical process. We took the top seven predictors and classified risk on the hyperplane in that seven-dimensional space, bringing patients into one of three groups, high, medium, and low. Here are some of the results from the pilot that we have ongoing currently. You can see that the control versus the test group, you can see that the control group on, here are two of the features, the MME interpretation, the daily MME, and then the geospatial interpretation is the other one. You can see that the people in the control group answered unknown, they were not able to answer the question relating to those features. As opposed to the people in the test group who did not have any unknowns at all, they were able to answer all the questions correctly. Additionally, as you can see from, here's the test group, not only were none of them unknowns, there was a lot of uniformity in the responses. Here is a chart from the usefulness survey. Again, the control group is the one above in green, the one below in blue is the test group. You can see the test group veers towards finding those features far more useful than the previous group did. Oh, never mind. How do I get back to that last slide? - [Man] I'm not sure. - [Jaya] Lunch break? There we go. - There we go. - So the people in the test group have all veered towards usefulness, whereas the control group that got that tabular, Excel-like view, not only did they answer unknown in a lot of the questions, they also found questions not very useful, I mean the features not very useful. So I'm going to conclude now with the last slide if I can get to it. I cannot see and I cannot click. But at least I have my brains. (laughs) Still working. So to conclude, what are some of the challenges? Do the tools reduce subjectivity, or is there still a reliance on physicians gestalt? To what extent does prescribing behavior change when presented with information that involves complex computations and not just straight-up raw attributes? So that last phase on how the prescribing behavior changed, we will be publishing that in a couple of months soon. What are some of the challenges and barriers? The absence of standard PDMP data model meant that in implementing this tool and making the pilot model transformation layer needed to be done, so we have a RESTful API, a RESTful web service. For example, even getting into the NDC code format that we need, the 5-4-3 format was a challenge. As you saw from the respondents, the lack of EMR-PDMP integration. Legacy systems make transition slower, and staging is piecemeal. It's hard to create dashboards without record linkages, which Peter Kreiner alluded to across disparate data sources. Thank you. - Thank you. Thanks. So we're gonna have a little bit, thank you to all the presenters. I think it set us up for some good discussion and we'll have, I wanna start off with a question about additional data. Each of you referred to additional data sources that could be useful in affecting patient outcomes and improving patient outcomes. So what are the types of holistic data? And if you could expand a little bit more, I wanted to talk a little bit about, more about the criminal justice data in particular and how that can be used in a non-stigmatizing way, how do we guard against that? So, perhaps Kelly, if you could start? - Sure. When we talk about data, I'm gonna make another high-level observation, that data are not knowledge, and knowledge are not skills. We have lots of data that prescribing antibiotics for viral illnesses is harmful instead of helpful. Doctors know this, it's not a knowledge gap. But there is a skill gap in saying no, therapeutically confronting the patient, saying, gosh, I can't hook you, they don't help you with this. We continue to prescribe, as physicians, when we know we shouldn't. So, just keeping that data not knowledge, knowledge not skill in place. When we look at sort of outcomes data, and I have to say, what are our goals? Are our goals for this to decrease deaths? Are our goals simply to decrease morphine-equivalent doses being prescribed, because that doesn't necessarily decrease deaths. Opioids are fairly cheap. People can pay for them if they're getting prescriptions without using their health plans. So I go to what are the goals, are the goals the functional goals of getting people back to work, getting people not involved in the criminal justice system, et cetera? I think one of the things that we really need to do is kind of come together and say, what are we looking to do as we confront this epidemic? Outcomes that matter. SAMHSA has some of those outcomes that matter and they are around, getting back to work, staying out of emergency rooms, et cetera, improving health, and so on. So when we look for the kinds of information that would be useful. In terms of addiction treatment, which is my specialty, what we would really like to know are data around types of, more data around types of interventions, types of providers, dosage of treatment and et cetera that provides good outcomes in treating addictive disease. We have a huge education gap where people still think that treating addiction means detox and rehab when what we know is that people who leave a 28-day rehab for opioid addiction treatment, if they're not on medication, ongoing medication, their risks of dying are greater than if they had not completed that program. The program that we use for alcohol does not work on a population basis for opioid addiction. One of the tools that we have are computerized tools, the ASAM criteria which become the continuum product, and we could talk a little bit about that later. But plugging in clinical data with payer data with mortality data with functional data are really the pieces we need to get to knowledge. - Great. Peter. - I think a big part of the point of the PDMPs that I mentioned that are including overdose, fatal or non-fatal, data as well as drug arrests and convictions is to make providers aware of possible illicit drug use by their patients. And among people who tend to misuse both prescription opioids and illicit opioids, there's a great deal of poly drug use. So chances are very good that they would return to a provider seeking a prescription even if they've begun using illicit opioids. And it's a chance for an intervention, it's a chance for the provider to try to have a conversation to work with that patient, to get them into treatment. You heard from Dr. Lewis Nelson about the warm handoff. So I think it's one opportunity for providing data that otherwise is not gonna be available to a provider about drugs obtained outside of the legal drug system. - Yeah, so criminal justice data was one of your thoughts there. Appriss Health is part of Appriss and Appriss as a company has a lot of criminal justice data in the country, about 85% of all the jails and bookings and events in the country that happen are managed by our company. And we do that so we can provide victim notification services for safety. That did put us in a position, though, to be able to look at the effect of criminal justice data on unintentional overdose death, as far as predicting risk. And we routinely work with states and we get data from the states along the line of unintentional overdose death and identities of patients that have suffered that. We have access with their permission to all their PDMP data and we have a data science team that goes through and looks at all that information and helps them to understand it and also builds tools. So we have a tool called an overdose risk score, it's a three-digit score and it numerically represents overdose risk on a continuous scale between zero and 999, effectively. We took all of the criminal justice data in coordination with the state, we took all of the criminal justice data in that state, we put it up against the PDMP data, we looked at the outcomes, and we assessed the relative impact of including the criminal justice data in the model that predicted unintentional overdose risk. What we found was about a 30% improvement. Okay, that's significant, double-digit improvement by looking at the criminal justice data. And that speaks to the fact that our opioid problem has transitioned from a prescription drug problem to an illicit problem. And the markers of risk nowadays for the patients who are dying from this problem are more and more outside the PDMP. Their problem may start with prescription drugs and initially may be represented in a PDMP, but you know, the ultimate folks that are dying, it's oftentimes from heroin and illicit fentanyl. So the criminal justice data and other types of data that represent those other sources of risk, incredibly important to put into that PDMP viewpoint. How to destigmatize it? As an emergency physician, I don't wanna see someone's arrest record in my EHR, I don't. I don't think it's appropriate. But I also don't wanna be blinded to potentially really important information about that patient and how I can manage them. So ways that we can do that. One of the ways is, is we can use this information and we can put it in the models that accurately predict risk. And we don't necessarily have to tell the providers all the sources of data for those models, sometimes that data, it's 80 different elements of risk within a prescription drug record, it's 80 little tiny points, they don't need to know that, they just need to believe in the model. So we can do it, we can solve it, and we can convert criminal justice data to medical information if we need to show it so we can just, perhaps someone who was arrested for heroin, we present that within an EHR as history of heroin use, and they don't have to know that it came from the criminal justice system. But to avoid using it, I think would be a mistake. - Thank you. Jaya, what additional data would be good, so we talked about clinical payer mortality in your study. - So there were three points I wanted to address, one was specifically law enforcement request data. So we did get, at an individual level, records linked with law enforcement requests with INSPECT, and what we found was in the absence of notes, they could've been, the requests could've been something unrelated to the specific risk that we are classifying for. So we found that within the absence of notes, it wasn't a very accurate means of joining together, however, we did use an actual language processing technique, some free text when we were doing the prescriber risk scoring, and they're rejoined with ResDAC, with NECA smart briefs, and so on. So the other point I wanted to address, Kelly's point about data. I think you said data is not harmful. It could be harmful. Oh, maybe I misunderstood. But I wanted to mention as a data analyst that when used responsibly and removing study biases, and if you use a multi-pronged approach as we did using descriptive analytics, visualization techniques, graph analytics, as well as predictive analytics, and done the right way, data is extremely useful and can inform on various dimensions such poly-pharmacies, geospatial payments, and so on. Lastly, I wanted to mention the Chapter 55 effort in Massachusets that MITRE and Brandeis also was involved in where we did join ambulatory record mortality data, all payer, all claims, PDMP together, at the individual level, and some of the data sets that were not as useful were the ones that were not at the individual level. So when you join disparate data sources, you're constrained by the level of granularity of the least, of the data set that's the least granular, and that's a problem. Almost you hear everyone who's in data analytics talk about the joining of these disparate data sources and what a challenge that is. But, if they're all together, it would be wonderful. - Great, thank you. The Chapter 55, if folks haven't seen it, from Massachusets has an incredible source of riches for policy makers who examine. So I wanna go back, did you wanna, add something, sure. - Yeah, if I could just go back to my comment on data before. What I was trying to say is data is not knowledge. And that knowledge does not mean that the person who has the knowledge, has the skill to utilize that knowledge to the end that they're looking for. And some of that goes back to your comments earlier about - Usability. - Visibility, usability. - Multi-purposability. - And that primary decision making tools that physicians use are their heads and then heart and then guts. And to your point about the physician gestalt overriding, sometimes that, sometimes where I was getting. - Did you wanna add something, Jaya? That's a great segue to the next question, which is about how providers think about risk assessment, risk assessment tools, Kelly you can speak on your personal experience as well as Jim, and then we can talk about the applied piece. - Sure. As physicians we're trained to look at data and we start with what we know, what's already in our head instead of having to look things up, right? And when we look at risks and benefits and alternatives which is how we get informed consent, it's what we have handy. It's our knowledge base that's right there for us, it's our relationship with the patient. And when we make risk-benefit assessments, we look at things about the patient, their attributes, their family attributes, what's gone well for them in the past, what's gone poorly for them in the past, what's gone well for their family, and not for their family because there are genetics involved. There are a lot of, the everyday decision makings that occur because of the relationship that the physician has with the patient. I assume most of you, if somebody is treating your chronic disease, you want it to be a human physician and not a computer that's kicking through algorithms. The gestalt will override. But we haven't armed with quite enough of are utilizing the data and the information that's out there into forms that are usable and knowledge-based for us, which is exactly with the points of my colleagues on the panel here. With addiction, we have a few tools for decision making, first is just knowledge that there are national guidelines for the use of medications treating opioid addiction, the use of labs in treating addiction, et cetera and how to interpret those labs to look for abuse and diversion. So those are actually sort of the basic decision tools are getting the knowledge on board, and the next level are things like ACMs continuum product which are the evidence-based, it's a computer-based assessment that a clinician and a patient goes through to capture the data on, well what does that person need at this point in time, for clinical, for chronic issues, people will need higher and lower levels of care over time. They'll need different modalities of care over time. But this is a decision support tool that really helps kind of kick out at the back end where the patient needs care, what kinds of care they may need, that sort of thing. And there's also a core triage which is an abbreviated version of this. These tools are evidence-based, they have been studied for validity and et cetera. So we do have those types of tools that are available for us. - Thanks. Peter. - So I'm not a physician, but I do wanna comment on the different risk measures that are used in PDMPs. So we've already talked about other data fields that a few states are including. And I mentioned that although there's commonality around the kinds of risk measures that PDMPs use to trigger provider alerts. So in particular, use of multiple prescribers and multiple pharmacies in a given period of time, high average daily dosage of opioids, overlapping opioid prescriptions, overlapping opioids and benzodiazepine prescriptions. The precise thresholds, the precise measures that are used to trigger alerts vary by state. And so I think there's two features about that that may actually be desirable. One is that the variation in states and state PDMPs offers a kind of natural experiment. What particular risk measures may work better in different states, can other states learn from that? If we had a common standard that might or might not mask certain differences. The second feature is just as the drugs of choice for misuse vary quite a bit across the country. In New England, where I'm from, opioids, heroin, fentanyl are the drugs of choice for misuse. Methamphetamine is hardly seen at all, whereas other parts of the country, methamphetamine is a big deal. Just as there's that variation, I don't it's been established that there may be different indicators of risk in different parts of the country, I think that's a question that Appriss may be in a position to start looking at, examining. But it may be that appropriate indicators of risk vary across the country. Wyoming's PDMP is now allowing prescribers to set their own thresholds by which they'll be notified of when their patients hit those thresholds. And again, as I mentioned, all the studies that try to establish risk are population-based studies. They say these kinds of measures are associated with higher odds ratio, higher chance of increase of fatal or non-fatal overdose, it's a population-level risk. What constitutes risk for individual patients as Dr. Clark mentioned, really has to be informed by the provider's knowledge and understanding of that patient. - Great, so a discussion of risk, I think in order to be complete, you have to look at the different dimensions of how risk can and should be used in clinical care. So as a clinician, I would say that obviously where risk is present, it needs to be visible. It can't be hidden in the details. The first step or the first dimension of using risk appropriately is understanding it objectively. For instance, a person who might have a risk of unintentional overdose of 3%. An objective fact, I can understand and apply that, or at least get to that objective measurement and say okay, so they have a 3% of unintentional overdose death. Next step, though, where is that risk coming from? So knowing their risk is important, but knowing where that risk is coming from, what's contributing to that risk is very important. And then the third step is, well what can I do about it? And perhaps what can I do about it, or what should I do about it is the question that really has to be asked. I think a good example of this type of risk analysis and processing is, I think the VA STORM tool, if anyone's familiar with that, I think the VA's done a really good job of highlighting risk and showing where those elements of risk are, even going so far as to saying if you make this change in their care, they're risk will change to X. So all very good information. So kudos to the VA for that bit of work, I think it's excellent. But that's my last point, though, should. If you see risk, and you understand where it's coming from and you understand the options in front of you, you still have to ask should I be doing anything about it and that means that you have to get to the nature of benefit. Using any amount of opioid is risky, period. It's being used for some benefit. That 3% unintentional overdose death risk that I talked about, that I started with, and I said, well that's the first step in understanding it. For one patient, that might be entirely unacceptable because they have no offsetting benefit to justify that risk. But for another patient, they may have incredible benefit at that risk level. And just knowing that individual risk assessment right up front can and probably should lead to very different actions and it's based on the downstream understanding of risk and support of applying that risk. - [Regina] Thanks, Jaya, did you have anything to add? - Yeah, I wanted to mention that it's challenging for, pain management is challenging for physicians, but it's particularly challenging for people in ED who have not seen that person before. And it's hard for them to discern between someone who is doctor shopping versus those who are in legitimate need of pain. And I think these decision making tools are not in lieu of physician gestalt, but they are in aid. And I think they remove, if done properly, less ambiguity in decision making. Studies have shown that even when presented with the same profiles, same patient profiles at the point of care, there's a lot of variability in decision making, and I think some of these tools can reduce. I wanted to clarify that I don't think they are in lieu of, but they are in addition to or in aid. As far as risk scoring goes, Dr. Kreiner mentioned multiple prescribers, multiple pharmacies, I think these are good indicators. But I think that we can do better than that. I think using compounded or derived features, we did the study, and I think there's a risk of alert fatigues from using straight-up just the number of, the distance they travel or the early refills, I think there's a lot of risk for alert fatigue. And using historical data and joining with outcomes, whether it's non-fatal overdose or mortality, whatever data, I think we can do much better than that with features that have a much greater discerning power in separating classes of people. - Thanks. So another, we've talked a lot about the tools, we've talked a lot about the risks, what are the barriers that we're currently facing? It sounds like these are good tools we can use, we just need more people to use them. But what barriers are there to further adoption and also integration with the other healthcare tools that are out there? Go ahead. - Okay. Basically, our primary decision making tool here that I talked about earlier has really failed us with the physician gestalt approach. And it failed us because we start at the basis of education and knowledge, we started with an incorrect knowledge that these medications were safe and people couldn't get addicted from use and et cetera. So if we start with just the basics of that, that we failed with this. And so now we're really at a point of, in order to utilize these, we need to start back at those basics of getting physicians educated, all clinicians educated, and to the earlier point, the patients educated with reasonable expectations of what they should have with care for their pain or addictive disease. The, I'm going to take a pill and this is all going to go away acute response, is not useful for chronic disease management. So one of out barriers to utilizing is really our educational barriers, a lack of reasonable expectations, and I think also some, some reliance on a thought that this complex problem can have some simple answers. And simple solutions usually just lead to unintended consequences. So with a shoutout to, I would be remiss not to say that I'm an alum of the Fuqua School of Business at Duke University, and there the economists will tell us that if we've got supply reduction without demand reduction, we'll have substitutes. And this is what we see with heroin and fentanyl. For the people who are, we have far over two and a half million Americans who are addicted to opioids now. And we could get rid of all of our prescribing of opioids and that demand will still be there, and it'll get filled through a different channel. So when we look at where are we aiming, I think some of the barriers are what are we trying to get at and being really clear about that in each of our areas. Payment, of course, is a barrier in medical care, particularly for some of the more evidence-based treatments, we have lots of problems in addiction medicine around that. I think when we look at it, or when I look at it, those barriers and the things that get us in the way are things that can look a lot like alarm fatigue. That we all have these little bits of things, and putting it together as a combined, universal approach, that's the piece that seems to be really missing. - One of the barriers to integrating PDMP reports into health information technology we haven't talked about is cost. So each time that a PDMP tries to link with, integrate with an electronic medical record system, or health information exchange, there's first of all an upfront cost to the EMR system or the HIE for doing the technical modifications that allow linkage to happen. I think that's been facilitated by sort of a common interface on the PDMP side, thanks to Appriss. But there's still an up-front cost and then in most cases, there's an ongoing cost to the system for use via whatever that interface looks like. Now, when physicians are made aware of the possibilities of receiving automatically a PDMP report on a patient, they'll put pressure on their hospital system, their EMR system, their provider system to make that happen, to front those costs. But there is an ongoing cost, and in some cases, for example in Maine, which has a statewide HIE, but most physicians do not belong to it. So there's a definite trade-off on costs for that, for integrating with that HIE for example. And in other state solutions, it just varies. Let's see. I wanted to say something else if I can remember now. Oh. So in light of costs, sometimes, and I think this is an evolving, it involves mostly technical barriers, but sometimes when PDMP reports are integrated into health information technology, there's a, I mentioned that PDMPs vary in terms of how they determine which prescriptions belong to the same patient. The same issue applies when you're linking PDMP data with some other data system, such as a patient record. And initial efforts to integrate PDMP reports relied on exact matching of patient information which means that the integrated PDMP report that's called up may not have all of that patient's prescriptions. I think that barrier's being addressed, but it's still exists. And there's a related barrier about to what extent can the integrated PDMP report call or request data from other state PDMPs, especially the bordering ones. Again, each piece like that adds a bit of a cost element as well as a time element. So I think these are, to some extent these are technical challenges but they're still there, they're being addressed. - [Regina] Thanks, Peter. - Keep going on that? Okay. Well I'm glad Peter brought up the cost issue. Certainly, in the last year or two, there have been a significant number of federal grants that have come down to the states that have helped offset that burden. The challenge with the cost issue, though, is basically ultimately gonna have to be solved probably by economies of scale and the creation of true value in the system. Luckily, here at Appriss, we have 42 states as our customers right now, so when we invest millions of dollars in new technology we get to spread out those costs amongst a large user base. So that certainly does help. But you know, at the end of the day, we have to create the value, the change in outcomes that ultimately allow the PDMP to not be seen as a cost but as ultimately a tool that can save healthcare dollars. But if we looked at the barriers, I would say mostly they've come down. Three years ago, the barriers were legislative, technological, and cost. The federal government helped out with cost, technology barriers, as I showed you earlier, we've been, as a company, we've been working for quite a while directly with the OEM manufacturers to solve the technology barriers. For the most part, they're solved. If you wanna integrate PDMP into your workspace, and get these tools in front of your clinicians, it's more a decision and not a technical project at this point. Legislatively, historically this data was highly controlled, and oftentimes only useful for law enforcement purposes, but now there's recognition that the data is very important and necessary to be used for clinical decision support. And along with that recognition, the changes in legislation are beginning to occur. I would say that the barriers are, my opinion, the barriers are mostly down at this point, they really are, it's a matter of making the decision to use. - [Regina] Great, thanks. Jaya. - Yeah. In my last slide in my presentation, I covered the IT technological challenges as well as some of the others. I wanna add a couple of points that I had not made before. One was, decision making, other than in the point of care or clinical practice environment, there are other tools that could be used, decision making for policies, for example. States do have limited resources and heat maps are hotspotting, more complex ones than just straight-up PDMP data, but joining with other data sources, or even looking at poly pharmacy. They help target the state's limited resources and where they want to target education outreach program, where to place the diversion officers and so on. So that's the point I hadn't made before. As far as cost goes, I really do not have much to say on that, except to say, maybe on the prescriber, since the prescriber mandates in 30, I think it was in 2012, Peter, that only two states, Utah and Arizona, had prescriber mandates and now it's 36 or 37. And so, the mandates are there in most of these states, but are the tools easy enough for these prescribers to use? Lastly, on the reporting side, in Indiana at least, the web portal has come a long way from where it used to be when I first began. Partly as a result of that, the reporting to the PDMP has doubled. Double the population is now reported in the PDMP tool in Indiana than was in 2011. Those are the only three comments I have. - [Regina] Perfect. - Could I? - Oh, sure. - I wanted to say a little bit more about the user interface which I think is still, it's still a barrier, at least in terms of, until some more concise presentation is widely adopted, but. Jaya mentioned the mandates, so. Most states at this point have some law requiring prescribers to enroll in the PDMP, set up an account, or especially to query the PDMP under varying conditions, which can range from once every 90 days for somebody who's being treated for long-term opioid therapy to every prescription, to every opioid prescription, so it's varied. For the more comprehensive mandates, and this reflects on the user interface question, when we looked at the effects of those mandates, there's mixed results on affecting rates of opioid overdoses and fatalities. But the most consistent finding is in large reductions in rates of so-called multiple providers episodes. Use of multiple prescribers and use of multiple pharmacies in a given period of time. There's some evidence for reductions in rates of overlapping opioid prescriptions. What they're, and just as consistently, there's been no effect observed on average opioid dosage, daily dosage. So this at least suggest to me that what when prescribers are looking at PDMP reports, what's most salient are the number of different providers on there. Maybe they can see overlapping prescriptions, but when it comes to actually computing average dosage, more complicated measures, those so far have not penetrated to the mainstream of the user interface. - Thank you, Peter. We have about 10 minutes for questions, if folks could line up at the, and speak into the mic and ask your question, please. - [John] Good morning. And thank you to the panel, very interesting discussion here this morning, especially as it pertains to IT. I'm Lieutenant General, Retired John Morgan. Did a lot of work within NATO, big data, information sharing, now leveraging some of that capability in the healthcare space, EagleForce Associates. And my question up front, in the objective for this panel where it talks to tools that may support improved clinical care by providing real-time information, two-part question to the panel and that is, can you be very, very descriptive about what real-time truly means? And it's connected in this way, since data is inherently latent and hosted in a number of disparate systems outside of existing EMRs, what do you see is a realistic timeline for universal EMR bi-directional sharing of PDMP data with pharmacy systems and linked to a master patient index? So no matter wherever the patient is, you see them in real-time inside of the healthcare system. Thank you. - Great, thanks. Peter, do you wanna take that? Or, Jim? - I'm happy, oh, go ahead. - Go ahead, I don't have good estimates of the time in terms of seconds. But in terms of five seconds, 20 seconds-- - [John] Considering the fact that what Dr. Gottlieb said this morning that 50 people were dying per day due to opioid abuse and another, for every death, another 60 individuals are affected. - It's a good question, and I'll turn it over to Dr. Huizenga in a moment, but I did wanna comment on PDMP integration with pharmacy dispensing software. There's been work on that that started courtesy of some SAMHSA grants to several states. In particular, Ohio with the Kroger pharmacy chain so that when the pharmacist opens the patient record, the PDMP report is called automatically and I think there are other pharmacy chains that are working with state PDMPs on that. The pharmacy chains of course, would prefer to work with multiple states at a given time, not one state at a time. But that work has been happening at any rate. Now in terms of a master patient index and any record would call all the other data sets, that's beyond my scope. - I can try to address some of that for you. Latency of data into the system. Pharmacies communicating with the PDMPs. It varies. It's mostly coming down to perhaps a 24-hour latency at this point. On average, I would say. We've migrated over time from when it used to take days or weeks to get data into the system to now it really taking hours. Some states it's actually real-time. I believe Oklahoma's basically has a system where it's real-time. So there's no latency there. And that's probably the model that we need to move towards, is no latency, get the data into the PDMP. The latency from the PDMP into clinical workflow, second or two at this point. At least that's what available. If you have to go and search for the data yourself and go to the website and all that, that's a four-minute process, but if you use an automated, integrated solution, it's a couple of seconds. So effectively negligible, 'cause most of the time it's not the provider even waiting for a couple of seconds, it's having it automatically while the patient is being moved through the system. And that MPI, yeah, MPI is tough. It is. We do run a system in associates with the National Association of the Boards of Pharmacy called PMP InterConnect. It connects 44 or 45 states where the PMP ID'd PMP data. So effectively it's all of the, or 44 states where the PDMP data searchable with one step. But it does require an MPI, and we've had a lot of, we've been doing a lot of work in that space. And ultimately to have a good functioning MPI in the system, we need to make sure that as much data is being reported about the demographics of that patient into the system. And we need to bring in third-party data sources. So we're actually doing that this year, we're bringing in to our MPI, to link all these patients across these systems, we're bringing in credit header file data and other types of information that allow us to find and track someone through the years of interactions that they've had through that system. It's never probably gonna be a perfect solution, it's a matter of how many nines can we get to? Is it three nines or is it four nines? Is it 99.9% or 99.99%? When you're dealing with 100 million people that we have in our system right now, even 99.99 leaves lots of room for improvement. - [Regina] Thanks. - Just add one. So Oklahoma's had their point of sale data submission for several years now. They were kind of unique in that they went through a whole process that involved pharmacists across the state and they ultimately agreed to do that in light of the problems in Oklahoma. Now, it was quite costly for pharmacists to change over their software to do that. Kentucky, in light of Oklahoma, did a study, a cost-benefit study, as to whether point of sale would be worth the extra cost versus end of next business day submission. And they concluded that end of next business day, in terms of a cost benefit assessment, was probably the better way to go. And I think most states have followed that lead. But anyway, there would be more information in terms of, from them, about exactly what they found. - So we have time for one more quick question and one more, and a follow-up. - [Alex] Okay. I'm Alex Rich, I'm gonna be as quick as I can. May be a deeply unpopular question. But, in the beginning of this field that we now call epidemiology, there was a cholera outbreak and at the time, we thought that cholera happened to people because they were poor and stupid and dirty. And if we had big data analytics back then, that may have been the outcome, that may be what we thought was a predictive factor. John Snow went in and diagrammed actual deaths and what he found was that the real predictor of cholera vulnerability was having two thumbs and drinking water. So in an era where we had a concerted effort to push prescribers to the right, to prescribing more, that co-opted academia, that co-opted state boards of medicine and federal regulators, all shoving them toward prescribing more. Why are we exclusively looking at patients when we think about risk when a patient may come in with the same condition to two different physicians of the same specialty and be prescribed opioids at a far different rate. There's nowhere that I see for a prescriber to see if they are an outlier in terms of what they're gonna give that patient, I think that's important, so I'm just kind of baffled as to why we're not looking at that. - So, having been a payer, I think you're gonna hear some about this a little later today. But payers really are looking at this and providing some feedback very directly to prescribers. And the flip side of your question is, okay, so we find out here's the doctor, this is how they're practicing. The doctor can look at the patient, can query the PDMP, find out that they have a number of, here's a patient who's got addictive disease, now what? And what we far too often see is that the doctor, having no knowledge and no skills, discharges the person from their care for violating their treatment contract with no referral and no follow-up plan. It's the only disease state where we make a diagnosis and discharge patients. So the information is only as good as how we can, how we make it knowledge and decisions on it and then how we can act on as well. - Thank you. Go ahead. - Is there time, yeah? Great question. And we have, in our doing provider assessments alongside patient assessments, and we have worked with some states to develop some tools to look at risk from a provider perspective. And they're there, and they can be used. We also are bringing relative performance indicators into the PDMP so providers can reference them and apply them to their situation. But I do wanna just take a moment to say that we have to be incredibly careful here. When we start looking at provider behavior, certainly we have to, but the double-edged sword here is that the providers who are willing to take on the most challenging patients, the patients with the worst chronic pain and the worst psycho-social background that just makes them incredibly difficult to take on, or the providers who are willing to take on those patients who have the worst form of addiction, who continuously relapse. When we look at provider metrics, those providers who are doing an incredibly important service and really extending what they're willing to do to try to help these people, are likely gonna be highlighted by these provider assessments and these risk models. So we have to be very careful. We have to be very careful to not drive providers away from being willing to do what they really truly believe is necessary for the patient. Some patients need those people that are willing to take those risks on their behalf. - So I don't wanna keep people from lunch. Our panelists will be around and I'm sure happy to answer any additional questions that you might have, and I wanna turn it over now. - Thank you, everyone. - Thank you very much, those of you. - My thanks, too, to Regina and the entire panel. We are taking a break now till 1:30, we'll be back on at 1:30 p.m. to talk about some of the health system roles and consequences of actions that they're taking to help manage the opioid crisis and improve patient safety. Thank you all very much. Great. All right, again, I'd like to welcome all of you back on behalf of the Duke-Margolis Center for Health Policy, I'm Mark McClellan, the director of the center, and I'm very pleased to have the opportunity to moderate this next session on provider and health systems approaches to manage opioid access and improve patient safety. So we've talked about a lot of topics today that bear on this goal, including prescription guidelines and how they're being applied in a variety of settings and systems. We've talked about risk assessment and monitoring tools to help support safer prescribing. We've talked a lot about Prescription Drug Monitoring Programs. This afternoon we want to focus on how these support systems and other support tools can interact with health systems capabilities to help address opioid access and improve patient safety. And just as with our discussions this morning, this is not just about presenting some ideas but thinking about how we can improve the evidence base. What kinds of measures are available? Whether it's prescribing rates or unusual prescribing patterns or actual outcomes like the occurrence of addiction or overdoses or the control of patients' chronic pain or other types of pain. So we wanna focus on, continue to focus on evidence as well. And we're talking about health systems today, you can from the people who are on the panel with me, we're really talking about a wide range of health systems in the United States, ranging from large, integrated systems to, as we've heard about this morning, individual practitioners who are working in the community. And we want to address these issues of tools and supports that health systems can bring, recognizing the very diverse nature of health care in the United States. So we discussed a whole range of components of a comprehensive strategy that could include provider outreach and education, prescribing and dispensing limits, monitoring, risk assessments tool, risk assessment tools, other kind of interventions geared at changing provider-patient behavior and again, very importantly, tools and data systems and analytics that can help assess the impact of these kinds of reforms. And to discuss all of this, we're very pleased to have four panelists with us who have implemented some comprehensive strategies and are also continuing to work on innovative approaches to help shift prescriber behavior and get to this goal of safer approaches to opioid access while still providing effective pain management. So we're gonna start out with some brief remarks from each of the panelists and then, as we've done in our previous sessions, have some discussion on the panel, and then we'd like to hear from all of you. Wanna again remind those of you in the room and everyone who's joining us online about the ability to contribute to discussion both at the microphones here in the room, but also through our hashtags at #SafeRX and #OpioidCrisis. So we do wanna hear from you as we further this discussion. Some of the topics that we think will come up during this panel include: What strategies and tools are being used by health systems to manage opioid access and improve patient safety? How are different types of health systems identifying risky prescribing or patients at risk of adverse events and have these strategies been confirmed through evidence, validated through evidence? What kinds of intervention strategies are being used to reduce risks and improve patient care? What are the emerging best practices or lessons learned, and what are the next steps to take based on the state of the knowledge and the state of experience in this diverse set of health systems in this country? So I'd like to start by introducing our panelists and then turn this over to them. First, Thomas Emmendorfer is the Deputy Chief Consultant for Pharmacy Benefits Management Services at the U.S. Department of Veterans Affairs, thanks for being with us, Thomas. Michael Kanter is the Regional Medical Director for Quality and Clinical Analysis for the Southern California Permanente Medical Group, and he's also Executive Vice President and Chief Quality Officer of the Permanente Federation. Jessica Kattan is the Director of the Primary Care Integration Unit for the Bureau of Alcohol and Drug Use in the New York City Department of Health and Mental Hygiene, working with a lot of community-based providers there. And Kit Delgado, who is Assistant Professor of Emergency Medicine and Epidemiology, the University of Pennsylvania, Perelman School of Medicine. So, as before, we're gonna have some opening comments from each, then the discussion and Tom, to you first. - Thank you, Mark. So the Department of Veterans Affairs, as a quick overview, we have 168 medical centers and over 1,000 outpatient clinics. And chronic pain in the United States, over 30% of all Americans suffer from some form of chronic pain. And our veteran population, it's a much higher incidence, between 50 to 60% of our veterans suffer from some form of chronic pain. So, in 2013, the Department of Veterans Affairs implemented the Opioid Safety Initiative at every one of our VA medical centers. And this article is from JAMA Viewpoint, by Dr. Gellad, Drs. Bernie Good, and Dr. Shulkin, and it highlights four broad strategies that VA has used to address the opioid epidemic. And that's education, pain management, risk mitigation, and addiction treatment. And just gonna introduce those concepts of what we've done in the Veterans Affairs in the opening remarks here. So, some of the programs that fall underneath the Opioid Safety Initiative that the VA has is our Overdose Education and Naloxone Distribution program and as of December 31st, 2017, we've dispensed over 125,000 naloxone kits to our veterans and we have reported 172 reversals through 2016 with those kits. And then, the Academic Detailing is a program where we have specially-trained pharmacists that go out and target some of our higher-risk providers or some of the providers that we wanna target, focusing in around maybe a morphine-equivalent daily dose or opioid use disorder or specifically the opioid Education and Naloxone Distribution program. We've heard the State Prescription Drug Monitoring Program. VA currently transmits data to 48 states and the District of Columbia. The state of Missouri is working on bringing one online. And then we have one state where we're just working with some of the technical issues so we can restart the transmissions. And along that lines in the last panel, a lot of discussion around the PDMPs. In VA, we've had over 2.6 million documented progress notes in our electronic health record of where our VA providers or pharmacists or health care providers are accessing and reviewing the PDMPs. Our Medication Take-Back Program is something that we're very proud of. We have over 100 onsite locations for receptacles where veterans can bring in their medication and dispose of them in a safe manner. And in addition to that, we have the take-back envelopes, which are provided free of charge to our veterans. And, if you can imagine 20 elephants in this room, we've brought back, so far we've had over 63 tons of medications returned through this program. Sorry, just trying to, my eyes aren't quite that good for that screen way back there. But the Opioid Therapy Risk Report, this also was talked a little bit about some the strategies that were used. So this is an online available dashboard that can be used for our providers while they're with our patients, and it pulls in all the viable, or all the important clinical elements into one place so when they're having that visit with our veterans. And then substance use disorder, Medication Assisted Treatment is available to veterans at all our VA medical centers. The last panel, this was mentioned as a tool, this was developed by our Office of Mental Health. They've done a great job with this, this is the Stratification, it's the STORM tool. It's the Stratification Tool for Opioid Risk Mitigation, and it brings in a lot of the different variables from our electronic health record to help our VA providers identify patients who may be at risk when they're on opioids to try to prevent and reduce that risk. Complementary and integrative health, over 1,000 VA providers have been trained in acupuncture. And then, we have an Opioid Safety Initiative Toolkit that's available out on our VA internet that provides a lot of different educational resources. And then we also have the Patient Aligned Care Team goal which promotes the six essential elements of good pain care that we wanna see in VA. So how do we measure this? We do have an Opioid Safety Initiative Dashboard. We focused in on being able to make the totality of all opioid use visible at all levels in our organization at the national. A VISN for us, a Veteran Integrated Service Network, that's a region, there's 18 regions in the Department of Veterans Affairs. The medical facility, the prescriber, as well as the prescriber and patient. And we only wanted to develop a few core key clinical metrics because we wanted those to be able to be very visible and easy for folks to remember, and we wanted them to focus in on utilization, how much opioid utilization do we have in our system? We wanted to have a safety metric, and for us that's how many veterans are on an opioid and benzodiazepine together? Then we wanted a monitoring metric, and for us that's long-term opioid patients greater than or equal to 90 days with a urine drug screen performed in the last year. And then finally, how high of a dose. So for us, we look at Morphine Equivalent Daily Dose Stratification, and we focus in on greater than or equal to 100 morphine-equivalent daily dose per day. And this is a national report that's updated quarterly, so it's a little bit different focused than some of the other tools that are more provider facing, which is the STORM and the OTRR report. And this just shows the results over time. And I think an important component of this is we did not set any target measurement goals for any of these metrics because what is the right percentage? And we didn't want to end up with some unintended consequences, so the main point for all of these slides is just to show the nice, steady decline over time with the different metrics. So all of these slides begin quarter four fiscal year 2012 and through quarter four fiscal year 17. So percentage of veterans on an opioid has decreased by 41% which is 280,000 fewer veterans. For our opioid and benzodiazepine metric, same type of trendline, it's been a 70% reduction with just over 85,000 veterans. And for our veterans on a long-term opioid therapy with a urine drug screen, you can see how that's gradually gone up over time, too. And that's up to 89% now. Veterans on long-term opioid therapy. This has decreased by 47% with just over 203,000 fewer veterans on a long-term opioid therapy which we defined as greater than or equal to 90 days. And then finally, the number of veterans on greater than or equal to 100 morphine equivalent daily dose, that's decreased 60%. It's now down. It started at just about 60,000 veterans, we now have that down to just under 24,000, which is a 60% reduction. So, making pretty good progress there as well. And then, whoops. That's my last slide. All right. (audience laughs) - Thank you very much, Tom, thank you. Michael, you're up next. - All right. Thank you. It's great to be here, I wanna thank the Duke-Margolis Center for inviting me. Gonna try and go through very quickly some of the work we've done at Kaiser Permanente and just as a brief refresher, Kaiser Permanente's an integrated delivery system with eight regions. The medical group is separate from the insurance company hospital arm of the organization. There's eight separate medical groups that take care of the patients within Kaiser Permanente, which has the pharmacy, the lab, most of the clinical care within one system connected by a single electronic medical record. So, having told you all about what we do, there's really four pillars to this thing to think through. There's physician education and support and it's more than support, it's creating a system around the doctor to actually move practice change, patient education, an understanding of the patient's safety aspects, and then there's also an understanding that this, unlike many of our quality initiatives, impacts the community outside of our organization. What you're gonna see on this slide was published earlier in 2017, I guess the year changed. But Journal of Evaluation in Clinical Practice, look this up if you're interested. What it does is it delineates a multi-step, multi-year program we had to improve our opioid prescribing. You'll see some high points here, 30% reduction in prescribing high-dose opioids. 98% reduction in giving prescriptions with more than 200 pills. 90% decrease in opioids combined with Soma and benzodiazepines. 72% reduction in long-acting opioids, and a 95% reduction in brand name products when generics are available. So good success, one of the points on this, as prior speakers were talking about, how do you measure this in a more granular way that might indicate something more than just gross opioid prescribing, and I think these target perhaps some high-risk behaviors better. How did we do this? It's actually complicated. I'm gonna actually skip this, but it's just showing every region has in a same downward projection and that we have, throughout our organizations, spread these practices with really a 40% reduction of people on over 90 morphine milliequivalents over a three-year period, roughly four years if you figure the starting. So how did we do this? And one thing I wanted to emphasize that I haven't heard is physician leadership. And this is key, you can put in all the systems and all this stuff, but somebody actually needs to make a decision that this is important, that it's a priority, and unless you're in a single-physician group, it requires one or more leaders to prioritize this up, lead it, model it, and I think that's one of the key ingredients that people should not forget. The other thing I wanna talk about, EMR is very helpful, it allows doctors to know not only what they're prescribing, but it can convert into how many morphine milliequivalents, it can look at other drugs like the benzos, put in alerts, so it's very key, and in our system, you can see what all the other physicians are prescribing. So it's not a substitute for the state databases, but it's a good facsimile when that wasn't available, and it's in some ways, perhaps, more accurate. In any case, training is key. It's not just teaching people, don't prescribe opioids, but it's really, there's a physician-patient communication skill that needs to be taught, tapering algorithms and approaches, this is complicated, there's no one size fits all. We needed effective communication with pain experts, and we actually created work agreements between primary care and specialty care on how they're gonna work together and manage patients and co-manage patients and when do patients go to addiction medicine versus pain management versus other places. We did promote use of shared decision making tools, medication agreements, opioid agreements. A key is feedback, performance feedback to the doctors so they know how they perform relative to their peers on a variety of measures with nobody wanting to be the outlier and they can kinda compare themselves to their peers. Let me move on here. Little bit more on this. I think the pharmacy has a key role in this thing as well, and I wanna, I can't say enough about the role of a pharmacist in this program. They're experts in pain medication and medications in general and drug interactions. They can educate the patients, they can look for unusual patterns of prescribing. We had a program where they were stopping fentanyl scripts in opioid naive patients and rechecking and holding the line until they were sure this was a safe practice. And they actually help with academic detailing of our physicians. We also used a lot of periodic visits with physicians who might be outliers in their prescribing patterns and did one-on-one education with those doctors. Patient safety was always emphasized. There's some interesting issues here with drug interactions that we, I think I showed you some data, got our arms around. I wanted also to make sure that we were not just cutting down drugs, but we actually have a whole potpourri of pain management techniques. Back pain is key, and there's a back pain algorithm and we're setting up back pain clinics, but CBT, cognitive behavioral therapy, acupuncture, mindfulness training, pain management classes or programs that are all part of the armamentarium here. One thing I did wanna mention, ED, this required some work in the ED to change prescribing patterns there and it actually has some community implications because we worked with outside EDs in Los Angeles County as an example. 'Cause if one ED starts to lower what they do with opioids, and another is giving it out freely, patients quickly figure that out and start going where the drugs are made available. So this really, I think, optimally requires cooperation of all of the emergency rooms and health systems in an area so patients get a reasonably uniform approach to opioids in the ED. The other thing we were doing, like the VA, is checking urine tests, looking for whether evidence of patients diverting drugs and seeing any unusual patterns of prescribing that might suggest diversion. One other barrier that we addressed, that I haven't heard anyone talk about, is patient satisfaction. And this is actually a concern of our doctors, if I start tapering opioids, will my patients get mad at me? We have patient satisfaction surveys that are reported out on a doctor level, and they take these very seriously. So we actually did a study comparing patients on high-dose opioids and whether they were tapered in a visit or not, looked at the satisfaction, and you can kinda see there overall, a minimal reduction, but when the patient and the doctor knew each other, it was the patient's primary care doctor, no difference in satisfaction scores. So this reassured many of our physicians who were concerned. I think it also may address a little bit the issue of are we ignoring the patient's pain level and needs, but at least based on a satisfaction survey, does not seem to be the case. And I just wanna thank you for your time, I'm sure there'll be more in the dialogue and Q and A, thank you. - [Mark] Great, thank you. Thanks very much, Michael. And next is Jessica. - All right, good afternoon, everybody. I will be presenting on the New York City Health Department's experience with public health detailing on judicious opioid prescribing. So first, I'll briefly mention our overall city response to the opioid crisis and detailing on judicious opioid prescribing is part of this. Our city response is known as HealingNYC, and represents a $38 million investment announced by the mayor in 2017. The goal is to decrease opioid overdose deaths by 35% over five years. HealingNYC is composed of 12 strategies and is a collaborative effort among multiple agencies. The New York City Health Department-led strategies include preventing opioid overdose deaths, preventing opioid misuse and addiction, and this is where judicious opioid prescribing detailing efforts fit in, and connecting New Yorkers to effective treatment. Oh, there's the red box. So we really view judicious opioid prescribing as prevention with the goal of preventing unnecessary exposure to opioids and new cases of addiction. Our judicious opioid prescribing recommendations are based on guidelines that our department previously issued, and they include prescribing less often, for a shorter duration, the lowest effective dose, and avoiding co-prescription with benzodiazepines. And one of the main vehicles for our prescriber education on this topic is public health detailing. So what is public health detailing? Our detailing campaigns are modeled after the pharmaceutical sales approach and involve selling good health. The visits are conducted by detailing representatives, or reps, and consist of brief one-to-one educational visits with health care providers and their staff. During the visits, the providers are presented with recommendations, resources, and tools. The reps perform what we call a total office call, meaning that we try to talk to as many support staff as possible in addition to the providers. And this is very important, since we know that support staff can be instrumental in changing systems within an office. The detailing visits are structured using these seven basic steps. And although the visits are structured, they also allow the reps to tailor the visits and the conversation to the needs of the provider. Our campaign strategy starts with using local data to select targeted neighborhoods, including overdose mortality data and aggregate prescribing data. We aim to reach 1,000 physicians, nurse practitioners, and physician assistants through each campaign. And the campaign duration is eight weeks. The reps attempt both an initial and a follow-up visit for providers. And we really take a public health approach to the campaigns, meaning that we try to reach the majority of providers in the targeted neighborhood, with the thinking being that if a large proportion of providers make small changes to their everyday practice, this could potentially result in meaningful changes on the population level. We also incorporate promotional events into the start of the campaign in order to raise community awareness. And examples include issuing a press release to generate media attention, or our Health Commissioner delivering medical grand rounds at a hospital at a targeted neighborhood. And the overall aim of the visits is to promote three key prescribing recommendations and deliver provider and patient resources contained in what we call action kits. So the action kit, which is shown here, and also in my hand, is a folder consisting of clinical tools and resources along with a letter from the Health Commissioner stating the three campaign recommendations and urging the provider to adopt them. One item promoted in the kit that I'd like to highlight is a free mobile app that we developed called OpioidCalc. OpioidCalc quickly calculates total daily morphine milligram equivalents, or MMEs, to help assess high-dose prescribing and overdose risk. An alert pops up for over 100 MMEs, as seen in red. We've conducted a total of three detailing campaigns on judicious opioid prescribing. The first one in Staten Island in 2013, followed by the Bronx and Brooklyn. Through these three campaigns, we reached a total of over 3,000 prescribers. Evaluation demonstrated knowledge change about key recommendations for each campaign, and notably, we saw decreases in high-dose prescribing in Staten Island. And this slide shows in more detail the greater decreases in high-dose prescribing in Staten Island after conducting the detailing campaign in this county. We learned several things through conducting our campaigns. First, that public health detailing works. Also the importance of using a data-driven approach to target and evaluate campaigns. That it can be challenging to identify a gold standard list of healthcare providers. It's critical to raise community awareness with promotional events prior to the campaign so that providers expect the detailing visits. And finally we learned the utility of framing the visits as a partnership on an important community problem. Thank you. - Okay. Thanks very much, Jessica, and Kit is next. - Just wanna thank Duke for having me. I'm Kit Delgado, I'm an emergency physician and a researcher and I'm motivated by this issue 'cause almost every shift now in my ER in Philadelphia, not a shift goes by where I don't treat a heroin overdose. And I often ask my patients, when did your addiction start? And inevitably, usually starts with an opioid prescription, sometimes 30, 60 pills for a simple injury and sort of motivated to do something about that. And my approach is to take a behavioral economics approach, nudge safer prescribing. So we tested this out in our emergency department, and the focus here is really on acute prescribing, we've heard a lot about chronic prescribing. But I think there's a lot of gains that can be made pretty quickly in acute prescribing. So in our emergency department, when we switched over from our old medical record to our new medical record, which is Epic, we included a default of 10 tablets for an opioid. And this study was recently published this past month. So basically, I'm sitting at my computer, I wanna send someone home with a prescription for Percocet for their fracture, I type in Percocet into the EMR, and I get a preference list where the first listed default is 10 tablets. I can just click it and sign it and then it prints out. I can go to the second listed default there, which is 20, or I can opt out in the order and type in whatever number I want, or I can go to a preference, sorry, database lookup and look at all the different dosing and formulations for that drug. So this is what happened when almost overnight, the number of prescriptions for 10 tablets shot up from about 22% to 43%. The number of prescriptions for more than 20 tablets decreased, which was good. But we also noticed a couple unintended consequences. So, we were already prescribing very low amounts, I think because we had a champion in our emergency department who was really banging the drum on this issue, so we're already at a guideline-consistent amount. And so we noticed that when we set the default for 10, it actually nudged people who were prescribing less than 10 up to 10. So the implication there is that you wanna set the default at sort of the lowest baseline prescription. We also noticed that there was actually an increase in prescriptions for more than 20 tablets, and it turns out, if you opted out of the default and went to the database lookup, there was a health system default that got introduced that we didn't even know about, which is for 28 tablets. So all these extra new prescriptions were for 28 tablets. So the bottom line is you need to nudge carefully. So, we're very excited about this in the emergency department and we looked throughout the health system and we saw that actually, it wasn't the EDs that were prescribing the majority of medications, it was really post-operative prescribing where the acute, in our system as it probably is everywhere, orthopedics was really one of the leading prescribers with a median quantity of 50 to 60 tablets. So I went to the chief of sports medicine, I said, hey, defaults worked in the ER, do you wanna try them for your sports medicine procedures? He said, hey, Kit, that sounds great. But I wanna make sure that we don't harm our patients, that they stay happy, that they don't go to our competitors, and I really don't want them to come back early for refills. And I said, okay, I'll get my research assistant to call all your patients a week after their procedures and just find out how many tablets they're taking. He said, all right, sounds good. This is what we found. At the top, we found that the median quantity prescribed was 30 tablets, and when we asked 'em how many tablets they took, the median taken was four. About almost a third took zero, and only a very small amount took more than 20. The median left over is 27, and the one patient that got 90 tablets for a knee arthroscopy only took one tablet. So, there's a problem with right sizing. So we showed him the data, he was very excited to do something about it so we implemented a smart set where we defaulted patients to acetaminophen, ibuprofen instead of Percocet, so remove the opioid from the Tylenol, and then included options for oxycodone by itself for 10 tablets and 15 tablets. And when we did this and showed him the data, so the little arrow's where the intervention is, the orange line is the quantity prescribed. Basically, within about a month, with very little effort, we basically halved the quantity prescribed down to one of the defaults of 15, and there was no change in the median number of tablets taken. So we decreased the number of tablets by 15, decreased the number of tablets left over by 10, and people are still taking around three tablets with 75% taking 10 or less. And then we also asked them when we called them, how would you rate your ability to manage your pain? On a scale of one to 10, 10 being the highest, and basically, we're able to capture that they're doing just fine, if anything, their ability to manage pain is better. We looked at this for other procedures, like ACL repairs, and we're actually finding that this multi-modal analgesic approach actually reduced the amount of opioids taken and people are having improved pain outcomes. And then you can see pretty dramatic shifts in where we were prescribing. At the top, the number of prescriptions for 15 tablets went from 8% to 55%. For more than 30 tablets went down from 60 to 22%, and for more than 40 tablets from 21 to 10%. After the intervention, there was actually less patients who had zero to five tablets left over, so this was very helpful for persuading the surgeons that they weren't gonna induce a bunch of early refills. So if we scale this nationally, this is data from the Optum data set, which is comprised of Optum United Health claims. And we just looked at the quantity prescribed for the same procedures, and this is the distribution. And if we implemented the same approach, we could reduce prescribing by a lot for this procedure and literally save hundreds of thousands of pills from getting out into the community. So basically the bottom line, we think that these defaults are a powerful way to change prescribing in almost a passive fashion. They preserve clinician autonomy. The key, at least for post-operative prescribing, is that the surgeons need data on how many tablet their patients need for given procedures. We're scaling this up across all of orthopedics, and now across the entire health system. Getting patients to consent to us to texting them after their procedures to check on how they're doing and sending this information to us via text. And we're gonna use that to develop more nuanced order sets. And then actually go after driving down the prescription rate, particularly for the opioid naive by showing them, hey, you probably don't need to take that many tablets, or any tablets for this procedure and try to keep those patients opioid naive. That's it, thanks. - Great, thank you very much, Kit. I wanna thank everyone for their presentations. We've heard a lot about what a wide range of health systems are doing to manage opioid access to improve, I get to say, for prescribing. I'd like to start out by going back to one of the comments I made at the beginning that we have a very diverse healthcare system in this country, with some big, integrated systems like Michael's, but also a lot of practitioners, including in New York, I assume that are still very small practices. And one of the themes that we've heard with a lot of the effective tools described today is that a team-based approach is really important as opposed to thinking of an individual clinician on their own as being able to effectively manage opioids. So working with decision support tools, working with pharmacists, working with other members of a care team. Wondering if any of you have comments about, particularly for the smaller, less integrated provider systems, where you all see the best opportunities for improving their supports and achieving some of the results and getting some of the benefits of the programs that you all talked about today. - Well, I'll take a stab at that. Can you guys hear me? I hope so, okay, great. I think every practice is different. With the accountable care organizations are I think still present and forming and I think there's an opportunity to virtually integrate without something like the VA or Kaiser Permanente, and people I think need to get creative with how do they associate with their local pharmacy or the physical therapist down the hallway or whatever it is, I think that's one option. The other that I think will move the dial a lot is NCQA has some new measures on opioid misuse and as those get publicly reported, the various health plans and insurance companies are gonna have some incentive to help coordinate the individual physician practices and create some kind of system for them. Leadership is still key. I think if a physician really focuses on this, they can make things happen. If they have the will, they'll figure out a way. - Agreed. I think we heard some of the community approach down towards the end and how the collaboration works well and getting the different disciplines together. I know in the VA that's been a key component of success, having the buy-in from senior leadership all the way down to our medical center leadership and making that commitment to appropriate pain management and having those alternative therapies available. - Yeah, I mean I think it's helpful to have guidelines that are specific to the setting that you're dealing with, so it's certainly, I think, the emergency department most prescribing I would say, and the evidence shows it's guideline consistent right now, I think that the risks are salient to us. I think for colleagues in surgery, the decision making about an opioid prescription, it was pretty remarkable, I talked to our chief of sports medicine, I said, how do you write prescriptions for your patients? And it's, in the emergency department, I see a patient with pain, I say, you need about this much or nothing. For surgeons, they get a list of all the patients who are having surgery next week, and then just go down the list clicking prescriptions for all those patients, it gets put in a folder, and then they show up, they get the folder as soon as they come out from the perioperative area with their Colace prescription, their Vicodin prescription, and I think there just needs to be a different mindset with the decision making there, and I think we really need to show them data that if they reduce the prescribing, it's not gonna cause them a headache, because surgeons, what they wanna do is to operate and not get called with patients who need refills. - [Mark] Jessica, you spent a lot of time with smaller practices in the community, any insight from that work about what they most need to do that? - Yes, so through our detailing efforts, we try to detail a large proportion of providers in the community, including both large practices and smaller practices and some of the larger practices that were part of hospital systems might have already had their own opioid initiatives being implemented, and so what we found was that the smaller practices were often very appreciative and very open and welcomed the Health Department coming there and providing their time and their resources to help partner on this problem. - Mm-hm, great. And keeping with this theme of team, strengthening teams and addressing opioid safe use, several of you emphasized the role of pharmacists and having not only expertise around the properties of medications, but also being right at a focal point connecting people with the prescriptions that are written. Any further thoughts about how to strengthen pharmacists as part of a care team for effective management of opioid prescribing, particularly, but maybe not limited to, smaller practices? What else can we do to strengthen the role of pharmacists? - I guess since I'm a pharmacist, maybe I'll go first. (laughing) So in the Department of Veterans Affairs, one of the keys to success is one, in our pharmacies, our pharmacists have access to the full electronic health record. So we have access to the progress notes, we have access to the labs, we have access to the physicians are often right down the hall from us. And we also have, a lot of VA pharmacists have a scope of practice. And so by becoming a member of that team, I think one of the key components is to have access to the full electronic health record, and that's a key issue. A lot of our prescriptions now, almost 80% of our prescriptions in the Department of Veterans Affairs go through our mail-order pharmacy. So over the years, that's allowed us to shift our focus from more of a... Since about 20% of prescriptions are dispensed at our medical center, that frees up our pharmacists' time away from being behind a traditional counter, and we're out, embedded into the primary care clinic teams, the mental health clinic teams, and we can help use our resources and our skill sets to help support the full team. - Not too much to add, we function a similar way with our pharmacists. I would point out that pharmacists play a key role in pain management clinics, and in multi-disciplinary teams when they're formed if there's a pharmacist practicing not associated with a delivery system, I think you still have the professional duty of the pharmacist to make sure whatever they're dispensing is a sound prescription, and the patient understands it, and the doctor is prescribing properly. So I think there's a higher burden on the pharmacists, but there should still be these discussions with patients, discussions with the ordering physician when something seems amiss with the patient or the prescription and chronic high doses I think is a good opportunity for pharmacists to be proactive with their physicians even if they're not in a integrated system. - [Mark] And does this, or have these steps in your systems required putting new resources into pharmacy services, or as you said, Thomas, this has coincided with a shift away from other types of prescription filling with mail order. Just trying to get a sense of how much additional resources are needed and maybe how to-- - We have the same mail order gimmick going on, it's really safer for patients to have it mail ordered and filled robotically than by human beings, and it does free up a lot of time. But if we had to make an investment, we would've, and I just can't overemphasize the value of the pharmacists. - In the Department of Veterans Affairs, our mail order pharmacies have been around since right around the late 1990s, I believe. And then in the last few years, probably about three or four years ago, VA did make an investment in bringing up very similar to what Jessica's talking about into our academic detailing. So all of academic detailing with our specially-trained pharmacists, that's throughout the whole country now. - I wanted to highlight a couple of issues related to electronic data system support for strengthening these provider teams and maybe sticking with pharmacists for a minute more. In the last panel, we heard about the increased used of Prescription Drug Monitoring Programs and Thomas, you mention that those are integrated into the VA's work as well. For pharmacists to be an effective part of this tool, though, are there opportunities around linking pharmacy e-prescribing systems to a Prescription Drug Monitoring Programs? What are some additional tools that would help with the pharmacist's role in these efforts? - [Michael] You're the pharmacist, go ahead. - So, in the VA, we do have to go outside of our electronic health record to access the PDMPs. I think as time goes on, we'll see. We heard a lot about the technology barriers that are being brought down over time. Obviously, being a national integrated healthcare system, having some sort of standards, as time goes on, I think we'll see where it goes, but I guess the short answer to the question is if it could be integrated into our electronic health care system record, yeah. That would be helpful. Because it is a manual process right now for our health care providers to look up the information from the PDMPs. But it's very helpful. - I would agree, I might point out that in an integrated system, we can see what at least people in our system are prescribing, and most of the patients on higher-dose opioids, it's not that they're running around doctor shopping and trying to get more, but most of the time it is the surgeon who starts a vicious cycle, giving too much, and then other well-meaning doctors give too much and there's nothin' in terms of, I'm not saying diversion doesn't happen, but statistically, more often it's just people not practicing carefully and so we end up with a deliberate, intentional overdosing, I shouldn't say overdosing, but giving too much opioids to patients and they're not doing the doctor shopping route. - And maybe sticking with electronic support tools, all of you talk about those in one way or another, the importance of data, the ability to use electronic prescribing systems and medical records to shift how physicians prescribe. Are there other tools that would be particularly valuable for clinicians, I guess I'm thinking of things like, we talked this morning about some of the challenges in identifying a high-risk patients, but where PDMP and other data may be able to help with that, maybe tools, reminders to physicians when their patients have been on for a while. Michael, I think these kinds of tools were a big part of what Kaiser accomplished, what else is needed for any of you, tools to highlight beyond just the default guideline-based prescribing tools that would be really helpful for physicians? - I think one of the things I did not mention is we do send out lists of patients on high-dose opioids to each of our medical centers to a multi-disciplinary team who reviews it and has case conferences where they, it's partly educational and partly care management, sort of like a tumor board where they'll review these patients and make recommendations. So, in theory, one doesn't need something directly in the EMR to do that. And that's been actually very valuable, particularly for the physicians less comfortable with how to taper patients, 'cause it is complicated. So that's one thing. The other that we should emphasize is patient education and the patients know what they're taking or should know, should know the drug interactions and should be aware of the hazards of opioids and have a plan if that's in place. I think that also assists the physician when the patient is more informed and has a same treatment goal. - [Mark] Other thoughts about tools for physicians? - Yeah. I think particularly for post-operative prescribing, I think although e-prescribing is legal in most places, it's not fully implemented, and one of the big barriers to getting surgeons to prescribe less is they don't want patients coming back for refills, particularly in a referral center where people come from very far away. And so the status quo is that if you need, if you were to prescribe a shorter course, say like five days, and they need extra 'cause they're still having a lot of pain, that patient would have to drive back to get a written prescription. With e-prescribing, they could send a prescription to that patient's pharmacy, wherever they live, and so I think increasing the uptake and decreasing the barriers of e-prescribing would help. Another one that's being used right now in our health system is just simple workflow. So like, using the data to identify, we now have a registry within Penn Medicine of everyone who's on chronic opioids, and so when chronic, a patient who's identified as a chronic opioid patient is going through the pre-operative process or contempt process, it flags them and says that they need a pain medicine consult prior to surgery to come up with a plan to manage them after surgery. And so simple workflow innovations would be helpful. - Can I briefly mention the OpioidCalc app? So the reason we developed this was at least in the past, calculating the morphine milligram equivalents has not been something routinely taught in clinical medical training, and so we wanted to develop a tool that would help clinicians do this quickly, for free, and a tool that would be accessible and easy to use. And so that's one example. - So we've talked a lot about tools for clinicians. On the patient side, Michael, you emphasized the importance of patient education, and it sounded like your focus there was on patients who have been prescribed opioids. Are there other tools that health systems can and should be applying to help with patient education? And maybe we can start first with some of the tools that would be used for individual patients who are getting opioid prescriptions, but I do wanna come back to something that many of you emphasized, is making this a community-level effort as well, engaging the public. - Yeah, I think what I was sort of referring to is the medication agreement, or an opioid agreement, which is an agreement between the patient and the doctor on how the drugs will be used and not used and what's the therapeutic goal. I don't know, is that patient education thing or a doctor education thing? To me, it's shared decision making, and so I don't how you'd classify that, but I think it's both parties' tool that we found extremely valuable. And it could be documented in the medical record. We can keep track of its usage. It creates a sort of standard discussion so we create some uniform, you have less unwarranted variation. To me, that's what's coming to mind. - So in VA, out on YouTube, believe it or not, we have three videos that surround our opioid overdose education and naloxone distribution. So those are out there for veterans or for the public to use. Our toolkit, our Opioid Safety Initiative toolkit, is also available on the internet, which has a lot of great provider and patient-facing resources. And then VA and DOD have a lot of joint pain management type of education materials that are also available out on the internet. - Kit, what works well for patients in the emergency room? - Yeah, so. From a guideline perspective, we're pretty emphatic about the need to keep opioid naive patients opioid naive. So I think that's in the guidelines, but we have PCORI-funded study, Zach Meisel, one of my colleagues in the emergency department, where we're testing different scalable ways to have that conversation. 'Cause often it can be difficult. It's very simple, just brief 60 to 90-second vignettes that you can click on on iPad, they're available on YouTube, that just highlight real patients and their experiences with getting addicted from a prescription painkiller and we're using that as sort of a sticky narrative to nudge patients away and sort of set expectations about the risks of opioids, and we have other studies of shared decision making aids that we hope to disseminate after our studies are done. - And in terms of engaging the broader communities around these kinds of effort, any additional comments about those kinds of steps? I know Jessica that was a big part of your work. - Sure. Well the past comments just made me think about our public awareness campaigns that we've done over the past several years. Coinciding with our first detailing campaign was a television ad featuring testimonials from New Yorkers whose lives or whose families' lives have been impacted by opioid analgesics. And since then we've also produced print ads about naloxone and television ads most recently about methadone and Buprenorphine as effective treatments for opioid use disorder. - And I would like to come back to treatment in a minute, but I guess again, sticking with patients, I guess the other way to reduce demand for opioids is to find more effective strategies for patient pain management. I think you all each touched on some of those in your brief remarks, and we're gonna ask you to expand on those a little bit. How much have they helped your sense in reducing the need or the pressures for opioid prescribing? - [Michael] You wanna go, Tom? - Actually, in your brief remarks, everything that Kaiser mentioned, VA is, I would say we're in line. - Yeah, I mean I view it as not sort of, it's the proper treatment. When we're talking about managing pain and we need to keep the discussion on pain management and not on opioid reduction. And to me that's a fundamental, it's easy to fall into the trap of opioid reduction, 'cause it's easy to measure, there's a lot of press coverage on it. But pain patients need to have multiple modalities and try multiple things to see what works best for them. Keeping things evidence-based, there's a lot of unproven pain management things out there which we're on guard for. But I just view that as part of a pain management program rather than something linked to a opioid reduction program even though, maybe historically, it's kinda come about that way, but we've always had acupuncture coverage and other modalities that help patients. There's actually a program we have that was co-developed with Stanford on living with chronic conditions that also teaches patients how to deal with chronic illness, and some of that's pain, some of that's not pain, but it's been also very effective and shown to actually make people's quality of life scores increase. - [Mark] Are there other modalities that you all would like to mention? - Well, I think. - Back pain, I mentioned, but it's really one of the common ones that get, besides surgery, it's the most common, but back pain, incredibly prevalent, incredibly easy to go from an acute, self-limited condition to chronic. We promote exercise heavily, and that's sometimes challenging, I personally get back pain and I make myself walk and not sit in a room like this all day long. So I think that there's specific high-need programs like back pain, migraine's another one. We early on found a non-insubstantial number of our migraineers were on opioids and had to stop that practice as well. It actually can make them worse, not better. And we've developed a whole migraine management program. - I think for surgery, and I don't work on this directly, but I've seen the results in our health system and other places, the enhanced recovery after surgery that you asked, and the sort of multi-modal analgesic protocols that involve nerve blocks, and other, regional anesthesia, and other types of multi-modal pain pathways, sort of to decrease the patients pain before they've left to go home. It actually dramatically reduces their pain and need for opioids after surgery. So that's very promising. And I think it cannot be emphasized enough that the biggest issue here is getting patients who have chronic pain adequate pain treatment, and I would say that the majority, or a good portion of the overdoses we're seeing now in the ER are a lot of patients who sort of in that fine line between chronic pain who, I think with the new guidelines, are getting abandoned by their primary care doctors and sent to pain medicine clinics, which are four-month waits. And I can remember very specifically a patient who had low back surgery then chronic pain after that. And a long wait for pain medicine clinic was sightseeing in Philadelphia, was in excruciating pain, couldn't go on, found heroin on the street and overdosed. And that was her first use. And so I think anything that we can do to increase access to chronic pain treatment is really what's needed now. The one thing that wasn't mentioned today was medical marijuana. Recent Rand study that came out last week showing that states that implemented medical marijuana dispensaries had like a 20% reduction in overdoses and new addiction treatments for opioid use. So it's something that could be considered as a policy, I'm sure. - And this leading to a last question and I wanna open this up to questions and comments from all of you here. You all, and then Thomas, you especially highlighted a few specific key measures that were indicators of progress or intended to be indicators of progress and safe and effective management of opioid access, and I think all of you talked about shifts in prescribing patterns away from very high morphine milligram equivalent dosing, shifts down in long-term use and the like. And, as Michael, you mentioned, there's some NCQA measures for health care organizations coming along those same lines on opioid use. But you also pointed out, several of you pointed out the importance of patient experience, pain control, and I guess the best measure we have of that is the patient satisfaction measures that you all talked about, I mean it seems like we're not, we don't quite have the full set of measures in routine use to match up with what the goals really are here for safe and effective opioid management. I wonder if you could talk a little bit about that. - Yeah, I mean the NCQA measures are gonna measure percentage of people on a high dose, I forget what cutoff they use, and then-- - It's along the lines that you'll-- - And then the percentage of people which we haven't talked about, percentage of patients getting opioids from multiple providers, which is something we haven't talked about, but it's a common sense kind of measure I think. The whole opioid epidemic started with somebody saying we need to measure pain as a fifth vital sign on everybody and make sure that people weren't in pain needlessly, and then everyone saw this is common and started giving opioids. So part of me says we should be measuring a pain scale, because it seems right, and part of me is very nervous of will that then make everybody with pain get put on opioids, and I think this is a great research question, is, does measurement of pain in a routine fashion increase or decrease improper opioid prescribing? I don't know, but the satisfaction we measured is kind of a bad measure. We're looking at things like, not just overdoses and deaths, but accidents, there's a lot of falls, there could be a lot of harms on opioids - Less extreme harms. - That are less extreme that aren't necessarily causally linked but they probably are that we don't measure. So I don't know if I'm totally answering you, 'cause I don't have the definitive answer in my head. - [Mark] Good thoughts. Any other thoughts on the panel? - No, I definitely don't have the definitive answer there, either. And so part of the reason I don't know if any health care system has kind of solved that issue. So one of the things that we did when we set up the Opioid Safety Initiative is from a systems perspective, is for each region to have an Opioid Safety Initiative or Pain Committee in every facility to have the same type of structure where you're bringing all the disciplines together. You're polling, kind of like your reference with the tumor board, and you know, and to remember metrics are just that, metrics. A lot of it is administrative data and the importance of building the infrastructure in your system around those metrics to promote the appropriate pain management. But getting to a good metric right now and how you would measure the pain outcomes, I would think that's a good research question. - Great, Tom, thanks. Thanks for the comments. We do have a little bit of time for questions from all of you, so I'd like to get as many as possible. If you could say your name and then keep the questions brief, that would be great, please go ahead. - [John] John Holaday, DisposeRX, long-time opioid researcher. I was speaking to Dr. McClellan in one of the earlier breaks about the need to get rid of leftover drugs in the medicine cabinet. Something that we've talked a lot about during the course of the day was 70% of the addictions and the overdose beginning with leftover medicines in the medicine cabinet. We sought a simple solution to this where you would take the vial in which those drugs are dispensed, add water halfway, pour in a powder, which is an edible powder, it becomes a very rapidly gelling matrix from which the drugs cannot be extracted. It takes one minute to do that. The drugs then are thrown into the trash, it's all biodegradable, and they can't be removed from this using the very clever chemistry that the street chemists would use if they wanted to try to get something out of it. If you want these, they're available from Walmart, they give them away with every opioid prescription, these little packets to get rid of the opioids. It's a simple solution. Thank you for the opportunity to talk. - Thank you. Any further comments on getting rid of excess prescriptions? Even Kit, with the steps you're taking, there's still a lot of leftover pills out there. - It's challenging. 'Cause I think even within our system, we're trying to have a safe disposal site, and that there's all these regulatory barriers to doing that, but I think from a behavioral standpoint, there's a strong endowment effect with prescriptions. You get something from your doctor, you know how hard it is to get, you're always worrying about that you're gonna have some other horrible episode, it's hard to get opioids, you don't wanna get rid of them. Until you can nudge people to dump it in the toilet at home, that's gonna be the easiest. Some other way that doesn't require them bringing it somewhere. - [Mark] Okay, I'm gonna go over here. - [Myra] Don't dump it in the toilet. (laughs) I'm Myra Christopher from the PAINS Alliance. My husband had a total knee replacement two weeks ago and so we've been doing pain management at our home and he got very good care. We still ended up with 25 oxycodone left over that we'll probably keep because of the difficulties with getting prescriptions these days. However, I wondered in the course of this as we were talking about managing it, in the past we've had prescriptions for steroids and some antibiotics where they were prescribed with a taper plan. So you would take five the first day, three the next day, whatever. Have you guys ever talked about or has there been discussion about prescribing opioids specifically with a taper plan in mind, I asked my husband's orthopedic surgeon yesterday about that, he said, I've never heard of such. So I'm just curious if that might not be something to think through. - I can say Michael Ashburn, who's our director of pain medicine, is trying to test this out with the orthopedic surgeons, something like a short burst of relatively high intensity up front, and then sort of tapering down quickly within three days, but I haven't seen any data on it, so I think it probably won't be used. - Probably also balancing that against the fact that a lot of people don't take that many pain meds after a procedure. Just like in your nudging, you don't wanna nudge 'em up on this. - And this would be for the more invasive type of surgeries, like a knee replacement. - Yes. - [Judy] Judy Racoosin, FDA. I also got up to talk about proper disposal. And I'm gonna have to take issue with what was just said by the last speaker or questioner, because FDA, on our website, has a very popular webpage and that's the proper disposal webpage. And we very carefully have considered what drugs should be disposed of with kitty litter or coffee grounds and sealed in a plastic bag and put in the trash and those for which the benefit outweighs the risk of flushing. And so, there's a small number of products, but it's primarily the opioids for which the benefits have been determined to avoid accidental exposure. I didn't hear when you guys talked about, actually, our VA speaker spoke about the take-back program and a lot of opioid that was returned that way, but as far as patient education, it seems to me that proper disposal is a key element of what that needs to be. - And easy disposal, easy and proper. - I think that's right, we have a disposal program, too. I think as we prescribe smaller quantities, though, the disposal problem and theory should get less importance relative to what it was five years ago when we prescribed a lot more, but you're correct. - [Judy] Dr. Delgado described today speaks directly to that point of there being fewer pills out there but there's still extras that are. - And I believe more retail pharmacies are adding the receptacles. The importance of the receptacles or some of the mail-back envelopes, like what we use, when those are returned and those medications are destroyed in an environmentally friendly way. And that's important. - Please, go ahead. - Thank you. I'm Lisa Hynes, I'm with the Pharmacy Quality Alliance, you be familiar with us, Dr. McClellan. - Oh yes. - [Lisa] We have a core set of medication measures. High dose, the first NQF-endorsed opioid measure of its kind, that's what NCQA modeled their measures after, in fact. And so I do recommend getting in contact with us, we have the multiple provider measures, we have a combination of the two concurrent use of opioids and benzodiazepines under NQF endorsement. Established in 2015, we have lots of data, happy to share it. Initial opioid use measures and I welcome ideas, collaborations, thank you for your mention of pharmacists, I'm a pharmacist, and this is just such a great presentation, so kudos to you. - Well thanks very much, and since I am familiar with PQA, - Thank you. - I would highlight they have done a lot of work around measures of appropriate use and engaging and supporting pharmacists as part of effective team management of the opioid problems. Penny, go ahead. - [Penny] Yeah, sure, Penny Moore from PCORI, and I just wanted to mention that we have funded about $74 million in large trials looking at strategies to prevent unsafe prescribing of opioids to managing people on chronic opioid therapy all the way to improving access to medication-assisted therapy. Kit mentioned one of the studies that we funded. But I also, you were talking earlier, Mark, about the pharmacists, and integrating pharmacists into the program. Actually, we have a very innovative study that Erin Krebs, who spoke earlier, we funded. It's a large pragmatic trial. It's comparing what they call telecare collaborative management where the pharmacist is assigned to the primary care team and is the care manager and is using a WHO step therapy protocol for determining what medications are appropriate as well as working to refer them to appropriate alternatives, non-opioid alternatives for therapy. But the comparator there is integrated pain management team which is well, prevalent within the VA but very difficult to replicate outside of the VA. And so if that particular model shows to be an effective model, it's something that may be more relevant for ACOs and some other less integrated systems. - Thanks for the research support and addressing this problem of individual providers needing more team support in this effort. Time for one last quick question. - [Nathan] My name's Nathan Langely, I'm with Safer Lock, it's just a combination lock and cap that fits on prescription bottles, designed to make sure medications stay in the right people's hands. Especially with those excess ones that maybe they don't wanna dispose of them. Questions not related to that at all. I saw some really nice graphs that you guys shared, some great work in terms of reduction of opioids being dispensed and prescribed. Do you have any other graphs or have you been collecting data on what the impact of that has been in terms of treatment admissions, overdose visits, and then on the negative side, has there been any unintended consequences, as we've been discussing is potential alternative use. For example, when Oxycontin reformulated, there was a spike in heroin use, has there been any data collected in terms of that as well? - Those are some important outcomes. (laughs) - We're waiting for all that PCORI money to come in to us and we can. No, we're actually starting to, we have not found a decrease in our overdose rates yet, but we are starting to look at other outcomes. I think-- - Well you're talking about maybe more sensitive measures, short of-- - But in terms of, yeah, terms of more sensitive measures, in terms of the benefits of the programs, in terms of the harms, we monitor complaints and we monitor patient satisfaction, haven't seen anything. Those are arguably not the most robust ways to do this, and I really think the proper way is in a proper research study that I am hoping one day somebody conducts. 'Cause it's definitely needed. - All right, I think that's actually a great note to end on. So a lot has been accomplished. Thank you all for sharing your experiences with it, but also thanks for sharing these potential next steps about questions that we'd like to answer and how to go about doing it. Thank you all very much for a great discussion. (audience applauds) We are gonna take a short break now until three o' clock. So please be back in the room ready to go at 3:00 p.m. Thank you. - Than I planned, we're still working on getting the next set of slides queued up, so. (audience laughs) (audience murmuring) Oh, we're good? Yes? Okay. Okay, welcome back, everyone. This is our last session that'll be examining some of the innovative payer and pharmacy benefit manager approaches that are currently being used by both commercial and public systems to help manage opioid access and improve patient safety. Some of the current strategies, many of those we've already heard today, are things like drug utilization reviews, formulary controls, patient and provider outreach efforts, as well as patient review and restriction programs. Throughout this panel we hope to explore the relative impact of these initiatives on patient outcomes. Specific questions that we're hoping to address during this session include: What strategies or tools are being used by payers and PBMs to manage opioid access and improve patient safety? How are payers and PBMs identifying risky prescribing activities or patients at risk for adverse events, and how these strategies are being validated. How do we know that they're working, and what data are being used to evaluate the outcomes associated with these interventions? Also interested in hearing emerging best practices and lessons learned from these experiences and continuing along the theme of data, what additional data might be needed to assess the efficacy or impact on patient outcomes? So with that, I'd like to introduce our next panel. Joined with me here on stage are Kate Berry, Senior Vice President, Clinical Affairs and Strategic Partnerships at America's Health Insurance Plans. Dick Creager, Medical Director, Medical Affairs at CVS Health. Ellie Garrett, Medical Director at the Minnesota Department of Human Services. And Anu Rao-Patel, Medical Director at the Blue Cross and Blue Shield of North Carolina. So I'll turn things over to Kate. - Thank you, good afternoon. I wanted to thank the Duke-Margolis Center for including us. You know, I think that it's been a great day. I've learned a lot. I do wanna just sort of highlight, this is such a complex problem and it's continuing to evolve. It's not just about opioid prescriptions, it's also increasingly about deadly, illicit opioids. And different strategies are needed to address this continually evolving public health problem. And I think another big theme is the importance of collaboration. Just to put it in context, it's estimated that this year, about as many Americans will die from drug overdoses as died in the Vietnam, Iraq, and Afghanistan wars combined. So it's an incredible challenge. In terms of America's health insurance plans, we represent health plans that provide health insurance coverage for all aspects of the population. Commercial, Medicare Advantage, manage Medicaid, as well as individual market, et cetera. And we've been convening a group of our members, we have an opioid work group, we have physicians, pharmacists, policy people involved in that work group. And a big part of our focus is to really gather best practices of what health plans are doing to address this crisis as well as identify some of the key challenges and figure out how can we tackle those challenges as well. And then are also advocating for pragmatic solutions. I was just mentioning, Congress is really working hard on this issue. There have been many hearings already, and there are likely to be many more, we've been responding to multiple different information requests. So they're certainly contemplating legislation and that sometimes has unintended consequences, and so we're hoping to really think about what are the kinds of things that actually could be addressed in that way. I wanted to just sort of provide high-level examples of the strategies that health plans are using to address this. And it really is a comprehensive view that encompasses prevention, early intervention, as well as access to evidence-based treatment and ongoing recovery support. So on the prevention side, we've heard this, I think that it's important and I think increasingly, plans are covering non-opioid, non-pharmacologic pain management. Whether that is non-narcotic medications, physical therapy, acupuncture, et cetera. The plans are also really very, very supportive of the CDC guidelines that were mentioned earlier and so are putting in place tools to implement the CDC guidelines with their network providers. Medical management tools are being used to help to ensure safe prescribing. Clinician education and training, that was mentioned earlier as well. We think it's really important that physicians have good pain care, understand safe prescribing, know how to screen patients for the risk of addiction. We also would say consumer education, helping people understand their options and the benefits and risks is very, very important. In terms of early intervention, plans are leveraging pharmacy claims data to sort of identify prescribing patterns that might be sort of high as well as identifying consumers or their members who may be at risk and then using that information, sharing it with the providers, and collaborating to intervene with those individuals early to provide education, counseling, or treatment if needed. And then in the whole area of treatment and recovery, plans cover evidence-based opioid use disorder treatment, counseling and recovery support. That of course includes medication-assisted treatment. And collaborate at the community level to help ensure that that treatment is available. Couple of potential policy solutions, I don't wanna go through everything we're thinking about, but a few I think are important. We've heard earlier about Prescription Drug Monitoring Programs, there seems to be much more inter-operability, real-time data. We certainly want to see those programs used and we also would suggest that health plans and PBMs should have access to that data. We have pharmacy claims, but we don't have cash pay, there could be people who are actually paying cash, so we may not have a whole picture of those prescriptions. One other thing that I would mention is that there's privacy rules called 42 CFR Part 2 that restricts the ability to exchange information on individuals who've been diagnosed or treated for substance use disorder. We would suggest that needs to be modernized to align with HIPAA, so secure sharing of that information because it's a patient safety and a care coordination issue. I wanna just really skip ahead to mention, I think it just came out in the New York Times, I thought it was really interesting, they did a quick, they asked 30 experts what would you do if you had $100 billion to invest in this crisis? And not surprisingly, everybody wasn't on the same page. But the top three things that came out of that, the first, at 18%, was they would invest in increasing access to medication-assisted treatment for those who are suffering with addiction. The next one, at 13%, was to invest in community support services such as child care and family services and job training to sort of revitalize communities and provide a social safety net. And finally, Medicaid expansion was the third top solution there. And finally, I just wanna mention, so we just, AHIP just released last week something we call the STOP Measure Methodology Baseline, and that's, STOP is for Safe, Transparent Opioid Prescribing and what we did is we used claims data to come up with a methodology to measure adherence with six of the CDC prescribing guidelines and we analyzed that based on a commercial database and released the findings last week. So there's some good news and there's some big room for improvement. On the positive side, a couple of the recommendations pertain to prescribing short-release opioids as opposed to long-term, and there's good adherence with those. In terms of the dosages as well as the duration, there's quite a bit of prescribing that is outside of those parameters. And so we hope that this can be a tool that can be used for collaboration across the industry with providers, plans, and others to improve adherence to these guidelines and ultimately improve patient outcomes. Thank you. - Great, thanks, Kate. Dick. Oh, sure. - Like one of the presenters before, I'm not sure if I trust my eyes to look back up there. But anyway, I'm with CVS Health and CVS Health consists of both the CVS retail stores and also the CVS PBM, the pharmaceutical benefits manager, Caremark, and so I'll be talking about mainly our Caremark programs, the enhancements we've had, but also mention some of those that we've done in retail as well. So this first slide, again, there's really nothing new for us here. I think this emphasizes that at lot of people who begin opioid use do so in adolescence. The vast majority of inappropriate opioid use and substance use disorder begins with a prescription opioid. But to me, the most important thing about this slide is the woman holding her face in her hands. Really the human cost of this. This woman could be someone with chronic pain, she could be someone who's suffering from opioid use disorder herself, she could've lost a family member, or had a family member afflicted with this problem. And so there's a very broad spectrum of problems, it reaches everyone. Many in this room, we're all concerned about this. So that's something I wanted to emphasize with this. So this is a very concise slide, this talks about, shows the risk, this is just one prescription. This is from a study that showed what's the risk of becoming a long-term user based only on the size of your first prescription. And you can see that the risk increases dramatically even after five days. The greatest risks are at prescriptions over 10 days and over 30 days, but you can see if a person gets for their first prescription a prescription over 30 days, their chance of going from a naive, totally naive user, meaning nothing in the previous 90 days to a chronic user, meaning being on an opioid in one year, that's 45%, just based on the size of that first prescription. So this slide emphasizes that another aspect of inappropriate prescribing, that the main source of non-medical use is the patients get them, people get them from unused prescriptions that are obtained for free from family and friends. Those are those large grays bars. Across the X axis is frequency of use, and it's only in that last area where there's patients who are very high frequency users that the getting free from family and friends is not the main source. And so the question is, where do the family and friends get them to give them out for free? And obviously they get them from unused prescriptions. And so if we could just simply decrease overprescribing, we can affect both patients from becoming chronic users, we can prevent those prescriptions from getting out into the system and adversely affecting other people. So as I mentioned, CVS Health consists of retail and of pharmaceutical benefits management, and so this is kind of the slide where I get to talk a little bit about what we've done in retail. The benefit of CVS Health is that we support each other, our retail and our PBM supports each other. We engage in patient education. We certainly support the comments that were made earlier about the important role of pharmacists who educate. We also in our retail, we've developed a video called One Choice Changes Everything, it's called our Pharmacists Teach program. They go into high schools, they've shown this to over 300,000 students and it's been very, very effective. We also have a video now that we've just prepared for parents. We also are very concerned about disposal. In the past, we've had over, we've mainly combined with law enforcement. We've given over 800 disposal units to law enforcement agencies. Starting the first of this year, we will now have these disposal units in our pharmacies. By July 1st, we'll have 750 stores that will have a disposal unit in them. We also support increased access to naloxone. 43 states, that's outdated, there's 46 states where people can get naloxone by standing order in the similar way that you can get a flu shot. We have supported the CDC guidelines ever since they came out in March. We have, since the beginning of 2017, we've had utilization management programs that really reinforce the CDC guidelines. Low-dose, brief duration of time, and watch for harms. And so we've had them available to all of our clients, they were available. But starting February 1st, we made these CDC guideline-based programs our standard programs for all commercial and Medicaid clients. If they don't want our standard program, they can opt out, but they'll have to actively opt out to not adopt these more rigid guidelines. So the first principle of the three is seven-day supply, if someone's opioid naive for their first prescription, they will only receive a seven-day supply. They can, for all of these things, they can appeal these and there's ways to get more medication if needed. The next is quantity limits. Our quantity limits will be MME-based. A patient may receive up to an MME of 90 without prior authorization. If the prescriber feels that the patients needs greater than 90 MME, they can do prior authorization, and if conditions are met, they can receive up to 200 MME. If the doctor feels that they need more than 200 MME, then they can request that by appeal. And then the third principle is that we'd like them to try immediate release before extended release opioids. So if a patient is not stable on an extended release opioid or there is no record of them using an immediate release opioid in the prior 90 days, then we ask them to try an immediate release opioid. Again, all of these are subject to appeal. So patients can receive them if there are exceptions need to be made. And also, these do not apply to patients with cancer, in hospice, or any other end-of-life care. If we get any message like that, actually, we look through our pharmacy records and we find if there's any oncology-related drug, they're immediately excepted from these programs. If we don't catch that, then we ask them to do a PA. The first question on PA is, does this person have cancer, hospice, end-of-life care, and if the answer to that is yes, well then they're automatically accepted as well. We really do recognize that there are patients who have genuine chronic pain needs. We know that the CDC guidelines are not a one-size-fits-all, and that there are patients who we certainly need to have exceptions made. So in addition to our utilization management programs, we also have prospective reviews. We talked about pharmacists evaluate the prescriptions, are there drug interactions, is this appropriate for the patient's gender and age, could this person be pregnant? Pharmacists do that prospectively, before the prescriptions are dispensed. We also, beforehand, it's not part of our opt out strategy, but we did this over a year ago. We have no prior authorization on Buprenorphine, naloxone medication-assisted therapy. We feel that that's very important. In addition to addressing it on the front end, we want to increase access on the back end as well. The next thing is retrospective reviews. We have quite elaborate algorithms that generate a risk score. Is this patient seeing multiple prescribers and pharmacies, are there medicines being used together, for example, gabapentin and opioids, are there high doses of gabapentin, are there benzodiazepines and opioids, are they using Buprenorphine with opioids continuously together? Anything like that, we evaluate all those records retrospectively, and if there's inappropriate prescribing, well then letters are sent out to the prescribers. We also have an elaborate way of not just identifying patients who may be engaged in unsafe opioid use, but also prescribers who may have a pattern. And so through our PBM and also through retail, we look for doctors that are compared to other physicians in their geographic region, they're compared by specialty, and if they're found to be an outlier, we reach out to them. We have an opioid prescriber toolkit. I make many conversations, we often have a pain management doctor, or an addiction medicine doctor speak with them. And we educate them. If after all of that, there are a few doctors who do not respond, and if we feel that if they continue to engage in unsafe prescribing, we will occasionally suspend their controlled substance privileges, and when we do that, we do that at both mail and retail. - [Greg] Great. Thanks, Dick. So next we'll go to Ellie. - Hi, folks, just one clarification. I am not a medical director, I work for the medical director at the Minnesota Department of Human Services. I'm a long-time policy wonk and as I said, I work for Human Services, which is the agency in Minnesota that houses, among other things, Medicaid. And so it's from that perspective that I'll be speaking. If I refer in my presentation to DHS, it's not Homeland Security, it's Human Services. We had the acronym long before they came around. (audience laughs) I'm also going to say thank you for sticking around for this full day. We are not the worst, this isn't the worst time slot, though, because it occurs to me that we're what's standing between you and rush hour, so just, you know, relax, and thank you again. And we'll move on. So just a little bit about Minnesota. We're the 12th largest state geographically, we have five and a half million people, nearly 70% of whom live in a Greater Twin Cities metro area and we're home to 11 tribal nations and just sent 20 winter Olympians to Pyeongchang. What this makes is us a great big rural state with a relatively large metro area. And I mention this because, and I show this map because I think it makes the point rather compellingly that when you get right down to the tactical level, a one-size-fits-all approach is unlikely to work. Granted, there are a lot of things that we can and are doing at the state level, but some of the most important work that we're doing is using state and federal grant funds to push out to communities so that they can build community-based partnerships and collaborations that build on their cultural and other strengths and that have that, build on that local connection. We also serve in Minnesota Medicaid in our small sibling programs roughly 20% of the state's population. We have a lot of things going on in Minnesota that concerns the opioid crisis. And the vast majority of agencies in state government touch it. We sort of felt like maybe we were tripping over each other, and so we convened this group that gets together roughly every month or two, called the State Opioid Oversight Project, or the SOOP, and addresses a wide range of topics that cut across the executive branch and the judicial branch of state government. And frankly, are done with legislative support, so it's all of us. The items that are bolded are ones in which our agency and Medicaid is taking either a lead or a co-lead with one of our sister agencies. So it's multi-faceted, and it's a hub for keeping the state players informed. It doesn't list some of the other things that we're talking about. Things like the restrictive recipients program that's been a long-time feature of Medicaid where we use a lock-in program for folks that for the most part are seeing a lot of different providers for opioids and other kinds of treatment. Just as a matter of safety, they're put into, to programs where they're restricted to a single set of providers for their care. And just yesterday, our governor announced that he would proposing funding to the legislature for integrating our PDMP with EMRs in Minnesota. So, there's a lot that's happening. I'm going to spend the rest of my talk talking about the Opioid Prescribing Improvement Program that is about non-cancer, pain that's not coming from cancer and not hospice setting. And hitting on a few of the things that distinguish the work from what some other places are doing. We have developed prescribing guidance for the entire temporal pain spectrum, from acute, post-acute, chronic pain. that guidance will be made available, well the draft guidance is already hanging on our website, and we just closed the public comment period and next week the work group that formed that guidance will be convening to consider all of the comments. Then there's also a measurement program. The guidance is just that, it's guidance, it's statewide available, it's voluntary. The measurement and quality improvement work is restricted to what we do in Medicaid because it makes use of our claims data. We'll be issuing confidential reports to prescribers, looking for variations in care, letting providers know where they are in their prescribing practices, compared to their anonymized peers. Can't emphasize enough the appropriateness and utility of having these reports be confidential to everybody involved. A prescriber will see their own data, and then the groups of the prescribers affiliated with will see that person's data if there's a mandatory quality improvement program that has to be implemented. But in terms of public reporting, individual names won't be made public unless and until we actually terminate a provider from the program, something that quite frankly, we hope we never have to do. We think quality improvement work will really help support the kind of cultural and behavioral change that we're looking to be part of. The other thing that is part of the prescribing improvement program is about common messages. We hear, and we've heard a lot today about how important it is that providers learn how to speak comfortably about pain and appropriate prescribing and tapering with their patients and have been told that it's a barrier, and so we've got some contracted funds for communications expertise and focus group analysis to help support this work. The process that we've used in developing this program has been really important. First of all, it's legislatively authorized but not micromanaged. We fully expect that thresholds and the definition of what evidence-based good prescribing means will be evolving over time and this will allow us to be a little more nimble than having something set in stone at the legislature. It's also very important that we're not doing this to providers, we are doing this with providers. And so, all of the guidance has been developed by an external community panel of experts. There are several state employees who sit around that table, but they're there as voting numbers. The work has been very data-driven, and I cannot speak highly enough of my colleagues who've been developing and diving in to our data like I've just never seen, it's been extraordinary. And then also based on the best available evidence that's in the literature. And the transparency of our process has been very useful, since day one all of the meetings have been public and simultaneously screened on the web. It also involves multi-level measurement. What I mean by that is illustrated on this little table here. At the base of the pyramid, we're talking about individual provider-level measures based on looking at variation in care, and we see a lot of variation in care. And so do our partners who have used these measures to check it out. Mayo published a very interesting article using measures around post-surgical prescribing for orthosurgery. And then in the middle, well I just mentioned Mayo, but there's optional space for clinics and health systems that we really hope will be engaging in this process collaboratively and they already are. What they can bring to the table are things that we can't, like linking the exact prescription with a diagnosis. Which you really need to do in order to affect good change at the individual level. And then up at the top are the kinds, again, variations in care. We're gonna be looking to see whether or not those variations begin narrowing. And we're also looking at a new measure that we've developed called the rate of new chronic use. So by new chronic use, what we're talking about is looking back from the first prescription in a given year to see, well, is the person naive to opioids, at least for right now, and then looking forward to saying how long are they on it? And we learned so much from this measure with a brief lookback and a brief go forward that, we got NCQA interested, and they have just issued a draft, a draft measure for public comment that's based on our measurement. So I am at time. So I'm just gonna mention one thing here, and that is that we've worked pretty hard in our program to eliminate the burden of data management and data input to our providers, and instead, keep the burden around changing behavior, and to do that, we've worked to align with other efforts on this so that we're all kind of going in the same direction. With that, I will end it, and thank you very much. - Okay. Great, thanks, Ellie. It was very helpful understanding of how you all are dealing with this in Minnesota. So, now turn to Anu. - So thank you to Duke for including the Blue Cross Blue Shield Association as well as the Blue Cross of North Carolina. And thank you to all of you for paying attention, because it's always hard to be the last panelist on a panel at the very end of the day, so I'm glad that all of you have stuck around. So Kate sort of gave an overview from an AHIP, a general plan of what all payers, commercial payers, and the payers that they represent are doing the opioids base, and I thought I'd get a little bit more into the weeds based on what we're doing in North Carolina. And I'll just mention that prior to joining Blue Cross of North Carolina, I was actually doing pain management in my practice in addiction medicine, so it's been very interesting for me to see the devastation of the opioid crisis, both as a provider and now in the payer community. Just a little bit about our North Carolina plan, we serve close to four million customers, we're in every zip code of all 100 counties of the state. We're the state's largest health insurer. The Blue Cross of North Carolina PPO network of healthcare providers includes 96% of physicians and 99% of all general acute care hospitals. We're NCQA accredited, not-for-profit organization as well. In North Carolina specifically, I know we've talked about a lot of stats today, this opioid crisis has claimed more than 12,000 lives in North Carolina since 1999, with opioid-related overdoses actually up more than 800% through 2016. So, especially in our state, we're very passionate about it, because out of the 25 highest states affected in the nation by the opioid crisis in overdose, North Carolina has four of the leading states there. In terms of Blue Cross initiatives, Blue Cross Blue Shield in general as an association and we as a plan support a very holistic, well-rounded approach. The association is promoting education and understanding through media education, public policy, safe drug disposal, increasing public awareness, they're fostering research, the Brain Research Foundation partnership, the Health of America report, and the Research Alliance. We support, both at an association and plan level, and endorse CDC guidelines, also recognizing that not everyone, it's not a one-size-fits-all approach. We also had recent legislation signed into law by our governor last summer known as the STOP Act, which some of the provisions include limiting fills for acute pain to five days, for post-op pain top seven days, requiring mid-level providers to consult with their physicians prior to prescribing opioids as well as e-prescribing and requiring providers to check the PDMP. Some of our local North Carolina plan efforts, we continue to look at our data analytics, we develop these and we're trying to understand claims data and opioid prescribing patterns, looking at outlier providers, looking at general costs in terms of both prescriptions and the costs of substance abuse treatment. We view our providers as an extension of our plan, they're the gateway to the first prescription of the opioid. So we like to be collaborative with them, we have provided education with them on the CDC guidelines, on the STOP Act legislation which was signed into law by our governor, as well as recent changes in CME requirements by the North Carolina Medical Board for prescribers of opioids. We have sent out educational letters, we've done webinars for them. We've also identified from our data analytics outlier prescribers and understanding that in some aspects, the prescribing may be entirely appropriate. These providers might be seeing the hardest of the hardest pain patients, and so understanding that it might actually be appropriate prescribing, we are not trying to use a very heavy-handed approach initially in educating these providers, but we have sent out these letters, we will be watching prescribing habits, and if they don't change, we do plan on sending a much more targeted letter with specific data for each provider. We've also posted blog posts on the Facebook page, on appropriate prescribing. Another bucket that we've looked at is our members and member education. Our care management operations has been educated and taught on how to identify high-risk members, so our case managers make direct outreach to members in more of an engaging way rather than in an intrusive way. We've looked so far at specific procedures where opioids have been prescribed. For example, like elective post knee replacements, post hips, elective coronary artery bypass grafting, where nurses make outreach. And we've gone a step further, so instead of having the nurse say I've noticed you've got a prescription for Percocet, do you know how to take it, the next step would be, do you know what to do with these medications if you're not using them? Do you know what signs of addiction and dependency are, and do you know how to store these medications in your home? From a pharmacy standpoint, we have an actionable point of sale edit message which can be seen by a dispensing pharmacist. This went live September 2017. This alerts the pharmacist if a member is trying to fill a concurrent opioid and, opioid script and script for opioid use disorder, such as Buprenorphine. So the pharmacist would actually have to go in and put an override to fill both scripts or one of the scripts. We currently have quantity limits on all long-acting opioids. Our Narcan nasal spray as well as the Narcan, the naloxone injectable, not the vial injectable, was moved to a brand, lower cost share tier for our members. Our pharmacy benefits manager also sends out quarterly reports to our providers with any type of suspicious behaviors in our members as well as members who are on high doses of opioids, one or more prescribers, opioids, using multiple controlled substances such as the combination of opioids, benzodiazepines, and muscle relaxers, or any type of signs of misuse or abuse, like appearing of doctor shopping. We have also partnered at an association and a plan lever with Walgreens Pharmacy for drug take-back kiosks. We currently have 25 in the state, and are planning on implementing one in every Walgreens across the state of North Carolina. We've done multiple external stakeholder collaboration and community events. As I mentioned, the association has a national, association-level work group which represent somebody from every 36 Blue Cross plans is represented on. We have met in D.C. with the Department of Health and Human Services, testified at the Senate Finance Committee. We have a collaboration going on with the association that focuses on addressing the geographical and social realities that employers face as result of the national opioid epidemic. We have partnered with our local North Carolina DHSS, we are serving on a prescription advisory committee with other payers to come up with some, improve health outcomes and see if we can come across multiple payer lines to come up with some standardized recommendations. We also have met with our in-state treatment facilities to see what the availability is for beds, see if we can address the issue at target areas like the emergency room departments and reforms from primary care physicians. We have partnered with the North Carolina Medical Board. We are participating in CME panels across the state to educate our providers on appropriate opioid prescribing. We provided input on the STOP Act legislation that I mentioned to you. We have met with and are working collaboratively with the North Carolina Attorney General office as well. On a national level, we participated at the national, at the Bipartisan Policy Center on the Appalachia Initiative to address realities with opioid issues, especially in rural areas. North Carolina has a very large rural population. We participated on the fourth of fifth meetings in Washington D.C. at the White House on the President's Commission, where we addressed insurance-slated issues related to the opioid crisis. We also participated on the FDA workshop that has recently been mentioned on the packaging storage and disposal options to enhance opioid safety. And again, we're looking at this from a very holistic, well-rounded approach, we're trying to engage with our members, with stakeholders on both the state and national level, we're trying to educate our providers and partner with them on this issue. Thank you. - Great, thank you. And I'd like to thank all of our panelists for very, very specific examples, and lots of examples about some of the strategies that your organizations are taking to manage access, reduce risk, and improve, hopefully, patient outcomes. I'd like to turn to that last piece, improving patient outcomes. I understand, for the most part, when you're talking about payers and PBMs, we're talking about having access to claims data, and that claims data can be very useful in, I think, that Kate, you even brought up examples of maybe being able to look in the data and measure adherence to guidelines and measure prescribing patterns. But how are we doing in sort of getting to the next level of outcomes in terms of, are we actually improving patient outcomes? Do any of these strategies, we heard earlier that the potential for unintended consequences that some of these strategies can actually result in, barriers to appropriate chronic pain management. How can we, where are we with sort of looking at those kinds of outcomes? Anyone? (laughs) So, Dick, I'm gonna put you on the spot, 'cause you did - Oh, I'll go. - You did specifically measure, talked about measuring adherence to guidelines, but even if you're not doing it, I'd like to hear why not and what can help you move into that path. - I think that's a very good point, we've talked about, how do we get from basic at-risk measures, for example, how many patients are over a certain MME, or how patients have a first prescription that is greater than seven days? I think if we can get to where we can transition those types of figures to outcomes, I think was mentioned in an earlier presentation, does this make a difference in patients who suffer from overdose or opioid use disorder? In constructing our criteria, we accessed a study that showed, that had incidence figures, what's the incidence of opioid use disorder and what's the incidence of death from overdose if you're a high utilizer, if you are above a certain MME? Or what's the incidence if you are a long duration, if you're on opioids for greater than 90 days? And we calculated that if we could get people out of those high-dose, high-duration areas, and we could get everyone, if we can get people to a low dose, low duration, we calculated based on the incidence figures that we could save, we could prevent 61 cases of addiction a year, and we could prevent three deaths per 100,000 insured population. And if you figure that there's been studies that demonstrate the financial costs of opioid use disorder, many different figures out there, one of the most recent is $15,500. Patients with opioid use disorder have excess medical costs compared to those with non-opioid use disorder. So if you talk about you're preventing 61 people from becoming addicted, that could save nearly a million dollars per 100,000 population. And I think that as we get, we've been a little bit limited because as a PBM and as retail, we really primarily have access to pharmacy claims data, we don't the medical claims data. Hopefully with our Aetna acquisition that might change. But, we're looking to those things, and I think that is an important thing. What is the effect of what we're doing? The CDC guidelines for MME, an MME, most people need less than 50, and the doctors should be very cautious about prescribing over 90. Well we looked at our population and found about a third of our prescriptions were for an MME over 50. We felt that would be very disruptive if we were to start with a 50-90 in terms of the pre-PA and NPA. And so that's why we adopted 90, the CDC figure, and then the remaining several pain management societies that define high-dose as an MME greater than 200. So, do I wish, I was talking to Dr. Franklin earlier, do I wish we could be 50 to 90? Yes, but on the other hand, we don't wanna have negative, unintended clinical consequences by being too aggressive on patients who can't be tapered down quickly. - Uh, Kate. - I wanted to just add, too that some of the measures I skipped over, but based on our analysis, 40% of people on opioids are also on benzodiazepines, which is a pretty dangerous interaction. So, I'd call it an outcome if you reduce that overlapping prescriptions that cause really bad outcomes. I think another one has to do with the issue of when somebody is first, gets their first opioid prescription, basically the recommendation is to have a follow-up visit to assess. Is it necessary to continue, so if you're getting into a chronic situation, and less than 50% of patients who have an initial opioid prescription actually have a follow-up visit within 30 days. It's definitely like, those are I think outcome measures in some ways, too. - Yeah. We are using of course both public health measures, population health measures, as well as measures that are more tuned to the individual prescribing level. One of the population health measures that I alluded to quickly in my presentation was our new chronic user. We've been creating roughly 5,000 new chronic users in the Minnesota Medicaid population annually. That has to turn around. And so if see fewer people becoming chronic users, that will be a significant success. The guidelines also talk a lot about assessing patient functionality as an input to guide therapy. And so at the individual level, we talk about using hopefully as objective as possibly measures that inform the individual prescribing. - [Greg] Okay, Anu, did you? - So the only thing I was gonna say to add to all this was that we have seen a reduction in the amount of these concurrent benzo, opioid, muscle relaxers through our PBMs sending out these quarterly letters, 'cause what I think what happens sometimes is, the patient might be in pain management and the primary care sometimes assumes that okay, well, they are writing your opioids and we're writing everything else. So a lot of times there's lack of communication between different providers as to what the member's getting. Sometimes I know, at least when I was in practice, I would see this and I was actively checking the PDMP, and I would still be surprised that some of these prescriptions were getting filled. So we have seen a reduction in that just by educating the provider. And I will say that one thing that, like Kate mentioned already about the PDMP access, is that the PDMP shows fills. So as a payer, we're looking at claims. So if a members goes and fills a prescription for an opioid or a controlled substance with cash, then those are fills that we're missing. And so that's why we thing as a payer it would be helpful to have access to that information. Definitely because again, anything that gets filled by cash is not gonna come through our claims system and we'll miss that. - Great. So my question about patient outcomes, I did hear, I'm gonna push a little bit more on this, I did hear a lot more on utilization-type measures. Like well, we could measure duration of use and concomitant use, and I fully appreciate that that's very much directly a limitation of the current access to data. But I did hear some sort of need to link PDM, which is largely the prescription drug claims data with the medical data. Also, just sort of in terms of making sure that we're capturing the exposures, linking to the PDMP data gives us sort of the cache and the data that you already have, but we're gonna push a little bit more on just asking about steps that you all might be looking at or taking to integrate your data with more clinically-enriched data, like electronic health record data, to get more clinical granularity about what's really happening with these patients as a consequence of some of these policies. Or, we heard in earlier panels, patient satisfaction data, patient-reported outcomes, clearly not largely collected by payers, but increasingly groups are looking at integrating payer data with some of those other sources of data. Do you all, just you get a pause sign, sort of where you all are maybe thinking about the possibility of doing that, or what some of the barriers might be in furthering that? - Well I'll just start, I mean, I can give a brief response because again, our data right now is mainly pharmacy claims-based. But I think in medicine in general, not just in opioids, medicine in general is moving to value-based care. And I think that there's going to be, there's a huge incentive to document the efficacy of what we're doing. Not just, we decreased these numbers, but I think on the back end, really what is the outcome, is this really improving outcomes? What's the value of this care? So I think that's happening. But there are things that we can do. For example, we can look to see, again, with these measures that we talked about, about decreasing the days' supply and concurrent use, are we seeing, as we're seeing these decrease are we seeing an increase in use of MAT? We also have programs where, as I mentioned in some of our retrospective review programs, we have had a long-time collaboration. Earlier we mentioned collaboration and coordination as a big part of doing medication-assisted treatment, and so when we identify high-risk patients, we actually, our clinical advisors and account teams, can identify those patients, take them to the clients, and then those clients can get them into their own case management systems. - I would, too. We heard about from Kaiser earlier, I think where you have the fully-integrated systems that have the physician electronic health record and you have the plan and the hospitals, potentially, I think where you have that integrated system, you have the visibility into not just the claims data, but the clinical data and you have a much more integrated approach both to engaging and to analyzing. So I think that's where you can see more the best practices there. - I agree with the comments about value-based purchasing, it's the direction that we're all moving in, including Medicaid agencies, or at least ours. One of our partners, St. Gabriel's Health System up in Little Falls, Minnesota, beautiful corner of the world, got a State Innovation Model grant, and their whole grant was centered around addressing the opioid crisis in their community. So there are ad-hoc examples of this. I think we are not yet at the state where we have really fully integrated approaches that well-integrate all of our purchasing, but I think that it's the direction that things are moving in. And we have partners that are interested in that, we have clinical partners that have said, we're gonna run this stuff in our clinics and see what we come up with and share that information with you. That's all to the better. - [Greg] Okay. Did, Anu? - Well I'll just add to what she said, which is, I think we all, as different stakeholders, have lots of data, but it's very siloed, and we're not able to for whatever reason, whether it's a regulatory reason, or proprietary reasons, able to share that. In a lot of our collaborative efforts with national and state stakeholders, for example, working with the medical board, or going to the Attorney General's office, or our North Carolina DHSS, we've discussed this, like we all see the same things from a different angle and a different lens, and there should be a way that we can all collaborate together and share the data within the scope of what we can share with each other to improve outcomes overall for all these patients. - Okay, perfect response, because that takes me exactly to my next question, which, we did hear a lot. Heard it here on this panel and throughout the day about the PDMP data and linking that to claims data. And I think you were touching on that and sort of sitting down at the state level and figuring this out. So is it happening routinely with all of your organizations or if not, what would it take to have that process expedited? - Well, I've been trying for years to get third-party payer access to PDMP data. And that just is in Minnesota, I believe, or in Minnesota briefly. Yeah, Minnesota had it briefly, where they could do that, but right now, there is not third-party payer access for the-- - I don't think anywhere. - Yeah, that we can use that data retrospectively. We can identify patients retrospectively and then we can access that. And so we would like to use that as a PDM. Certainly our pharmacists, our retail pharmacists, but the way most of the statutes are constructed, it's really for a face-to-face encounter, with a physician or a pharmacist right there. There's certainly valid confidentiality concerns. But we would like to be able to get some of that data and use it in some of our retrospective review. I think there's states, I believe California and Washington have where they do proactive notifications for some of the state plans like worker's comp or Medicaid where they're set up, and they will, the PDMP has criteria, they identify at-risk patients, and then they contact the prescribers. So I think that's a very good thing, but that would, if anybody... Peter Kreiner who was here earlier, we would support increased access for third-party payers to PDMP data. - It would require legislative policy change in Minnesota. - [Greg] Statutory, okay. - Yep. - Okay. Okay, so I'll turn, I'll get off that topic and move on. So one of the things, and Kate, you brought this up around the plans are routinely covering non-opioid and even non-pharmacologic modalities for chronic pain management. There's sort of a sense on how's that going, are payers even going a step further to not just cover, but maybe incentivize, or lets help channel patients to those even non-pharmacologic modalities, and if not, what are, I know that there can be some challenges in terms of evidence of effectiveness, but where are you with that kind of thinking? - Yeah, I think the trend is certainly in the direction of sort of encouraging and covering some of those other approaches initially. I think there's a few examples, there's a small plan, actually in Massachusets, Neighborhood Health Plan, who has eliminated copays, done all sorts of things to encourage people to do physical therapy or other things, a few visits before doing medication. So there are certainly examples of that. There is limited evidence of what works under what circumstances. A lot of people have talked about lower back pain and how common that is and chronic and there are really, really wide variations in how that is treated and wide variations in outcomes and wide variations in cost. So I think there are some challenges in terms of the evidence. I think another big challenge here, even though it is the direction, is an expectation issue and a time issue. So whether it's the physician-patient dialogue around I have this pain, what should I do? I mean the easiest answer is here's a prescription and even if exercise or physical therapy or acupuncture or other treatment modalities could be just as good or better, it may require more time, multiple visits, it might require multiple copays, et cetera, so I think another opportunity is to start to think about bundled payments or some other mechanisms for how to care for people. I think another key thing is, and this has been mentioned earlier as well, but even with pain, it's very individualized, and so some types of pain, you'll recover with no treatment, and with others, people really need a very holistic, integrated approach. And so much of this type of pain presents in primary care, and they may have limited tools for knowing when is the time to try this, when is the time to try that, when is it important to refer to a specialist? So lots of challenges, but moving in the right direction. - Yeah, I'll just add to that and say, for example, with our North Carolina plan, we don't have prior authorizations on things like chiropractic visits, physical therapy, we cover things like trigger point injections, massage. So there's a lot of, I don't wanna call it alternative, but complementary, non-pharmacologic pain management modalities that we try, that we cover, I apologize. One of the things I will say that Kate made a great point, is that, and I can sort of relate back to when I was in practice, is that oftentimes, by the time a chronic pain management patient gets to a pain management specialist, a lot of things have already been tried and failed, and I found in practice, it was almost a struggle to get patients to revisit physical therapy, aquatic therapy, chiropractic care, trigger point injections, all these complementary things that the payer side already covers. Sometimes I think, like we talked earlier in the morning about the expectation and the conversation between the provider and their patient about what pain management means. I would tell my patients, pain management does not necessarily equal opioid management. And so if it's appropriate and the benefits outweigh the risks, then we'll prescribe opioids, but if it doesn't then we're gonna have to find something else that works. So really looking for a functional improvement is really what the goal is, and I think providers need to have those conversations with their patients. They're not easy conversations, and I would be actually surprised to see how many patients would actually be wanting to take an opioid prescription if the provider actually told them straight out, this will not take away all of your pain and you might still have pain after this. So I'd be curious to see if patients would even wanna take a prescription, an opioid prescription at that time. - Okay. So that brings us to maybe time for Q and A from the audience, does anyone, if you have questions, go ahead and line up at the mics. Session before had like three or four people at, okay, here we go. (laughs) - [Larry] Larry Greenblatt from Duke and was a earlier moderator. Question for the panelists. When we've tried to implement some of the safe opioid prescribing strategies, the things that the CDC's recommending, there's always been this issue about time. And you all know that in healthcare time is money, and you guys got the money. So what could you do about actually paying providers, or systems of care, to do the things that we're supposed to be doing? Reviewing the PDMP, educating patients, talking about alternative strategies, et cetera, could there be some add-on code whereby those who are actually doing all these things actually get paid for their time? And then perhaps could be hiring staff, nurse, pharmacist, an advanced practice provider, whatever, to take on that role so that really overburdened, busy clinicians don't just push it off and say I can't? - Yeah. Great question. - I think the points have been made, too. Like the transition to value, right? To outcome so that there is incentives to keep people healthy and to better manage chronic conditions like this. So I think that's kind of, we're getting there slowly, but we're getting there. - And if I could just add, as the other panelist to chime in, too. As we transition to a largely volume-based system to a value-based system, what comes hand-in-hand in that is sort of evidence. Real world evidence, evidence that these new care models, or better coordination among the care team actually results in better outcomes for patients and results in outcomes related to cost. It might come with, as these approaches are becoming more widespread, also demonstrating that they are effective brings evidence to then the payers to say, okay, this is worthwhile. Any other additions to? - Well, within Medicaid, we don't only have claims-based payments, we also channel a lot of funds from state grants and federal grants and so that kind of infrastructure development that needs to happen in busy practices is actually one of the things that we're supporting with some of our SAMHSA money from the state-targeted response grants to help office-based opioid treatment programs build and expand in Minnesota for the management of opioid use disorder and for the diagnosis of that disorder as part of a transition, perhaps from chronic opioid management into a different diagnosis. So that's transitional funding, of course, it's not institutionalized, but it helps bridge until we get to that more robust, value-based system that we have a lot of in Minnesota already, by the way. - [Greg] Uh, yes. - [Erin] Hi, Erin Krebs. I thought I would just bring this up since it didn't come up in the prior conversation about covering other therapies, and that is covering mental health care. Because I think an underlying issue behind the opioid crisis is that we are not only treating pain, clean pain, pain alone, with opioids, we're treating comorbid pain and PTSD, depression, anxiety disorders, distress. The data that shows that 40% of patients on long-term opioids are also on benzos, why would that be? You know? I currently practice in VA, which has, I have great access to mental health integrated in my primary care clinic. I've previously practiced in safety net systems in two different states where I had almost no mental health access. Or if I did have any access, it was like sending people way off to some mental health clinic. Again, with the copays and the barriers and the limits on the visits. So that's a question for the payers at the table, is how are you integrating mental health with all other health care so that we don't paste biomedical solutions on underlying issues that aren't being addressed? - Great points, totally agree. I think a lot of people who have these conditions have multiple chronic conditions, right? And so you need to be managing the whole person, not just one piece, right? And so I think it is absolutely, plans would integrate, I'm sure that integration of those multiple types of care. I mean, even with pain, cognitive behavioral therapy and even when you think about if the time comes when someone needs treatment, managing the chronic conditions as well as the medication-assisted treatment and counseling and other support. So, totally agree, needs to be a very, very integrated approach. - We have the luxury of having the single state authority for alcohol and drug abuse responses in Minnesota also housed within the Minnesota Department of Human Services, and so we've been working hard to address those internal silos that are also reflected in the community and are making some strides. I will also say that our partners in that division successfully got a very substantial package of legislation through last year to really bring some needed reforms to mental health access and treatment in Minnesota. So this is happening as that's being rolled out. - And I'll just say from our perspective, we're obviously compliant with mental health parity, which obviously states that we can't treat mental health issues any differently than we would medical issues. We do cover things like telemedicine, and we've expanded additional CPT codes for specific mental health disorders with telemedicine. We've actually looked in the state and are looking into possibly partnering with one of our universities there where, like I mentioned earlier, there's a lot of rural areas within the state. So you have areas in the state where there's nobody who does pain management and then nobody who does MAT. So there's access issues. So partnering with universities and other stakeholders who are already using platforms like Project ECHO and things like that to sort of integrate and support rural health providers as well as engage with primary care physicians, to support them with not just treating opioid addiction, but all the mental health stuff that you're talking about as well. - Go ahead. - I have a second question. - Sure. - [Larry] Nobody else took the mic. So my second question is one that, is a hard one, I'm not sure anybody has an answer to this, but I and many of my colleagues back at home, we worry a lot about these prior authorization processes, limits on opioid prescribing, not that we're against them, but what we worry about is that, is this driving prescribers out of the business? And we've certainly seen in our state, at least amongst the Medicaid population, or Medicaid provider population, which I have access to, there's been a steep drop off in the number of people actually prescribing. And we've also seen concomitantly with less prescriptions going out there, this steep rise in illicit drug use. Now is it that, is one the cause of the other, we obviously don't know that, but it's certainly concerning, and we don't have a way within claims and PDMPs to capture how many of these folks that we're driving down their prescribing, how many of them are turning to illicit drugs, and what ideas does anybody on that panel have thinking creatively about how might you measure that and how can we make sure that we're not essentially driving people to dealers who are very, very happy to meet this demand when we don't meet the patient's needs? - Great question, and definitely brings up, sort of highlights gaps in current data and maybe opportunities, but initial thoughts from the panel? - Yeah, I'd. That's a good question about, the question about our utilization management programs driving people into more adverse situations, such as street drugs, and I'd just refer everyone, probably the best study that I saw on that was Grant Baldwin and Chris Jones of the CDC in January of 2016 published in the New England Journal of Medicine a study addressing the relation of heroin use to PBM and utilization management programs. And that found that there really was not a correlation with those types of things. People turn to heroin, the conclusion of this study, or what they brought up in this study, was that the heroin epidemic started mainly because of heroin market forces. About 2012, heroin became very available, it was very inexpensive, it was very potent. And so that happened in 2012, and that's really the etiology of the epidemic. That happened actually before a lot of the utilization management programs came into place. So we're certainly concerned about that, and that's why we want to, with medication-assisted treatment, we've taken PA off that, we have no duration of approval, if people need MAT, they should have it for life. But there have not been studies, and I would suggest that people look up that study, I actually have it here and I can give you that reference, it's in the January issue of the New England Journal of Medicine. And it really showed that there was not a cause-effect relationship. There may have even been some type of temporal correlation, but there's really not been a cause-effect relationship in terms of our utilization management programs related to getting people into these more illicit drugs. - I'll say in Medicaid that concern is pretty prevalent. We're a human services agency that also manages SNAP and also manages child protection and child welfare and manages cash assistance and a whole range of social services. So the thought that we might actually be driving somebody out of the medical system and then onto streets and into more vulnerable settings, is just, well first of all, it's, the human suffering, of course, is the real cost there, but there are a lot of fiscal costs that come along with that as well. That's why we think that things like tapering, it needs to be done individually, needs to be managed well, and we also don't like the notion of contracts that would ever end a provider-patient relationship because the patient's not compliant. Patient who might be noncompliant with chronic pain treatment could very well be exhibiting signs and symptoms of a different condition, opioid use disorder, and so it's not that they're not compliant, it's that they need treatment. And so it's about making sure that we get the right treatments to people and aren't erecting policy barriers to that. But, agree with the concern. - And I wanna add, too, that I think that this issue of how you need a waiver to prescribe medication-assisted treatment and we have too few providers with that waiver, and those that have them aren't caring for the full patient cohort that they can, so why is that? Similarly, for people who may be, what you're implying is they're cut off from the prescription that they need. I think the issue, you don't hear a lot of people talking about how it's not trivial to taper them off safely. And so not every prescriber, not every physician may be well-equipped to manage those people, and I think those are capacity issues that are kind of nuanced but not necessarily talked about very much. - We've just launched a Project ECHO on that very point. If you're familiar with that concept out of the University of New Mexico to help support primary care practices as they expand into this area. - [Greg] Great, thanks. Time for one last question. - [Marylee] Hi, my name is Marylee Grosso, and I'm a pharmacist with the American Society of Consultant and Senior Care Pharmacists, and since I'm the last question of the day, I'm hoping that this is one that will be thought-provoking. Earlier one of our panelists, Dr. Kelly Clark, during one of her responses said, what are our goals? Is it to decrease deaths? Is it to decrease the number of prescriptions? Is it to decrease the number of people addicted? So at the end of this wonderful, thoughtful, and thought-provoking day, I wondered if the panelists, or perhaps any of the other panelists that are here in the audience, could perhaps give some thoughts about what we're taking away today as our goals. - Wonderful question, it did come up many times today, and does get back to what are the goals and then what data do we have to actually measure impact toward those goals? But any final thoughts from our panel on this topic? - I would say that our goals are all of the above. I think part of our, the way we're gonna manage this crisis and this problem is going to be reactive because of everything that's led up to what's happening now, and part of it is going to be proactive to prevent future overdoses, continue to educate our providers, continue to educate members on alternative to opioids, et cetera, et cetera. So I would probably say that all of the above. - Great. Thank you, well that brings us to the end of this panel. I'd like to ask the panelists to go ahead and stay up here at the table while I close out the day, but thanks for your great presentations and a lot of very timely examples of what your organizations are doing to address this issue. I did hear, we covered a lot of ground today. Thanks to all of you for sticking with us throughout the day. We did cover a lot of ground, and we found some common ground as well in that process. It seems that consensus on the need for better chronic pain management and substance use disorder care coordination, preserving access for patients with chronic pain and developing person-centered approaches that are targeted at different types of, and reflect different types of patients. We talked a little bit about, or a lot, in the earlier sessions about the guidelines. In that they are quite helpful, but still, as more is needed in terms of educating our providers, educating patients, as well as developing more person-centered approaches that do reflect not one-size-fits-all, but different types of patients who are harmed by this issue. We also discussed the role of not just guidelines, but education, both providers and patients around the tools that are available to help manage chronic opioid use and improve patient safety, as well as safe disposal, as FDA has put together a white page to help patients and providers with best opportunities for safe disposal. Data. Data, data, data. That came up throughout the entire day. It was a very big topic, and we discussed the need for better and more efficient collection data, of data, including linking claims data to not just pharmacy and medical data, but to PDMP data as well as clinical data, mortality data, and almost most importantly, as a collection and linkage of data that comes directly from patients. Patient satisfaction, collection of patient reported outcomes in the ambulatory setting, and linking those in with other existing data that we have to get a better picture of what's happening out there and whether or not these policies are leading to better outcomes from the patient's perspective. But also still needs to come, and the very last question gets to this, in consensus on what outcome measures are we driving toward, and how can we collect the data to better address and to better measure those patient outcomes? However, data themselves, this was a point brought up earlier in the day, aren't the answer in themselves. It's one thing to collect and link data, and that's really important, strides are being made on those fronts, but translating that data into actionable evidence is a critical component that can't be lost, and using that evidence to help develop more improved guidelines, but also to help educate patients and providers, and most importantly in training patients and providers and using these tools to help manage chronic pain and to help manage access to these drugs. So we haven't solved these problems here today, but hopefully with this and our subsequent work, we can help identify high-priority, practical steps to addressing this epidemic. So in summary, I'd like to thank all of you today in sticking with us throughout the day. The folks who have dialed in online and stuck with us throughout the webcast. And I'd like to thank our esteemed group of speakers and panelists for all of your contributions before this event, but helping to make today's event a success, and especially our partners at FDA. Big thank you to Commissioner Gottlieb, Judy Staffa, Adam Crutch, Adriana Hughes, and the rest of the FDA team who have worked tirelessly with us throughout this process and to help get us to today and then beyond. And lastly, I'd like to thank the team here at the Duke-Margolis Center. Katie Green, Isha Sharma, Sarah Supsiri, and Nick Fiore as well as moderators Larry Greenblatt and Regina LaBelle for their efforts in helping us put together the paper and moderating today. So again, thanks to all of you for sticking with us, enjoy the rest of your afternoon, thank you. (audience applauds)
dissertation review job politics dissertation order apa dissertation manuscript order past dissertations online buy dissertation writing methodology online outline of thesis proposal richwine harvard dissertation guidelines les parties de lintroduction la dissertation philosophique uni marburg medizin dissertation topics supply chain dissertation ideas dissertation topics for mba general degree postgrad early childhood studies dissertation droit dissertation exemple finance dissertation topics 2015 sample thesis topics for nursing event management dissertation questions dissertation website reference alvetta thomas dissertation andreas scheuer dissertation examples le formalisme contractuel dissertation buy dissertation service dissertation loi de finance 2019 order dissertation umi heike hanagarth dissertation abstract write for me affordable dissertation writing cheap dissertations and theses full text jmu background section of dissertation hume dissertation sur les passions explication essay dissertation guantanamo biotechnology dissertation exemple de dissertation philosophique sur la conscience du marketing phd dissertation pdf amorce dissertation theatre tickets current evidence for thoracic aorta type b dissection management digital the ses and dissertations in education lorraine hedtke dissertation help sample argumentative essay apa style dissertation explicative introduction pyziak dissertation order dissertation breakdown dissertation committee listing on cv 20 dissertations sur lespace musique ideas for dissertation in management define dissertation defense ppt sujet dissertation laventure bruno blanckeman the ses and dissertations dissertation plan dialectique methode 1 month to write a dissertation dissertation process timeline diagram dissertation la jurisprudence en droit administratif how long should a dissertation bibliography be les memoires de la guerre dalgerie composition ucla doctoral dissertation approval form sample thesis apa format do my dissertation writing a research journey for money football marketing dissertation dissertation chair responsibilities conclusion de dissertation sur le roman buy buy dissertations online online do my dissertation experts conclusion dans une dissertation juridique structuring a law dissertation titles how to begin a dissertation proposal doctoral dissertation citation software dissertation philo conscience corrige example of graphic design dissertation dissertation opening sentence do my dissertation writing chapter 4 cheap gis dissertation examples ronny thomale dissertation meaning order dissertation statistics order nova applied dissertation procedures manual la dissertation francais msc dissertations lse sandra nitz dissertation writing best app for dissertation sampling techniques in research methodology example dissertation political theory dissertation prospectus neo federalisme dissertation puis je me connaitre moi meme dissertation exemple pour dissertation de philosophie dissertation prepa mpsi dissertation sur le poeme familiale de prevert lobjet du desir en est il la cause dissertation sample phd thesis evaluation format restrictions after aortic dissection surgery finishing the dissertation uri treisman dissertation defense latex template for phd thesis harvard all but dissertation letter exemple de plan detaille dissertation francais 3 minute thesis mcmaster dissertation acknowledgment section mischo kelling dissertation gliederung fur dissertation phd thesis contents page argumentative thesis statement on human trafficking dr mgr medical university dissertation dissertation topics in food microbiology course rasta city dissertation mba thesis topics in marketing pdf example library based dissertation meaning dissertation les dynamiques de la mondialisation postgraduate dissertation titles order best dissertation assistance dissertation sur la corruption pdf lecole assure t elle legalite des chances rupert pfab dissertation outline do my dissertation abstract international for money order dissertation help in london for money rguhs dissertation topics 2005 bachelor thesis presentation ppt quejas la maja y el ruisenor dissertation neuroscience thesis topics thesis dissertations dissertation proposal sample harvard university of liverpool online dissertation dissertation correction service for money dissertation biases and limitations lkk dissertation published theses and dissertations redaction dun sujet de dissertation philosophique conclusion composition histoire 1ere guerre mondiale dissertation uzh mnf 2016 dissertation apa editing mpt 2013 results rguhs dissertation exemple de dissertation critique les belles soeurs timo friedrich dissertation defense dissertation binding london same day service sources of data dissertation do my dissertation revision service order riba silver medal dissertation abstract angela booker dissertation sociolinguistique dissertations katja windt dissertation defense do my sports dissertation online ann bonitatibus dissertation meaning rguhs ayurveda thesis download dissertation writing services illegal idee de dissertation sur le theatre social science quarterly abstract format for dissertations comment on fait une bonne dissertation sample thesis chapter 4 methodology dissertation interview layout format dissertation caracteristiques du romantisme dissertationen lmu tiermedizin berlin mit proquest dissertations tipps schreiben dissertations order how to write a dissertation proposal for money dissertation honey lyrics lenfer cest les autres dissertation gratuite buy psychology dissertation ideas online karibi dede dissertation topics dissertation entrepreneurship pdf do my history dissertation help online mitgutsch konstantin dissertation format buy edit dissertation online write for me nursing dissertations for money philosophie dissertation conscience effort dissertation histoire des idees politiques information technology management dissertation rwth dissertationen online stopwatch essay thesis meaning dissertation uke rechtsmedizin hamburg construction industry dissertation dissertation critical path institute apa format unpublished dissertations do my theses and dissertations online for money mathematics dissertation pdf aortic dissection diagnostic criteria do my custom dissertation writing uk technique de lintroduction la dissertation en causes of ascending aortic dissection how to write a bachelor thesis introduction can you write a dissertation in 3 weeks do my dissertation title page architecture dissertation subjects ntu msc dissertation what to write in dissertation introduction audit dissertation titles write for me writing dissertation questionnaire cheap how to report descriptive statistics in dissertation abstracts dissertation help near me cheap order professional dissertation online exemple dissertation de philo redige walter l arnstein prize for dissertation research in victorian studies write for me writing your dissertation in fifteen minutes a day thesis topic international business write for me dissertation writers goal crossword order mmu dissertation word count dissertation distinction droit prive droit public dissertation techniques daniel stickler dissertation defense dissertation scholarships psychology classics dissertation ideas sociology of sport dissertation titles lerreur et le dol dissertation dissertation on breast cancer marketing dissertation proposal structure how many hours to do a dissertation dissertation on transition from reception to year 1 birmingham university dissertation binding chicago what is the function of a dissertation proposal dissertation philo culture nature dissertation structure university of nottingham ideas psychology dissertation fellowships order concluding a dissertation order dissertation abstracts international journal of climatology completing the dissertation process letter to thesis supervisor dissertation help uk review of american syrian refugees dissertation abstracts les differentes parties d une dissertation pdf nyu dissertation writing workshop order ford foundation dissertation fellowship notification tones best font for dissertation spontaneous coronary artery dissection symptoms dissertation teaching learning introduction de dissertation de francais proquest dissertations and theses tutorial on excel dissertation largent fait le bonheur dissertation sur mobilite sociale write for me dissertation headings order dissertation fellowships 2019 form dissertation medizin basel get my dissertation done library dissertation in dentistry statistician for hire dissertation titles order hr dissertation for money further research dissertation le cumul des mandats dissertation proposal example write for me dissertation writing manual order kosten lektorat dissertation abstracts dissertation writings vg wort dissertation 2018 chevrolet florian klenk dissertation definition complete medical thesis free download order dissertation writing pakistan order dissertation sur le totalitarisme 1ere sti2d relation dom juan sganarelle dissertation dissertation sur soi meme translation write for me dissertation synonym research objectives dissertation examples on leadership comment rediger une dissertation en histoire cardiff university dissertation cover problematisation dissertation topics exemple introduction composition histoire open university msc mathematics dissertation phd dissertation in one sentence une dissertation sur le marketing ntic et organisation du travail dissertation vs verlag dissertation defense write for me dissertation writing day for money le beau et lutile dissertation writing dissertation structure leeds met examples of social work masters dissertations merger and acquisition case study dissertation proposal sample of acknowledgement in thesis pdf do you have to write a dissertation for medical school write for me best dissertation writing website eidesstattliche erklarung dissertation hamburg narrative review dissertation ms word dissertation template ohio dissertations do my proofreading dissertation for money shende dissertation format dissertation topics on environmental law quest ce quun plan dans une dissertation narco terrorism dissertation abstracts dissertation theme argumentation tum dissertationen media tumbler oliver pragal dissertation help order dissertation organization dissertation ideas for child obesity dissertation philosophie lamitie dissertation copyrighted materials computer science dissertation bath dissertation examples for physical education angela tillmann dissertation latest dissertation topics for mba marketing strategy dissertation abstracts international de sciences and engineering la conquete romaine de gaule dissertation order editing dissertation cheap digital marketing master thesis topics anjana jain dissertation undergraduate business dissertation examples for education dissertation grammer how to turn my dissertation into a book on est tous letranger de quelquun dissertation sampling strategy dissertation do my dissertation in a day dissertationen tu graz sujet dissertation cess 2019 write for me dissertation help edinburgh online dissertations examples nursing metaparadigm le cumul des mandats dissertation proposal do my writing a dissertation discussion cheap proposal template for thesis sapiens wenner gren dissertation boris michailow unvollendete dissertation proposal dissertation quote frank engster dissertation examples types research methods dissertations can you do a masters without a dissertation dissertation topics in clinical microbiology fellowship voir est ce savoir dissertation abstracts dissertation abstract advice a dissertation hypothesis examples write for me professional dissertation writer order phd disertation on dissertation bonheur et devoir shodhganga phd thesis in finance michele oh land dissertation writing write for me proquest umi dissertation publishing online write for me creative dissertation order dissertation writing services cost cheap apa dissertation headings 6th hinderclay hall dissertation do my finance dissertation help online publication dissertation kostenloser musica nova clareou ela me dissertation process defending your dissertation dvpw dissertationspreis 2019 nba maplestory lionheart castle proquest dissertations do my dissertation pronunciation online what needs to be included in the introduction of a dissertation comment faire une citation dans dissertation outline chapter 2 dissertation outline mirjana ristic dissertation format thomas waitz dissertation definition aalto library dissertations on bullying do my writing dissertation in 3 days cheap nursing intervention for aortic dissection template dissertation word the best dissertation is a finished dissertation dissertation download sites uncg dissertation calendar dissertations on management do my buy dissertation paper online hotel management dissertation topics sample masters thesis methodology university of manchester politics dissertation abstract article 16 ddhc dissertation topics mba finance dissertation proposal sujet dissertation ses croissance dissertation le theme le livre comment ecrire une dissertation francaise national association of counties dissertation grant program sports journalism thesis topics dissertation verlag zitieren deutsch research design section of dissertation commercialization in football dissertation sample dissertation results help for money university microfilms international dissertation services in los angeles histoire geo dissertation proposal example jaw pain aortic dissection do my business dissertation help write for me geography dissertations order selecting a research problem for preparing a dissertation introduction dissertation poesie pdf writing a methodology for dissertation online zero carbon 2016 dissertations apa division 7 dissertation award oxford brookes dissertation help exemple introduction dissertation sur le theatre nick sousanis dissertation help comment faire un dissertation juridique phd thesis ppt presentation sample order research proposal dissertation sustainable dissertation master thesis topics entrepreneurship dissertation on business communication order dissertation in english online sqa adv higher english dissertation dissertation structure b cubed cambridge dissertation guidelines pourquoi rit on dissertation dissertation 4 seiten theoretical framework example thesis pdf order dissertations uk online plan dissertation philosophie conscience lucia naldi dissertation sample martin hackl dissertation topics gionet cooper dissertation culture universelle dissertation example of an undergraduate dissertation proposal sujet de dissertation philo sur la religion harvard dissertations online how to write a contents page for dissertation dissertation of phd thesis dissertation a t on raison daccuser la technique do my dissertation help writing cheap data analysis dissertation pdf creator dissertation johanna liers dissertation memento mori translation treatment type b aortic dissection plan dissertation philosophie religion order philosophy dissertation online dissertation sur lobligation naturelle vinyl dissertation binders bristol dissertation coaching services for money dissertation order cheap electrical engineering theses and dissertations methodologie dissertation bac title for dissertation in hrm dissertation innovation management companies sven zimmerlin dissertations literature review layout dissertation definition 3 article dissertation writing msc dissertation guidance buy write my dissertation proposal order 50 cent back down dissertation dissertation role de la poesie dissertation uk database dissertation ideas us politics andreas dressel dissertation ntnu phd dissertation proposal format sujet dissertation inconscient university of leeds thesis guidelines buy advertising dissertation order sujet dissertation roman corrige buy professional dissertation writers fsu thesis and dissertation examples of acknowledgement in dissertations fiche methode dissertation seconde write for me how to write your dissertation order writing your dissertation in fifteen minutes a day online elargissement du monde seconde dissertation writing rackham dissertation timeline template types plans dissertation littraire construction dissertation ideas dissertation ideas for health psychology les cours constitutionnelles dissertation defense sample dissertation purpose statement data collection plan thesis child abuse dissertation titles in educational leadership zero carbon 2018 dissertation writing three dissertations on the teachings of jehovah witnesses phrase dintroduction dissertations buy dissertation experts edinburgh university dissertation library dissertation printing perfect bound lucian leahu dissertations michael ardelt dissertation proposal example dissertation ghostwriter cheap do my previous dissertations cheap video games a dissertation corrige dissertation bac francais 2017 ideas for dissertation topics in construction introduction composition sur les memoires de la seconde guerre mondiale hypothesis examples dissertation best laptop for dissertation fabian giehler dissertation examples motivation dissertation ideas in education methode plan de dissertation dissertation uzh rwi trucking nursing dissertation topics on pain of idiom dissertation title on customer service for money chapter 4 thesis sample introduction holger helten dissertation format how to present thesis proposal dissertation consulting jobs dr axel stoll dissertation funny dissertation acknowledgments 2019 chevrolet buy conclusion de dissertation for money buy master dissertation proposal order ucl dissertation cpt code for aortic dissection repair order binding dissertation for money do my best dissertation assistance for money examples of dissertation papers hongtao sun dissertation hospitality management dissertation ideas soas dissertation deadline 2019 anna reska dissertation format buy contents page dissertation cheap dissertation thesis psychology proquest dissertations and theses password recovery lunams cygnus proquest digital dissertations ford foundation dissertation fellowship 2019 dodge look up dissertations corrige bac francais 2014 dissertation decentralisation et deconcentration dissertation proposal researching and writing dissertations in hospitality and tourism hongbo zhang dissertation proposal example order dissertation year fellowship online le parlementarisme allemand dissertation help sujet de dissertation plan analytique bristol university history of art dissertation do my writing dissertation in 10 days cheap dissertation on pharmaceutical industry exemple de dissertation sur la verite master thesis data analysis leeds university dissertation cover sheet master of arts thesis examples writing a good conclusion dissertation order theses dissertations for money methode pour faire un plan de dissertation msc dissertation uk dissertation help nursing do my dissertation writing tools cheap extended project example dissertation edexcel dissertations on mental health and crime dissertation on service improvement cheap peut on ne pas etre soi meme dissertation thesis on customer relationship management in banking sector pdf defense presentation of dissertation gifts for finishing dissertation 20 dissertations la parole office ceyhan prensi 2019 dissertation nonverbale kommunikation dissertation usu dissertation defense how to write a thesis faster dissertation ideas for autism dissertation gerfried sitara la france sennuie dissertation definition cambridge dissertation format sample nursing dissertation gutachten beispiel dissertation proposal example equine nutrition dissertation ideas phd dissertation on learning jamaica example dissertation template abma uk dissertation linconscient philosophie dissertation examples database dissertation ideas accounting the qualitative dissertation master thesis list of tables buy professional dissertation writers online dissertation afrique berceau de lhumanite newspaper how to write undergraduate dissertation introduction comment conclure une dissertation abstracts citation dissertation humanisme literature based dissertation introduction chapter dissertation statement meaning questionnaires used in dissertation criminology library based dissertation help referencing published thesis apa style ricarda steinmayr dissertation format dissertation philosophique corrigee dissertation topics in islamic banking and finance buy dissertation help leeds online de fractionibus continuous dissertation writing deutsche dissertationen datenbank gesprochenes write for me dissertation interview transcription service for money marcel gromaire la guerre dissertation examples graphic design dissertation topic les reforms de onu dissertation format dissertation art moderne comment rediger un bon sujet de dissertation desk study dissertation shock advertising dissertation ideas for education order dissertation writing bot order sports psychology dissertation titles ispi dissertation award buy dissertation proposal on customer service distinction des contentieux administratif dissertation do my education dissertation for money sven krieck dissertation help description of dissertation wissenschaftliche methoden dissertation abstracts la prescription de laction publique et le juge penal dissertation what is a dissertation nz ama citation style dissertation buy phd thesis or dissertation cheap dissertation bury the chains university of texas dissertation defense order history dissertation proposal order verteidigung doktorarbeit medizin powerpoint dissertation ddhc 1793 wreath is a dissertation published la guerre froide cours unpublished dissertation pdf viewer dissertation conflit au theatre do my best dissertation services for money do my legal dissertation cheap iium library thesis dissertations dissertation olaf bogdahn international gmbh exemple dissertation utopie methode commentaire droit penal banking dissertation topics in finance how to write good discussion in thesis dissertation economics example buy dissertation research design for money social work thesis title examples buy business dissertation writing online path to success write a doctoral dissertation writing latex phd thesis template uk dissertation on diversity in the workplace 15000 word dissertation how many pages dissertation droit civil la faute konstantin von notz dissertation abstract dissertation and middle school transition loi des douze tables dissertation proposal example do my dissertation fellowships social science online buy dissertation help in uae for money how long is a political science dissertation dissertation proposal information management alexander bartelt dissertation meaning thesis topics ideas for computer engineering kings college london geography dissertations writing doctoral dissertation pdf to word example of dissertation background aacte dissertation award american dissertation source du droit romain research aims and objectives dissertation defense dissertation droit plan type maximus tyrius dissertations samples vorwort dissertation beispiel bewerbung media and communications dissertation examples thesis response essay exemple dissertation philosophie bac stg diversity management dissertation danksagung dissertation medizinfuchs difference between research report and dissertation contents page psychology dissertation topics exemple dissertation philo sur le travail cesar sampayo dissertation berlin dans la guerre froide dissertation abstracts khieu samphan 1959 doctoral dissertation database dissertation conclusion example pdf dissertation sur leuthanasie pour ou contre le comment faire une bonne composition histoire limpot est il juste dissertation definition musalla yeni dissertation dissertation existe t il des guerres justes find harvard dissertations vorlage doktorarbeit medizin le beau et lutile dissertation meaning dissertation proposal education types of qualitative dissertations dissertation online uk dictionary tips dissertation proposal meaning of dissertation in gujarati dissertation shared service center for money write for me dissertation writing service uae cheap thesis dissertation doctorate dissertation proposal example uk postal code methodology chapter for dissertation quest ce que la beaute dissertation marhofer gk dissertation table of contents example dissertation defense msc dissertation how many references do employers music theory dissertation topics elearning dissertations online de broglie dissertation help write for me dissertation writing depression order alan terry dissertation proposal examples of acknowledgement for a dissertation dissertation length by discipline in schools jochen lawrenz dissertation topics holger helten dissertation topics order dissertation ne demek online martin arft dissertation berlin dissertation drucken us dissertations database architecture thesis synopsis pdf dissertation alexander morlang film morgan state university thesis dissertation handbook dissertation on customer service vorlage gutachten dissertation outline do my dissertation binders order que mettre dans la conclusion dune dissertation de philo dissertation sur la revolution russe doctoral dissertation writing grants biggam masters dissertation pdf free topics for dissertations philosophie dissertation sujet amene dissertation de philosophie synthese dagmar enkelmann dissertation proposal example intellectual property dissertation ideas dissertation philosophie devoir bonheur georg gasteiger dissertation titles order writing dissertation background aiish mysore dissertation introduire citation dans dissertations university of edinburgh thesis binding sizzy saint dissertation topic of dissertation in m ed masking as a coping mechanism dissertation dissertation writers uk cheap le conseil constitutionnel dissertation sample referencing a dissertation harvard style humour noir dissertation format what causes pain in aortic dissection outline proposal dissertation frank flemish dissertation definition dissertation results layout sample proof of concept paper for dissertation write for me affordable dissertation help dissertation le devoir de fidelite entre epoux plan de dissertation sur la constitution 1791 md microbiology thesis topics masters in education dissertation titles dissertation today order write for me best books for dissertation writing dissertation research proposal help ipo underpricing thesis croissance et mondialisation depuis 1850 dissertation defense buy dissertation writing style online methodology chapter of masters dissertations dissertation writing retreat umn for money concurrence et prix dissertation dissertation traductionanglais dissertation lists expose dissertation uni wien phd thesis front page exemple de dissertation sur la technique lotta zernial dissertation examples free digital dissertations apa dissertation award application management of type a aortic dissection why complete a dissertation reference doctoral dissertation proposal example lzhs 7 2 example dissertation dissertation methodology quantitative and qualitative dissertation nul nest mechant volontairement rguhs mpt dissertation do dissertation get scan for plagiarism comment faire une bonne intro de dissertation de philo example history dissertation proposal computing topics dissertation country of origin effect dissertation help dissertation topics on mathematics education how to write a masters thesis proposal lart est inutile dissertation abstract dissertation tu darmstadt biologie totale how to write a dissertation in the reference list master thesis systematic review apa page numbers dissertation writing iatrogenic type a aortic dissection during cardiac surgery lilavati narlikar dissertation examples dissertation englische begriffe kursivle do my buy a phd dissertation cheap mary thumann dissertation dissertation on service delivery for money list of titles of past thesis and dissertation order phd thesis dissertation for money database of african theses and dissertation annonce du plan dans une dissertation de philosophie ana cecilia jimenez doctoral dissertations hospitality and tourism dissertation topics dissertations on louisiana les cours constitutionnelles dissertation writing lewen b 1991 dissertation rwth aachen good dissertation topics for geography pay for dissertation for money service quality thesis pdf margarita mathiopoulos dissertation writing cheap dissertation help uke thesis topics for ms finance dissertation le controle de constitutionnalite des lois en france richwine dissertation committee guidelines evaluation of dissertation dissertation questions learning disabilities apa 6 how to cite a dissertation water architecture dissertation pdf example of dissertation proposal form buy acknowledgements dissertation order how to write website reference in thesis how many words should a thesis be criminology dissertation police write for me law dissertation writing service online psychological dissertation subjects literature based dissertation on fgmnx how to write first class dissertation dans quelle mesure dissertation ses dissertation fellowships and literature dissertation chapters methodology danksagung medizinische doktorarbeit sarrazin dissertation help history dissertation how many references on a resume do my architecture dissertation help cheap chicago style online dissertation ut dissertation submission muhs dissertation topics 2012 composition seconde guerre mondiale premiere es traitement de la dissection aortique bibliographische angabe dissertation definition buying a dissertation cheap expose dissertation verlag thesis binding princeton nj research topics in marketing dissertation sample dissertation proofreading service london order apa format dissertation writing introduction paragraph mla format truman institute doctoral dissertation fellowship dissertation topics related to landscape architecture dissertation methodology sample pdf document do my do my dissertation dissertation ideas for abnormal psychology invite family to dissertation defense introduction dune partie de dissertation durham classics dissertation dissertation fellowships wiki 2018 japanese leberlebendspende dissertation proposal example dissertation fellowships wiki 2014 japanese buy writing thesis and dissertation proposals no dissertation doctorate cheap dissertation philosophie sur le vivant ordonnance article 38 dissertation how to choose a dissertation topic for undergraduate dissertation dun poeme dissertation sur le voyage introduction dissertation philo conscience soi purchase dissertations order umi proquest digital dissertation dissertation proposal examples pdf the core elements of dissertation are dissertation awards online introduire un exemple dissertation retail led regeneration dissertation help dissertation concours gardien de la paix buy dissertation preparation cheap dissertation zur erlangung des doktorgrades englische dissertation definition wikipedia retractation mots de liaison pour une dissertation previous dissertation titles on educational management dutta obstetrics and gynaecology dissertations new product development dissertation pdf creator does anyone fail their dissertation defense acknowledgement for phd dissertation plan analytique dissertation litteraire order writing a dissertation in a week cheap how to write analysis for dissertation dissertation inspection du travail mohr siebeck dissertation merkblatt elterngeld jan marco luczak dissertations sarah schaschek dissertation la cohabitation en france dissertation titles melitensia special collection dissertations in education dissertation concours aide soignant buy dissertation scholarship for money order affordable dissertation write for me professional dissertation writers reviews online outline for dissertation literature review buy dissertation writer uk order order dissertation editing services reviews online philosophy of education dissertations methodologie dissertation juridique peut on rire de tout dissertation dissertation writing in practice cooley phrase daccroche dissertation geographie examples of dissertation rationales for activity sample history dissertation topics dissertation uk examples un exemple de dissertation en histoire anmeldung vg wort dissertation examples elektronische dissertationen uni bonn how to write your undergraduate dissertation review do my history dissertation introduction cheap developing the thesis or dissertation proposal dissertation divorce par consentement mutuel sans juge order social work dissertations cheap nursing dissertation topics on pain kahlil phd dissertation sample topics for argumentative essays jeanita richardson and dissertation order dissertation essay order julia garfield dissertation definition buy buy masters dissertation for money sarah unterkofler dissertation writing dissertation ionesco rhinoceros final dissertation thesis lse dissertation help online ecrire une dissertation litteraire cags umi distinguished dissertation awards current dissertation topics in computer science umi dissertation services order help for dissertation writing online order professional dissertation writers online edinburgh geography dissertation ideas cross cultural marketing dissertation titles dissertationis prodromus theses and dissertations in english literature maplestory scar and targa proquest dissertations dissertation ubi societas ibi jus liu xiaohu dissertation define dissertation introduction example ba hons early childhood studies dissertation ideas for theology how much time does a dissertation take composition histoire seconde la revolution francaise yaelle biro dissertation proposal example dissertation en attendant godot unpublished thesis apa citation dit dissertation titles do my essay and dissertation writing service cheap how many references should be in a dissertation methodologie dissertation droit fiscal la discipline dans enterprise dissertation examples ihr mellon pre dissertation fellowships for african jhu dissertation guidelines corrig dissertation philosophie terminale s was bedeutet inaugural dissertation meaning public finance dissertation topics do my full dissertation help cheap do my citing dissertations cheap dissertation american football doing your masters dissertation pdf free examples of music dissertations evaluation study dissertation marketing branding dissertation ideas in school how to present and defend your dissertation instructional technology dissertations dissertation systemic lupus erythematosus publikationspflicht dissertation proposal thesis template word mac 2018 joscelin bertrice thomas dissertation acknowledgements example dissertation rationale ahmed el shamsi dissertation writing ba hons education early childhood studies dissertation list of dissertation topics for architecture students work buy summary of dissertation dissertation content order dissertation sur le bonheur et le desir samvith srinivas dissertation mohr siebeck verlag dissertation defense order writing a methodology for a dissertation for money social psychology dissertation examples chemical engineering pierre saly dissertation histoire dor dissertation sur la dette publique en europe la dissertation deconomie au bac action research masters dissertation topics le transhumanism dissertation titles lobligation morale de bergson dissertation writing my ph dissertation markus rauber dissertation help gute einleitung dissertation zero carbon homes dissertation writing ideas for thesis topics in economics scholarship dissertation writing dissertationen unizh topics for international relations dissertation dissertation assistance with statistics do my dissertation writing pakistan order 20 dissertations le monde des passions pdf files do my analysis dissertation ascending aortic aneurysm anatomy notre dame graduate school dissertation formatting list of dissertation topics write for me dissertation for phd for money mikael lovkvist dissertation writing plans dissertation bac de francais plan dissertation crise economique sources to use for dissertation louis de broglie doctoral thesis or dissertation buy dissertation proposal outline cheap un sujet de dissertation traite just defended dissertation topics la verite dissertation philo do my dissertation proofreading service london order do my medical dissertation help cheap dissertation on sports nutrition write for me uk dissertation online how to write thesis word 2018 introduction dissertation philo la culture order dissertation help forum online dissertation on erp systems hubert haderthauer dissertation outline dissertations on distance learnng 1990 ma passion cest moi dissertation isyanqar26 toplu katliam dissertation guide to dissertation abstracts sujet dissertation apologue define intranet computer system dissertation uclan dissertation printing dissertation proposal uk examples of alliteration berkeley dissertations online oguzhan alagoz dissertation meaning do my dissertation tutor order help introduction dissertation how to write a good dissertation analysis exemple de dissertation redigee sur le roman shock advertising dissertation ideas accounting sujet dissertation economie bac la dissertation gratuite questions to ask during a dissertation defense dissertation phd proposal sample do my writing thesis and dissertation proposals give me another word for dissertation dissertation book review demystifying dissertation writing pdf download deckblatt dissertation rwth aachen hochschulbibliothek thesis dissertation guide thomas revol dissertation abstract human rights dissertation titles ttu dissertation format how much to get a thesis bound the charlotte w newcombe doctoral dissertation fellowship dissertation philo bac s 2017 von fintel dissertation definition rediger une dissertation en ses android version history a visual timeline for dissertation how to write thesis for dissertation lindividu en droit international dissertation abstract buy apa dissertation order dissertation sur les elements sceniques dissertation peut on juger la culture a laquelle on appartient aktueller forschungsstand dissertation defense parametric architecture dissertation dissertation review comments a dissertation on liberty and necessity pleasure and pain pdf write for me phd dissertation services i havent done my dissertation robert klopfleisch dissertation abstracts dissertation topics in retail management rhodes university the ses and dissertations order quality dissertation help order la nature dissertation philosophie dissertation sur la traite negriere et ses consequences for bad cambridge history faculty dissertation help vg wort dissertation auszahlungsbetrag dissertation tu berlin deckblatt biologie dissertation or dissertation nus dissertation for good le conseil constitutionnel dissertation plan example masters theses and doctoral dissertations titles of theses and dissertations plan de dissertation sur le travail free help dissertation writing and editing defense of dissertation proposal reverse transkription dissertation defense mohr siebeck verlag dissertation abstracts coaching dissertation pdf dissertation discussion chapter pdf bac francais 2019 dissertation abstract phd dissertation proposal guidelines bauernhansl dissertation examples disruptive innovation dissertation outline order finish your dissertation once and for all online dissertation qui suis je asu dissertation format latex dissertation proposal abstract online methodology examples for dissertation troponine dissection aortique pitt dissertation guidelines popular marketing dissertation topics doctoral dissertation coaching immo schilde dissertation definition dissertation totalitarisme second guerre mondiale image write for me lse dissertation help online exemple de sujet de dissertation en philosophie corrige arnim kraatz dissertation definition doris bredthauer dissertation writing appendix dissertation example topics argosy university dissertation guide dissertation litteraire au bac dissertationen verzeichnis schweiz tourismus medical dissertation writing services exemple phrase daccroche dissertation droit dissertation topics for special education dissertation about blue porterweed buy order dissertation order critical evaluation of dissertation dissertation open innovation platform research dissertations education four dissertations david hume abstract dissertation english dissertation sur lagriculture americaine hemorrhoidal ointment in the dissertation buy masters dissertation writing services uk cheap methodology for history dissertation jsums dissertation sample letter for thesis respondents marlies heckwolf dissertation sample est il normal de faire du sport dissertation interview request letter dissertation outline buy buy dissertation help cheap university of manchester msc dissertation discuss the procedure of writing a dissertation history dissertation format thesis dissertation latex template thesis statement for cloud computing dissertations on school leadership llc funny dissertation acknowledgments 2018 gsxr ponge le parti pris des choses dissertation examples dissertation literature review paper phrase daccroche dissertation heros dissertation committee members roles le jeux de lamour et du hasard dissertation do my master degree dissertation online birmingham city university dissertation binding companies mathematics phd thesis latex defending phd dissertation dissertation juridique sur le controle de constitutionnalite how many chapters should a dissertation have dissertation specificite de la regle de droit sociologie des organisations dissertation outline utexas dissertation format dissertation troisieme partie dissertation topic on bank phd thesis citation latex buy dissertation introduction composition seconde guerre mondiale guerre d aneantissement et genocide order dissertation index nadja dwenger dissertation examples sreb dissertation fellowship vorwort juristische dissertation titles international dissertation abstracts connaissance de soi philosophie dissertation defense is it easy to fail a dissertation order dissertations writing services order how many words for phd dissertation how to reference figures in dissertation definition un exemple de dissertation francais vorlage gutachten dissertation defense fashion dissertation methodology exemple de dissertation philo terminale s monographische dissertation definition order starting a dissertation online dissertation fhnw how to make appendix for dissertation dr karl lauterbach dissertation proposal buy dissertation editing for money masters dissertation layout template online dissertation writing order dissertation pronunciation order mthodologie de la dissertation en philosophie can i pass my degree if i fail my dissertation example of a dissertation appendix lenfer cest les autres dissertation write for me dissertation help deal thesis statement on environmental issues what to include in a dissertation proposal macmillan international dissertation research fellowship how long does aortic dissection surgery last jrotc dissertation acknowledgement letter for adviser in thesis abbildungen erstellen dissertation writing introduction sur argumentation dissertation examples dissertation le pilotage des finances publiques literature review structure dissertation format methode dissertation philo bac st2s enfer c est absence des autres dissertation order dissertation research question example timetable for dissertation thesis topics related to landscape architecture dissertationspreis uni bielefeld stud dissertation sur le juge dinstruction best undergraduate dissertations completed dissertations icd 10 code for type 1 aortic dissection sujet de dissertation sur la democratie pdf methode pour une dissertation en philosophie juge constitutionnel dissertation examples bettge susanne dissertation meaning methode de dissertation seconde online doctoral program no dissertation dissertation progress report template editing dissertation uk uk undergraduate dissertation examples methodology how to write a good phd thesis introduction searching dissertations la france un etat unitaire dissertation autrui nest il quun objet dissertation ford foundation dissertation fellowship 2014 dodge subjectivization de la cause dissertation proposal dissertation droit civil l1 compression free dissertation topics hrm example of chapter 4 in a dissertation examples de dissertation juridique pdf wesley seminary dissertation dissertation classification tripartite des infractions mba dissertation database example of thesis proposal in mathematics how to lay out a dissertation phd thesis on health economics educational leadership and management dissertation pdf dissertation table of contents ukraine dissertations on technology education phd dissertation database of pakistani early years dissertation gender methode pour la dissertation de culture generale literature review for phd dissertation sample free dissertations finance composition dhistoire sur les memoires de la guerre dalgerie buy dissertation writing retreat umn order la methode de la dissertation litteraire diversity dissertation fellowships michael overesch dissertation topics no dissertation phd dissertation visuelle kommunikation le libraire dissertation defense dissertation on gambian banking industry annamalai university llm dissertation discuss briefly the essential features of dissertation defense 3 minute thesis uwa order dissertation help in dubai order do my writing methodology for dissertation glauce dissertation application pour faire une dissertation reflective diary dissertation example topics lintime conviction du juge dissertation defense le droit damendement dissertation le marivaudage dissertation abstracts methodologie dissertation 3eme bibtex dissertation statt phd thesis online dissertation finder thesis theoretical framework pdf martin watzinger dissertation summaries of dissertation research dissertationen online uni wien do my dissertation help cyprus masters dissertation word length in computer sentrainer a la dissertation francais phrases de transition en dissertation decision support system dissertation lhomme peut il perdre son humanite dissertation exemple conclusion dissertation how to write up interviews in a dissertation how to write chapter 3 of a thesis methodology dissertation ideas order www dissertation corrige bac francais 2019 dissertation dissertation business questions and answers literature review example dissertation pdf to excel orhan bars dissertation find a doctoral dissertation la fille laide dissertation order best dissertations dissertation topics in innovation management services executive summary of thesis sample order dissertation data collection help for money wiley housewright dissertation award gutenberg vorwort dissertation meaning master thesis proposal finance how to choose a thesis topic ppt dissertation ifrs 15 do my bsc dissertation order constructing a good dissertation for money published history dissertation topics top marketing dissertations dissertation philo bonheur plaisir allison harbin dissertation committee sujet de dissertation histoire 3eme la formation du contract dissertations dissertation on right to education buy doctoral thesis or dissertation online la terreur 1793 dissertation methode de dissertation svt case study methodology dissertation sample write for me conclusion dissertation online banking dissertation ideas constitutional union party dissertation dissertation examples university of leicester secc dissertation research funding awards binding for dissertations free dissertation in hr examples of dissertation methodology section average pages for dissertation difference between dissertation and theses aufbau dissertation chemie proquest dissertations and theses password reset etat unitaire decentralise introduction lexperience combattant pendant la premiere guerre mondiale dissertation proquest dissertations and theses 2008 olympics dissertation sur le theme de la lecture sabine buckl dissertation dissertation topics in quality management lektorat englisch dissertation msc project management dissertation topics le vrai monde michel tremblay dissertation help the dissertation mentor com business intelligence dissertation pdf editor history dissertation fellowships for international students dissertation breakdown cheap blank mathieu dissertation abstracts advertising dissertations dissertation topics on entrepreneurship apa 6th doctoral dissertation reference msc finance dissertations do my dissertation projects for money les mots de liaison pour une dissertation writing dissertation formes detats first class biology dissertation recent thesis topics for hrm is a dissertation a report scott ortman dissertation sample dejudiciarisation du divorce par consentement mutuel dissertation phd comics dissertation writing guide literature based dissertation discussion of results dissertations on economic development order fashion dissertation online buy dissertation project sheri goldstrohm dissertation how to write research methodology chapter in thesis dissertation proposals in finance introduction de la dissertation economique phd dissertation defense procedure lagent fait le bonheur dissertation writing crise du 16 mai 1877 dissertation sample la guerre froide bipolarisation du monde dissertation definition dissertation histoire et geographie tanja von fransecky dissertation titles psychology qualitative dissertation curriculum vitae for dissertation mortality of aortic dissection edit de caracalla dissertation format dissertation service dogs for autism do my doctoral dissertation help online bac de francais dissertation psychology dissertation topic thesis and dissertation pdf to word buy history dissertation introduction cheap piirtola dissertation abstract dissertation upon roast pig essayist crossword extinction de laction publique dissertation titles write for me phd dissertation assistance for money le quinquennat dissertation outline typical phd dissertation length requirements order research proposal for dissertation order write for me dissertation draft for money comment rediger une introduction en dissertation dissertation homework help cheap dissertation process timeline examples buy dissertation academic writing cheap thesis for phd in commerce order dissertation discussion help online free examples dissertation proposals yorku dissertation dissertation binding birmingham university annales sujet dissertation bac the dissertation research process write for me abstract in dissertation online constitution de lan viii dissertation dissertation on the ancient babylonian empire buy writing a good dissertation cheap write for me top dissertation plan de dissertation sur le parlement europeen la peinture sapprend dans les musees dissertation do my phd dissertation proposal cheap order writing a dissertation introduction cheap thomas eigel dissertation meaning evaluation dissertation proposal dissertation warwick university do my finance dissertation help for money rundfunkbeitrag verfassungswidrig dissertations dissertation sur ua published dissertation apa citation best way to choose a dissertation topic masters dissertation introduction length times dissertation titles on law dissertation agreement form ncsu plan de dissertation historique writing theoretical framework dissertation dissertation histoire du droit mthodologie robbe grillet pour un nouveau roman dissertation outline how many words in a thesis chapter internationalization des entreprises dissertation topics marketing communication mix dissertation buy editing services for dissertation online dissertation writing guide topics sujet de dissertation madame bovary sample theoretical conceptual framework thesis dissertation layout for money ed dissertation introduction dissertation francais prepa mmu dissertations uk dissertation uni salzburg rechtswissenschaften wien dissertation sur les regimes totalitaires en europe oxford university dissertation guidelines dissertation leadership research equine business dissertation ideas for psychology yorku ca dissertation academic wiki dissertation dissertation la poesie nous eloigne t elle du reel writing journal article from dissertation dissertation msw doctoral dissertation defense powerpoint write for me i need help with my dissertation for money etre payspan au moyen age dissertation examples examples de dissertation sur largumentations niklot krohn dissertation meaning dissertation writing services in mumbai buy editing dissertation order dissertation binding middlesbrough cpt code for type a aortic dissection illc dissertation series online ketose rind dissertation meaning dissertation croissance economique et developpement durable subjectivization de la cause dissertation proposal example exeter university history phd dissertations dissertation on eu law chicago style citation dissertation footnote 4000 word dissertation write for me dissertation abstracts online online do my dissertation writer for hire cheap dissertation argumentation sujet khouasweb sujets de dissertation order dissertation writing in practice exemple de dissertation en philo stmg roy thomas fielding dissertation pdf file verlag publication dissertation meaning dissertation critique dialectique exemple de demande electrical and electronic engineering final year project ideas bachelor thesis hardcover oder softcover how to write a thesis in 1 week thesis phd database dissertation on hr management dissertations examples of literature reviews mla pe teaching dissertation ideas dissertation korrekturlesen preis dirk wandke dissertation abstract karibi dede dissertation format do my help dissertation defining terms in a dissertation pamela luckau dissertation defense how to write discussion chapter in dissertation write for me phd dissertation writing for money dissertation assignment help valijita de juliana doctoral dissertations international business dissertation ideas for hr dissertation writers cheap buy dissertation binding service coventry cheap write for me uk dissertation writing for money write for me finance dissertation proposal for money how to do secondary research dissertation chapters in phd dissertation methodologie dissertation culture g higher education qualitative dissertation buy thesis dissertations zuhair shihab dissertation help critias proquest digital dissertations que mettre dans une intro de dissertation dissertation proposal sample uk post write for me doctor dissertation for money write for me dissertation proposal template for money crpe 2018 sujets de dissertation university of manchester dissertation cover page exemple dune introduction de dissertation de philosophie antje von dewitz dissertation help doctoral dissertation proposal defense hidden street zakum proquest digital dissertations abstract dissertation format thesis topics ideas for information technology lexistence du consentement dissertation defense how to write an analysis for dissertation la constitution et le pouvoir de letat dissertation eatcs dissertation award dissertation ideas for strength and conditioning researching and writing a dissertation for business student buy economics dissertation for money buy research dissertations for money uclan library dissertation binding uni heidelberg dokumentvorlage dissertation format hayat cixflow dissertation electronic dissertations university arizona dissertation worksheet do my writing dissertation outline for money dissertation abstracts international citation chef doeuvre inconnu dissertation proposal ppt dissertation proposal buy dissertation binders online vuw dissertation muhs dissertation format 2018 thoracic aortic aneurysm anatomy lascenseur social est il en panne dissertation research methods thesis proposal free sites for dissertations order dissertation paper help for money masters dissertations database nursing dissertation topics on diabetes rundfunkbeitrag verfassungswidrig dissertation proposal example all but dissertation cmu baseball dissertation wikileaks cables tips for dissertation presentation ntu dissertation submission ms word dissertation templates dance my dissertation literature review layout dissertation format introduction pour dissertation de philo renaissance carolingienne dissertation abstracts uky dissertation defense jewellery design dissertation gouverneur la france depuis 1946 dissertation titles writing abstract for dissertation example with tables dissertationsdatenbank uni due dissertation medizin la guerre froide composition histoire literature based dissertation criminology board national excellent doctoral dissertation sujet dissertation lhumanisme bits wilp dissertation sample sujet de dissertation proposer au bac gabon hard bound dissertation cost phd dissertation or thesis online order dissertation critique for money order dissertation editing online for money kunstchronik begonnene dissertationen online when do you write a dissertation example of thesis proposal for education msw dissertation sample writing acknowledgements for a dissertation correction dissertation culture generale acs thesis dissertation policy dissertation service public et police administrative cheap nicole becker dissertation outline dissertation research help order vg wort dissertation frist clinic lattente du bonheur dissertation order dissertation writing order how to prepare for your thesis defense elektronische publication dissertation abstract dissertation sur la traite negriere introduction letter loi gombette dissertation examples masters level dissertation examples in history buy masters dissertation help order michelle obama undergraduate dissertation dissertation contents online dissertation topics in criminal law india trente glorieuse dissertation format dissertation report on working capital management appel public a lepargne dissertation buy buy dissertation online online methodologie de la dissertation dialectique martin grininger dissertation dissertation ses bac 2012 fi dissertation research question help order zeitplan expose dissertation beispiel dissertationen tu darmstadt chemie tabelle do my dissertation statistics help uk dissertation outline example finance and business dissertation topics crm dissertation free samples dissertation project definition free law dissertation mla cite dissertation dzejkopp dissertation geography dissertation proposals hoblin pq proquest dissertations uni ulm bibliothek dissertationen deutschland how to become a dissertation coach jack london dissertation outline nursing diagnosis for aortic dissection repair zugang zu dissertationen breastfeeding dissertation literature review write for me dissertation writing experts apa research paper template with table of contents dissertation online uni bonn master in public administration thesis titles childrens nursing dissertation topics lyx thesis vorlage buy write my dissertation online order abstract definition in dissertation dissertation la philosophie est elle dangereuse de lisle dissertation sur larticle 3 de la constitution 1958 les fonctions de la poesie dissertation titles unequal blood pressure aortic dissection sample acknowledgments for dissertation aim of dissertation abstract case study dissertation titles for human samurai camara dissertation help order master dissertation for money thesis on municipal solid waste management in india dissertation on management accounting do my good dissertation online deconstruction music online dissertation inaugural dissertation bedeutung review and acceptance of proposed doctoral thesis dissertation research research methodology section dissertation constructing a good dissertation order dissertation writing manual order msc thesis dissertation marketing thesis pdf philippines how to defend dissertation proposal dissertation la culture rend elle lhomme meilleur effect essay thesis statement examples what is the purpose of a dissertation abstract good introductions to dissertations mondofacto dissertation meaning social work practice dissertation examples nursing water architecture dissertations richard dawkins dissertation kennedy dissertation ideas phd dissertations on managing strategic change process buy qualitative dissertation help phd thesis in human resource management download iles et archipels dans le monde dissertation proquest dissertations theses login www corrige dissertation bac francais 2018 best dissertation writer order dissertation help books for money how to write an abstract for a dissertation pdf sultanbeyli dissertation dissertation verlag kostenlos aortic dissection vs aneurysm usmle dissertation free sample unpublished university of edinburgh politics dissertation examples thesis methodology chapter quantitative do my dissertation write up cheap dissertation citation ieee doctoral thesis front page dissertation formatting service sujet dissertation agregation externe lettres classiques 2017 action civile dissertation writing warwick university dissertation guidelines inaugural dissertation medizinische write for me undergraduate dissertation introduction online phd thesis free download syndicalisme et socialisme dissertation meaning writing dissertation proposal history whole life cycle costing dissertation abstract shaker verlag dissertation format textile thesis topics simple definition of dissertation do my abstract of a dissertation online order dissertation project proposal order dissertation sur la valeur ajoutee credit risk management dissertation pdf writer dissertation database for money university of malaya the ses and dissertations titles apa dissertation easy bib what is the dissertation proposal doctoral dissertations in economics one hundred third annual list waltraud wende dissertations dissertations on homelessness dissertation michel tremblay les bealls soeurs salti roland baker dissertation help write for me dissertation writing google scholar for money dissertation corrige en philosophie dissertation help singapore for money dissertation ethics checklist example latest dissertation topics for mba marketing online dissertation critical discourse analysis methods dissertation sur les passions hume garnier flammarion jeunesse dissertation project proposal cheap nsf doctoral dissertation research improvement grants sample of conclusion in thesis sebastian metzelder dissertation titles do my cheap dissertation help for money buell dissertation colloquium 2018 ina ehnert dissertations write for me university of michigan dissertations cheap ann christine ehlis dissertation meaning topics for a dissertation in psychology dissertation topics for tech computer science how to write a findings chapter for dissertation classification systems for aortic dissection employee retention dissertation topic iv republique dissertation proposal example doctoral dissertation writing services dissertation quest ce que la nature ucsd dissertation order dissertation on marketing online comment bien reussir sa dissertation de philosophie write for me online theses and dissertations order introduction dissertation commerce international international dissertation research fellowship idrf program dissertation support online dissertation andreas judt historiography dissertations presentation analysis and interpretation of data in thesis methodologie de la dissertation en histoire geographie 4eme desir dissertation philosophie icd 10 code for aortic dissection repair speech recognition thesis report format of literature review in thesis order dissertation introduction help online la dissertation dialectique exemple sebastian gradinger dissertation writing zeitplan dissertation literaturwissenschaft linguee jan brademann dissertation writing inaugural dissertation duden deutsch dissertation printing birmingham perrine chancerel dissertation titles do my scientific dissertation la coutume en droit international dissertation la quatrieme republique dissertation proposal example grant dissertation writing ba hons early childhood studies dissertation proposal example undergraduate dissertation grade dis dissertation sample chercher le vent dissertation how to display findings in a dissertation proquest theses and dissertations search engine uea dissertation marking criteria dissertation du bac get it done faster secrets of dissertation success plagiarized his doctoral dissertation anzahl dissertationen deutschland lied proofreading dissertation cheap questions and answers in thesis defense writing a phd thesis literature review dissertation editors minneapolis robert lukowski dissertation sample reference of dissertation write for me methodology in dissertation for money plan de dissertation en histoire benotung dissertation magna university of tennessee dissertations present research findings dissertation nsf dissertation grant proposal dissertation corrige en droit constitutionnel do my writing your dissertation derek swetnam cheap write for me struggling with dissertation online dissertation area of research and focus primary education science dissertation ideas do my dissertation proofreading help hypothesis dissertation online dissertation meaning in punjabi song dissertation critique marie calumet patientenkollektiv dissertations kahuna dissertation dissertation pierre bourdieu habitus alpa gun sido dissertation rachel szekely dissertation a dissertation on oriental gardening history of art dissertation ideas for educational leadership doctoral dissertation assistance for money dissertation guide leeds aefp dissertation award american taner aydogmus dissertation sample data analysis dissertation order do my sociology dissertation proposal order dissertation sur emile ou de leducation dissertation sur tartuffe de moliere fashion illustration dissertation dissertation methodology definition wiki nina hagemann dissertation write for me dissertation writing course for money marcus grosh dissertation haynes dissertation fellowship computer science phd thesis proposal order dissertation scholarships cheap dissertation formatting guidelines for typing introduction dissertation 2nde master thesis topics list 20 dissertations le monde des passions pdf to jpg write for me buy dissertation online online dissertations international management life span after aortic dissection dissertation totalitarisme 1ere l proposal dissertation meaning dissertation in 6 weeks formulierung forschungsfrage dissertation abstract redistribution des revenue dissertation sample frank engster dissertation proposal dissertation open innovation platforms biography for dissertation how to define a dissertation undergraduate dissertation acknowledgement examples dissertation vs thesis pdf dissertation juridique 20 dissertations justice pdf to jpg dissertation philosophie introduction exemple order dissertation on service improvement online critical success factors dissertation proposal example pamela luckau dissertation definition order dissertation resources online do my dissertation proquest perfect dissertation defense meme making a dissertation timetable software philosophie introduction dissertation casey wells dissertation sample dissertation philo exemple sujet uni kiel bibliothek dissertation abstract lektorat dissertation legal phd dissertation writers online phrase amorce dissertation outline wkwsci dissertation characteristics of current doctoral dissertations in education linguistics thesis topic ideas proposal examples for thesis average length dissertation proposal dissertation interview questions for money holm putzke dissertation abstract dissertation ricart brede write for me dissertation methodology section cheap university of leeds dissertation deadline le theatre dissertation introduction discussion part of a dissertation what makes a good dissertation proposal sample of mba thesis in the philippines life expectancy with aortic dissection family business dissertations dissertation le capitalisme et ses consequences examples of acknowledgement in thesis order cheap dissertation writing uk online dissertation development finance usc dissertation committee form cambridge university phd thesis template dissertationen rub medizina dissertation project cloud computing data analysis sample dissertation proposal defense carolin tolksdorf dissertations conradin cramer dissertation abstract writing an acknowledgement for a dissertation ruptured type b aortic dissection danksagung dissertation familie vtm doctoral dissertation and other research experience on resume wolfgang eden dissertation meaning master degree thesis word count doctoral dissertation definition example dissertation questions examples business dissertation format michael scoles dissertation titles rudestam and newton surviving your dissertation a comprehensive guide dissertation philosophie le travail denature t il lhomme possible thesis topics for law ma history dissertation define empirical dissertation buy dissertation writing uk for money national institute of justice dissertation fellowship accession or order number dissertation help with mba dissertation dissertation topics on rural marketing companies leberlebendspende dissertation outline theses and dissertations in applied linguistics journal dissertation structure solent fort phd programs with creative dissertation do my working on dissertation for money tipps schreiben dissertation format asylum seekers dissertation examples for education 3 minute thesis mcgill urheberrechtsverletzung dissertation ucsb grad div dissertation sample do my dissertation help online for money tobias debiel dissertation defense dissertation critical discourse analysis fairclough usf ethnic minority dissertation fellowship write for me dissertation statistics consultant order do my dissertation writing timeline master thesis topics social work dissertation de culture generale enm hewlett foundation iie dissertation fellowship ecu three minute thesis acknowledgments dissertation examples free history cambridge dissertations rpi dissertation template dinky dissertation intro d une dissertation abstract sample dissertation in computer science how long is a literature review in a dissertation do my professional dissertation editing services online order dissertation requirements order write for me can you write a dissertation in a week cheap order digital dissertations cheap writing a method section in a dissertation do my dissertation proposal outline dissertation sample outline dissertation topics on rural marketing policy instead trial aortic dissection 2018 verteidigung medizinische doktorarbeit professional font for thesis thesis dedication to my family dissertation apologue instruire bdne uke dissertation leitfaden pre dissertation grant texas disd dissertation online rwth aachen mechanical engineering dissertation initial proposal example comparative education dissertations dissertation ideas forest school do my msc dissertation for money buy steps to writing a dissertation cheap charlotte newcombe dissertation completion fellowship how to approach writing a dissertation bra dissertation dissertation proposal internet banking parlae dissertation example of table of contents for baby thesis dissertation la methode cobast buy publishing dissertation cheap commonly asked questions in thesis defense comment faire un plan dissertation philo rapport de jury dissertation hec motorsport dissertation ideas health crime dissertation ideas accounting dissertation writing suggestions presenting qualitative data dissertation help mathematics masters dissertation topics dissertation topic proposal example la decentralisation en france dissertation outline tips oral defense dissertation dissertation sur linvestissement et lemploi how to reference a phd dissertation buy dissertation in 2 days for money fernando orejuela dissertation interior design thesis statement examples failed my dissertation defense education dissertation help dissertation lmu abgabengutschrift doctor of theology dissertation dissertation software quality thesis topics on critical discourse analysis dissertation dino bortot jan mending dissertation format order phd dissertation writing service online buy dissertation support uk cheap dissertation brainstorming motivation dissertation ideas for educational leadership veroffentlichung dissertation uni erlangen dissertation proposal examples uk transesophageal echo aortic dissection 50 modeles de dissertations philosophiques domestic violence dissertation ideas example of analysis of data in thesis order dissertation coaching for money la dissertation de la television aera dissertation award 2019 jeep dissertations about mission and vision dissertation urban dictionary andreas blutke dissertation topics write a dissertation in a day corporate reputation management dissertation awards exemple de dissertation sur la guerre que es research proposal en espanol cours de dissertation historique order constructing a good dissertation order martina krogmann dissertation fana asefaw dissertation help order dissertation writing program lea stechmann dissertation examples dissertation topics in dba methodologie de la dissertation en franais pdf order dissertation requirements cheap thesis title proposal for criminology in philippines construction cost overruns dissertation dissertation de culture generale exemple techniques de redaction dissertation texmaker dissertation template word banking law dissertation dissertation encoding law dissertation writing do my how do you spell dissertation for money heike egner dissertation proposal example nicolas ziebarth dissertation proposal ece gatech dissertation schedule le premier ministre sous la veme republique dissertation butin education dissertation topics types of research methods dissertation dissertation introduction subheadings order phd dissertation search cheap sujets dissertation droit administratif tips for writing an ma dissertation dissertation sur la litterature exemple dissertation service public et droit de la concurrence for money dissertation explicative therese raquin movie dissertation zitieren dgps cookie monster phd dissertation proposal format gantt chart example for thesis graduate studies thesis and dissertation phd dissertation writing services uk online brita hohmann dissertation proposal example acknowledgements dissertation boyfriend tag email marketing dissertation dissertation writer finance dissertation methodology thesis synopsis writing guggenheim dissertation writing fellowship further research in dissertation bibliographic index to pg dissertation abstracts international wonderland online ds proquest dissertations dissertation culture generale travail accounting information systems thesis pdf sample dedication page for thesis il faut imaginer sisyphe heureux dissertation abstracts french dissertations online mba thesis title examples critical analysis dissertation writing help dissertation introduction section clinical psychology thesis topics pdf difference between dissertation and proposal phrase damorce dune dissertation meaning international trade dissertation ideas umi proquest dissertation search free do my dissertation analysis for money comment rediger une introduction de dissertation en histoire dissertation vs hypothesis dissertation health education technique de lintroduction la dissertation pedagogique ford dissertation fellowship 2019 silverado dissertation proposal sample uk border difference between research paper dissertation and thesis buy best dissertation writing services reviews wvu dissertation fellowship peine de mort dans le monde dissertation topics quest ce quune bonne gouvernance dentreprise dissertation when to write a psychology dissertation art based research dissertation proquest umi dissertations publishing on amazon dissertation method online where can download dissertations olin shivers dissertation advice meaning dissertation chapter one example order dissertation writers academic writers needed online bds results rguhs dissertation le conformisme dissertation format sujet dissertation philo bonheur buy dissertation abstracts international b the sciences and engineering online wu dissertationen dissertation publizieren verlag dashofer john piper love your enemies dissertation art and design dissertation ideas autism csr and financial performance dissertation defense write for me writing dissertation in 2 weeks bridget nolan dissertation humanities dissertation structure dvpw dissertationspreis 2019 corvette geleitwort dissertation format example of dissertation data analysis thesis and dissertation templates difference between thesis and dissertation pdf download order master dissertation cheap dissertation journey pdf by carol roberts marx manuscrits de 1844 dissertation sample buy uk dissertation writing services cheap noelani arista dissertation proposal example vg wort dissertation steuerrechner efecan bozan aytocan dissertation research aims and objectives dissertation outline how to write dissertations project reports pdf dissertation apologue overture networks dissertation philosophique le travail louis xvi dans la constitution de 1791 dissertation thesis sample apa dissertation zitieren endnote for mac uni bibliothek kiel dissertationen do my write my dissertation uk for money pgce dissertation ideas structure of sociology dissertation dissertation writing services australia dissertation report on banking michelle mccraney dissertation ask me about my dissertation one more time exeter university dissertation plan dissertation droit public cardiff university dissertation format dissertation questionnaire for money writing dissertation background cheap dissertation on customer service online qualitative interview dissertation order dissertation sur le service public cheap glasgow university law dissertation sociology dissertation order springer vieweg dissertation format bibliographische beschreibung dissertation defense buy dissertation questions online dissertation de romantisme what to include in the introduction of your dissertation phd dissertation how many words in english language open directory of the ses and dissertations database dissertation customer service cheap international relations dissertation methodology example corrig dissertation de philo gratuit do my dissertation introduction help order dissertation sur limprevision en droit des contrats nursing dissertation lived experience uni dissertation order chapter 3 dissertation outline ecjs seconde sujets de dissertation dissertation binding kent eu law dissertation titles in public administration dissertation icp dissertation proposal writer cheap lhomme est un animal social dissertation science dissertation pdf how to conclude a dissertation chapter philosophie la culture dissertation barbara budrich dissertations kd4 maplestory proquest dissertations international business dissertation ideas for psychology do my dissertation help help order jakob kreidl dissertation defense apa citation unpublished dissertations le divorce pour separation de fait dissertation apa format for research paper headings order dissertation abstracts international b the sciences and engineering konstantinos karachalios phd dissertation exemples de dissertation en francais dissertationen uni stuttgart 2018 movie conformisme anticonformisme dissertation defense linda riebling dissertation arbeitsplan zeitplan dissertation titles format for writing thesis statement sujet dissertation agregation anglais 2018 acuns dissertation award multicultural dissertation agregation contents page example dissertation papers guidelines for thesis writing pdf buy dissertation help edinburgh cheap do my top dissertation order kumulative dissertation universitat zurich juge administratif et le droit de lunion europeenne dissertation dissertation topics on family business dyslexia dissertation titles auditing dissertation pdf dissertations on student motivation dissertation sur lhumaine condition hot to cite dissertation apa dissertation on second language learning chapter 5 dissertation example apa saisine in rem du juge dinstruction dissertation cheap dissertation writing services india undergraduate dissertation introduction outline albert schliesser dissertation abstracts sample research dissertation proposal thesis personal essay double bind and stress dissertation approche des territoires du quotidien dissertation help presenting findings in library based dissertations gratuites utexas dissertation defense dissertation expressions util es subjectivity statement dissertation titles ieee thesis format introduction dissertation bac francais dissertation topics on low cost airlines gay movement 1970s dissertation technique de dissertation francaise pdf corrige bac de francais 2015 dissertation dissertation structure northumbria police dissertation sur lamour bac service innovation dissertation defense dissertation topics for english language statement of the problem thesis example writing a desk based dissertation abstracts le mal existe il dissertation examples sujet composition histoire seconde chretiente medievale dissertation on comic books evaluative doctorate dissertation research dissertation revision service order music dissertations database order empirical dissertation online methode dissertation svt carmen arranz dissertation aorta dissection icd 10 order apa dissertation dissertation topics of architecture do my research methodology dissertation order buy dissertation services online richard cosgrove dissertation ideas for a social work dissertation bealls department store dissertation dissertation boot camp university of delaware writing an abstract for a dissertation la dissertation en histoire pierre saly asi styla rapresyon dissertation andy goulding dissertation write for me marketing dissertation help for money masters dissertation schedule csulb thesis dissertation do my writing dissertation in 2 weeks cheap ipa psychology dissertation titles tib uni hannover dissertation sample discuss briefly the essential features of dissertations dissertation philosophie exemple stg dissertation binding holloway road apa dissertation abstract reference brian may phd dissertation topics aortic dissection endovascular treatment dissertation hand in status order dissertation writing prices online dissertation report on performance appraisal pdf order mla dissertation dissertation ideas for education sen masters thesis abstract les formes detat dissertation pdf ideas for nursing dissertation topics dissertation revolution francaise 1789 dissertation skills audit writing an excellent dissertation introduction dissertation bac francais 2016 dissertation culture generale em lyon jesus galindez dissertation la garde a vue dissertation abstract lopposition parlementaire en france dissertation abstracts how to get a first on dissertation conclusion sur la guerre froide multinomial logistic regression dissertation do my dissertation assistance writing online dissertation sur le dernier jour dun condamne buy dyslexia dissertation help history dissertation prospectus template tobias angert dissertation proposal example dissertation dedication quotes for graduation science dissertation discussion samples of research methodology for thesis corrige dissertation histoire seconde theoretical approach for dissertation dissertation writing berkeley cheap la machine peut elle remplacer lhomme dissertation le travail assure lindependance dissertation order dissertation academic writing for money search doctoral dissertations samples de letat gendarme a letat providence dissertation dissertationsdatenbank innsbruck map order writing dissertation on ipad order write for me introduction of a dissertation for money desk based research dissertation topics media studies dissertation proposal buy dissertation essays for money dissertation proposals format masters degree dissertations discussion for dissertation example qualitative dissertation 2018 andreas scheuer dissertation definition formal written dissertation synonym order online dissertation editing service for money buy dissertation data analysis dissertation on advertising pdf dissertation la philosophie est elle dangereuse bouchard dissertation online rwth aachen architektur how to start a dissertation question resit masters dissertation mba dissertation writing tips dissertation ideas for sports education edgewood college dissertation handbook exemple de dissertation philosophique sur la conscience soi nlu dissertations vde verlag dissertation abstracts write for me methodology section dissertation order larval therapy dissertation proposal example creative writing phd dissertation book write for me affordable dissertation help order lutilite de la regle de droit dissertation dissertation open innovation challenge scholarly dissertation crossword clue write for me biology dissertation help order dissertation research help online doctoral dissertation and other research experience sample how to change a dissertation in book personality psychology dissertation topics rhul history dissertation alternative format difference between dissertation and research report business management dissertation titles dissertation writing gifts dissertation sur les apologies meaning heiner fangerau dissertation abstracts designing a dissertation questionnaire dissertation droit a lavortement uva dissertation formats quest ce que cest quune dissertation les caracteres de la regle de droit dissertation pdf nicole rieth dissertation examples write for me science dissertation for money haderthauer dissertation abstract how to get 90 in dissertation how to write literature review for thesis ppt latex dissertation bibtex dissertation sur ledit de caracalla how to acknowledge dissertation supervisor 3 minute thesis okstate 60 in dissertation step by guide to dissertation customer service dissertation titles online writing doctoral dissertation 1 how to write a dissertation in one month la france en villes dissertation topics buy dissertation proposal layout cheap writing phd dissertation outline hart c 2005 doing your masters dissertation dissertation dedication to wife university of greenwich dissertation binding order dissertation paper help ieee thesis template latex dissertationen suchen dnbsurvey masters dissertation introduction chapter alphabet master thesis human rights pdf write for me service quality dissertation order classes laborieuses dangereuses dissertation sample dissertation binding london same day three article model dissertation dissertation sur limportance de lecole questionnaire for thesis pdf stefan kolev dissertation proposal example florian scheuer dissertation format traitement du sujet de dissertation buy dissertation history help online dance your phd 2015 chemistry dissertation critique mba thesis proposal kate heavers dissertation definition methodology of dissertation order whats a dissertation for dissertation sur l education msc computer science dissertation titles ussing kammer dissertation abstracts how to write a dissertation analysis claire renaud dissertation 2010 dissertation proposals online psd tha drivah dissertation write for me finish dissertation order office du juge dissertation dissertation de type 1 en geographie sample dissertation prospectus effet direct du droit communautaire dissertation literary review dissertation example papers kumulative dissertation zusammenfassung icd 9 code for aortic dissection la decolonisation en afrique noire dissertation dissertation footnotes how long is an undergraduate thesis methode dissertation litteraire prepa dissertation article 1415 code civil camerounais choosing a dissertation topic in history lse media and communications dissertations on leadership custom dissertation writing help online write for me writing dissertation in 10 days cheap bac philosophie commentaire ou dissertation 4 cities the ses and dissertations order dissertation service public et droit de la concurrence cheap roland baker dissertation outline ucl dissertation help buy dissertation university for money examples de dissertations litteraires phd dissertation writing online ideas for clinical psychology dissertations matthias grzeschik dissertation abstract peut on se passer d un maitre dissertation doctoral thesis vs doctoral dissertation write for me top dissertation help order la guerre froide composition dissertation paper weight corrige dissertation bac ses 2018 dissertation aufbau der arbeit dissertation binding service glasgow do you have to do a dissertation help with my dissertation online comment commencer une dissertations problematique dissertation exemple vorlage projektskizze dissertation sample free dissertations on marketing do my dissertation writing berkeley dissertation critique agaguk dissertation advertising pdf ucl dissertation search buy dissertation abstracts international for money langoisse du roi salomon dissertation definition how to write analysis section of dissertation format dissertation wesley muhammad buy english dissertation help order dissertation philo autrui dissertation topics in environmental microbiology books undergraduate business dissertation examples in education presenting qualitative data in dissertation outline write for me affordable dissertation assistance cheap exemples dintroduction de dissertation juridique negationnisme economique dissertation exemple dissertation generale litteraire dissertation topics for aviation management lse media and communications dissertations for sale la structure dun sujet de dissertation stefan rohrabacher dissertation definition sujet de dissertation sur le marche du travail importance of research question in dissertation another word for dissertation buy dissertation on value for money forums nursing dissertations andreas moosmann dissertations dissertation monarchie constitutionnelle 1791 christoph lindenberger dissertation unibas medizin dissertation abstract masters dissertation pdf to jpg australian theses and dissertation dissertation bpjeps aan sommes nous fait pour travailler dissertation proposal dissertation english dictionary sport dissertation questions who can do my dissertation best way to start your dissertation write for me methodology dissertation proposal order do my dissertation research design harvard referencing phd dissertation examples dissertation ideas for management evangelina rodriguez doctoral dissertations thesis topics for public administration philippines what is a psychology thesis dissertation tu graz gehalt buy mba dissertation cheap child abuse dissertation titles buy dissertation on service learning online stephan menzel dissertation sample dissertation interview transcription service cheap buy crm dissertation online non adherence concept dissertations dissertations in social psychology ideas for dissertations dissertation proposal oral presentation ideas order dissertation writing service usa methodologie dissertation sous officier gendarmerie sample draft dissertation proposal methode de dissertation franaise davin pavlas dissertation proposal example a quoi sert la philosophie dissertation pdf art history dissertation example msc dissertation only doctorates sample dba dissertation le survenant dissertation definition sujet de dissertation bac francais 2014 dissertation timeline examples images order dissertation writing services org for money this dissertation is dedicated to developing dissertation on sra naoum dissertation research and writing for construction students do my economic dissertation for money verlag dissertation kosten david vasella dissertation methodologie de la dissertation de ses write for me dissertation advice online write for me dissertation editors order apa thesis template word 2019 architecture dissertation topic list sujets dissertation droit administratif des biens roy thomas fielding dissertation pdf reader m tech thesis topics machine design doctoral dissertation editors in ohio order dissertation writing advice online plan dissertation francais seconde dissertation geographie methode dissertation lmu tiermedizin berlin holm putzke dissertation defense noah webster dissertations on the english language 1789 george psychometric society dissertation award american jeannette armstrong dissertation examples starting research an introduction to academic and dissertation writing pdf dissertation recommendations for future work dissertation writing service uk samsung fonctions de la poesie dissertation writing labsolutisme monarchique dissertation abstract buy dissertation editing services rates order obesity dissertation titles abstract thesis sample tagalog nora szech dissertation sample order writing dissertation university leicester what is a synopsis of a dissertation l ecume des jours dissertation ncu thesis template dissertation mit welchem programm schreiben human resource dissertation ideas apprendre rdiger une dissertation social sciences dissertation structure dissertation dissertation dissertation writing an abstract in a dissertation dissertation le tiers au contrat best dissertation writing services uk online how to present thesis ppt student dissertation one page thesis summary comment faire une bonne dissertation francaise masters dissertation methodology template drickey n a dissertation alejandro chaoul dissertation help lil durk brick squad dissertation writing up interview findings dissertation abstract buy mba dissertation weai dissertation workshop example physical geography dissertations inhalt vorwort dissertation abstract inaugural dissertation dissertation chapter 1 format le gouvernement des juges dissertation titles acknowledgment in dissertation example action research masters dissertation image florian klenk dissertation format cardiff jomec dissertations database quest ce quune ouverture en dissertation intro dissertation plan dialectique annonce de plan pour dissertation do my best dissertation help uk for money library based sociology dissertation advice writing dissertation the night before order order academic dissertation writing cheap how long does a dissertation introduction have to be mary kay dissertation competition 2018 university of malaya the ses and dissertations available from proquest latex bibtex incollection write for me geography dissertations cheap order custom dissertation proposal methodologie dissertation droit de la famille dissertation free master topic do my writing an introduction to a dissertation order introduction dissertation francaise university of utah dissertation dissertation uni stuttgart chemie online writing about methodology for a dissertation green marketing dissertation pdf viewer david davies dissertation labsolutisme monarchique dissertation definition dissertation only phd theology oxford dissertation binding university of kent composition guerre froide plan dissertation example hospitality sample thesis title in information technology sample introduction for psychology research paper do my history dissertation for money master thesis topics business administration sustainable design dissertation examples for education dissertation svt terminale s quapporte la lecture dun roman dissertation dissertation fundng sources hud dissertation fellowship examples of msc dissertation free mba dissertation projects was gehort in die einleitung einer doktorarbeit buy uk dissertation service online how to make a dissertation presentation rechtsgeschichtliche dissertation meaning andreas scheuer dissertation sample dissertation printing and binding swansea ngri dissertation 2017 the catcher in the rye dissertation dissertation reflective report writing foobar2000 resampler ssrc dissertation