Running late with the deadline for your work? Then we are your reliable assistant in paper help.
Get ready to ask for our assistance when you need essays, research or course works, reports, case studies, etc. Our experts have seen it all and are ready to start working on your assignment right away. Go for it!
With over 6 years of experience in the custom writing service, our team of support agents, managers, editors and writers has got a lot of knowledge about everything that may be required by you. Here’s what you get for sure when cooperating with us:
Everyone needs some paper help from time to time, because we are only human.
Our prices start at $10 per page for works completed from scratch and from only $6 per page you need to be edited and proofread.
What factors influence the cost of our paper writing services? There are 5 of them:
You’re a lucky client! Why? Because you never pay for everything. You have lots of freebies to go with every single assignment. They are:
Asking for our paper writing help, you don’t only pay us. We also pay you! You can receive up to 15% bonuses back and even earn money with our referral program.
We understand that sometimes you may want your deeds to go unknown. That is why we guarantee your complete privacy and security with our paper help writing service. After registration, you receive a unique ID and that is the only thing along with your instructions visible to our experts. Only our support team will see all the details you provide to be able to contact you in case any questions arise and send you a happy birthday discount on your special day.
Our custom writing service is completely ethical and provides busy students with great resources for their assignments. In the modern world when we need to do a lot of things at the same time, it’s nice to know you can count on someone for back up. We are always here to create the needed sample or perfect your work through editing/proofreading or explain the solutions to any problems you may have. Find out how much more free time you can get with our writing help.
Lutheran gassire lute analysis report latest weather report in the phils Downtown Brooklyn campus Livingston, dictator movie review 123telugucinima thesis on business intelligence at umae 1st Avenue zip 10035 need someone to do my dissertation abstract on financial aid as soon as possible Livingston County Empire State College, Saratoga Springs human trafficking in texas essay 139th Street, West zip 10030. Peconic purchase dissertation methodology on accounting as soon as possible South Street zip 10005 employee tips report form New York State College of Veterinary Medicine gUIdEd ReADiNg centERS 4Th grADE burnham park academy ofsted report school.
All right, I hope everyone enjoyed the music this morning to get you going. Anyway, I'd like to open the proceedings. My name is Mark Stolorow, and I'm a NIST employee, National Institute of Standards and Technology. >> [LAUGH] >> What was that? I can't smile any wider than this, this is as big as it gets. So all right, and I wanted to welcome everybody this morning. It's a large group for an early time at 8:00, and I'm so grateful that you're here. We also have a large number of people who are on webcast, and to those who are online and watching us, welcome and good morning to you. I also wanted to mention, as the NIST Director for OSAC Affairs. It's a pleasure for NIST to have the opportunity to administer the annual status updates for the accomplishments at OSAC, and I am not the one who will be doing it. I'm going to be introducing those who will. And in order to give them time to give you their jokes, I wanted to make sure that I got off the podium. So I'd like to begin the introduction this morning with the new Chairman of the Forensic Science Standards Board at OSAC Starting October 1, Steve Johnson from Ideal Innovations where he is the Senior Vice President. And manages, among other things, Department of Defense contracts in forensic science when doing his day job. But we have the benefit of Steve also doing another job that requires almost as many hours. And that is being the Charmian of the FSSB. So without any further ado, I'd like to introduce Steve Johnson. >> [APPLAUSE] >> [LAUGH] >> Thanks, Mark. I probably won't stay at the podium very long because I don't want you to be distracted. As long as I have to. If you're expecting somebody younger, better looking and more intelligent, that would have been Jeremy Triplett. He's the man I replaced. I never leave the room when they are looking for volunteers to serve as a chair of anything. With that in mind I just want to give you a little bit of information. I love it when a plan comes together. A little bit about the FSSB, as Mark alluded. Forensic Science Standards Board is some individuals with vested interest in how we move [INAUDIBLE] incredibly difficult but rewarding effort, and that's to develop standards in all the disciplines that we represent. And with luck I'll have some slides that will facilitate that. There we go, awesome. So 550 members, roughly, that are serving in OSAC right now, with most of them, the vast majority in fact, well over 50% practitioners. But that does not exclude the folks that have a vested interest as well, the statisticians, researchers, academics, legal community. And before I get too far down the path, I'll be all over the map, but I will try to reflect what's on the slide. Cuz I think you'll be able to capture these slides about a week from now, they'll be posted in a PDF format on a website so if I miss something feel free to go to the website a week from now and pick up on anything. Also we have an opportunity at the end of these presentations to ask us questions. Either directly or via email, so don't feel like you got to capture it right here. 25 different disciplines right now represented by OSAC, it started out a couple less than that. We've added digital evidence and crime scenes since the initial establishment of this organization. And 250, how many AAFS members do we have your presently in the audience, including myself? The AAFS represents the vast majority of the folks that are serving on OSAC right now. Take a great deal of pride in the fact that you represent so well in this organization. Just under half of the folks serving on OSAC are AAFS members. I think that's an incredible tribute to you as an organization. And a lot of us serve in multiple organizations. I'm a member of three different forensics organizations, and I'm sure many of you are as well. That just articulates the the urgency and the importance of this work that we do. And part of the FSSB membership is including some of these organizations. Soft Name, Aft-DII, ASCLD and the academy are all represented on the FSSB. When this all started in early 2014, the academy was instrumental in getting this off the ground. They were in the primary talks that started the whole program. But they recognized the need to get other organizations involved that have an interest in this process. So other organizations, associations were included. It's done nothing but made things better for us as far as moving forward is concerned. As Mark alluded, I'm the new Chair. Jeremy, who served that position the first three years of our organization, had probably had enough, and realized it might be time to step away. Mark also alluded, it's very time consuming. Like any organization such as this, it's volunteer. It requires dedication. You're not being paid, it's taking time away from your regular job. I'm already on a number of hit lists at work because I'm really, really there. But I'm close enough to retirement, I can take a hit or two. And also in addition to my being appointed or elected chairmanship we appointed three new members to represent either SACs or our association representatives. And that's Ray Wickenheiser with ASCLD, David Fowler with NAME and Melissa Gische replaced Arthur Nicklin, w ho is the pattern evidence chair. So, Can you hear me now? Okay, awesome, I need to get off the stage. [LAUGH] We've all discussed basically what the standards boards makeup is in addition to those association representatives and the SAC chairs. There's research scientists and most recently the chairs of the various resource committees. Quality infrastructure, legal and human factors initially were just there to provide us some insight, input, advice, if you will. We found their representation on the board was gonna be valuable as actual voting members. So they were included as voting members within the last year or so. There are 25, as we described, 25 disciplines. I'm not gonna name them all, but suffice to say it's a good cross section of the disciplines that are Currently working in the United States in the forensic sciences. The numbers don't really say it all. When we use percentages sometimes it doesn't really identify the numbers of people actually engaged. To see 1% judge is just not fair to Chris, and other judges that may be involved in this process. So basically multiply those numbers by five, and get a pretty good idea of the number of people serving in that capacity to support our effort. So basically we herd the cats a little bit, we're more or less an administrative support for the organization. Forensic Science Standards Board implies maybe a little bit more heavy-handed work than we do. We do work on the governing documents that support our organization, the charter and the bylaws, the terms of reference documents, those are all reviewed and developed and approved by us. Of course, we are here to encourage standards development and application. And I would actually defer to the OSAC affairs folks for that bullet. They really carry the water for this organization, Mark, and JP, Donna, Matt, Sabrina, they're all doing a great job helping us stay out of trouble and that's a pretty tall order, frankly. So one of our other main duties is to approve these standards that come to us. We trust the organization and its subcommittees to develop the standards that we need to get out there for approval. It's not that we're a rubber stamp per se, but we do trust the judgment of a lot of these subcommittees to get us the documents and the standards that they want us to review. And then it goes into a standards development organization for a final approval process. I'll get into the needs and gaps, on a following slide. What are we doing? Well, we are developing these documents and getting some technical publications out for public consumption. We're proud of the framework for harmonizing forensic science practices, and digital multimedia evidence. That was a real challenge for us, but quite an achievement for the folks that put that document together. I can tell you, frankly, a lot of work went into this document. Not only from the personal development standpoint of the folks that authored it, but the people that provided input and the back and forth that occurred thereafter. There was a lot of back and forth but it was all taken with the right frame of mind, very professional in the process, and we'll continue to see that kind of document development as we move forward. I know Bill Thompson's group is working on a document right now, and he's received some feedback from the board, and others. And I'm excited about more and more of these documents being produced by our organization as we move forward. The standards bulletin started just this last fall. Keep you updated on those standards that are being pushed out for review and approval. And we're working on how to prioritize our work in the organization. I'll get into that a little bit more in a later slide, but I'm seeing some movement in the right direction, let's say, on how we're gonna make this organization work better for us and for the forensic science community. So as of right now we've got the registry's approval. It was a guidelines and standards registry, now it's just a registry. We've put them all under one write-off, a lot of standards in development. Eight standards that are on the registry with four in the approval process. That doesn't seem like a lot, but if you've ever been involved in developing a standard, and getting it out there, reviewed, approved, embedded, it's a challenge. And doing it as a group the size that we are, has made it even more daunting a challenge. But once the ball gets rolling, I think we'll see a lot more of these standards get approved, and they will start coming out in much larger numbers. We get a lot of help from some of these other standards development organizations in addition to the Academy Standards Board. There's ASTM, ISO, and National Fire Prevention Association, American Dental Association. They're all very helpful in us developing standards as they apply to those disciplines that have the best interest and mind for the specific SDO. So, part of what are we are looking for, as an organization, is to fill some of the gaps from a research standpoint that the national academies of science report certainly articulated back in 2009. One of the complaints about a lot of disciplines is a lack of research, a lack of scientific foundation for their discipline. So we're encouraging our SACs and subcommittees to try and identify what those research need gaps are, and that we can reach out to the NIJ and potentially other resources, to provide us the funding and support to get some of those research efforts underway. We're also looking at what applications between disciplines can benefit the disciplines, working together, and that's interdisciplinary projects. There's certainly a lot of crossover in a lot of what we do. For instance, speaking personally, when I was working in the lab, we had to be very cognizant of potential DNA evidence before we processed an item for perhaps gunshot residue, or fingerprints, or a trace. So understanding the needs of each discipline will help us moving forward with these interdisciplinary projects. And of course, we all need to recognize, despite what we believe are all our strengths, there are weaknesses that need to be addressed, and to take corrective action if necessary. So this is a busy slide, and I wanna thank Karen for giving it to me and cutting it down. Actually this was much smaller than it was originally presented, but there is a limited font, you can't go below seven. So we didn't want to get below seven on the font. Needless to say, I think the importance about this slide is, this lexicon is a huge lift for us as an organization. We're talking over 4,000 terms that we've identified, that have relevance to us in the forensic science community. A lot of these, we still don't know the actual source thereof, and if you're gonna do a lexicon and do it right, you need to do it right identify source. So you'll see a lot of these that will have a source yet to be identified. Source being verified, we want to let you know that this is ready for publication. Next week the lexicon will be available to you, feel free when you get an opportunity to look and see it on our site. If you have any suggestions, maybe find a source for us, that would be nice. We would appreciate it, believe me. There's a number of people and some of them in this room, Donna, Chris Kruge, Mindy Rains, of course Karen Reczek for the work they did putting this together, and Bruce Coolhan as well. Again, a lot of volunteer time and effort went into this. I think Karen told me she put in 30 hours one weekend alone just to try and get this thing ready for publication. That's a lot of time, again, a reference to the work we do out of pocket, no reimbursement for us, and our effort. But also, explains the fact that we are really involved, and believe in this process. The 29 terms, there were some, what we call 29 terms, that were believed to be somewhat problematic in the legal community. Dr Richard Vorder Bruegge took it upon himself, Patrick Wiessini was involved with this, George Cronin, Angelo Dela Manna, Paul Kish, Karen again. Poor Karen, does she ever sleep? But these terms became a challenge for us to try and work with the legal community and Judge Ployd was very helpful with the Legal Resource Committee on how do we really get our arms around some of these terms, that sometimes are Amongst us in the discipline we take for granted as far as how they're applied, but in the public eye or the legal community, they aren't as easily recognized. They're all up there. I'm not going to go through every term, but I think you've all seen them before. A simple term like bias, to us may mean one or two or three or four different, have four different meanings. Depends on the application, so we're encouraging our subcommittees to take that into consideration. When we talk about bias, how does it apply to you as a discipline? How does it apply to you as you testify in court or as you do you work in the lab or at the bench? So, this is an ongoing process. I think we found what seemed to be a fairly easy thing to do. Richard found out wasn't that easy. But we've certainly continued to work on it, and I can't thank Richard and others enough for the heavy lifting on this particular subsection of our lexicon. So we've got some challenges on the horizon. We're working on a compute inclusion scales in the Pattern Disciplines, trying to find some commonalities and common ground in the conclusions that we come to in the Pattern Disciplines. As I described earlier, definitions can vary between disciplines. We wanna make sure that we either establish some consistency or articulate the differences so they're satisfying our customer, and which ones are the most important significant differences. How do we define that, what's a significant difference? We're gonna have to do a better job of articulating that. Who do we write these standards for? It's not just us, as disciplinarians or practitioners. But we have to understand there's a community out there that uses these as well, the legal community. Even the public at large has some buy-in in our stakeholders in this. And then these documents as we develop them, we recognize they're not gonna be the end all be all. They're not gonna be perfect. We want them to be treated as living documents. We want people to continue to criticize in a professional way the documents so that we can make them better moving forward. And finally, the uncertainty in qualitative identifications. Identification, even saying identification sounds like you're getting tomatoes thrown at you if you say identification. But we have to understand terms like when I was a kid,100 years ago, to the exclusion of all others. To 100% certainty, no longer reasonably or remotely acceptable in a court of law. So what's the next step? Where do we go to best articulate what we're trying to say about our analysis, about our decision-making process? A lot of work to do. A lot of work to do within the OSACs. We were trying to improve the registry approval process. It can be very tedious, we're trying to streamline it as much as much as we can. I know there's been frustration expressed among some of the sub-committees about the process itself, but we've encouraged input from those subcommittees to make the process a little less tedious. We've worked on the merit worksheets and guidance documents, to help people put these documents for review together. We're including the resource committees and statistician task groups as part of the FSSB I mentioned earlier. They were just peripheral to us for a period of time. Their involvement, their active engaging with us as a board has been invaluable. I mean, it's just been phenomenal the work, that we've accomplished, once they became full members, and voting members of our board continuing to work with our outside stakeholders. We have to continue to reach out to the government, Private industry. Others that can potentially help us and if nothing else, give us a better perspective, a different perspective of what we're trying to accomplish. And then promoting the implementation of the OSAC's standards to service providers is our ultimate goal. We'll continue to plug away at that as we move forward. Priorities, short term and long term strategic plan. We're calling these roadmaps. We're trying to figure out the best way to plot a course, not only for the organization at large but for all the subcommittees and the SACs that are involved underneath it. This is something that's challenging in and of itself and because everybody has a right idea or a best idea how to best attack that problem. We're talking about putting together some kind of a template of not necessarily how to do it, but what ideas we want to see from these roadmaps we're putting together. We're definitely trying to reach outside of OSAC to engage our stakeholders. We've got an implementation plan that's identifying the various elements of the forensic scientists and the stakeholders from without to bring them together and best articulate our mission. Of course, we wanna get more registry-approved documents out there and standards out there for everybody. Continue to work with the research and development gaps. Increase our publications. We talked about a couple of them that are in the works or already published. I think we'll see a lot more of them over the next 12 months. And then this implementation plan I kinda alluded to, we're hopefully getting enough of a plan together to involve the great community of forensic scientists. And the legal community, and the public, in a joint effort to make this the best it can possibly be. So can you help us? You're darn tootin. I want people to really consider getting themselves involved in this organization. As I mentioned, 550 roughly, current members 350 roughly affiliates. But there's 2,000, nearly 2,000 JP that have actually applied to be involved. So that says a lot about the people that are eager to be a part of this operation. Don't let that number fool you. This organization will continue to turn people over. We roughly lose a quarter to a third of our membership every year, with the fact that our term limits are coming up. And it will definitely be a third every year once we've gotten into the second end of our second term. We're allowed two consecutive three year terms. At the end of that second term, you have to step away from the organization and the role you're serving in at that time. So there will be a lot of turnover, and it's important that the people that are involved in the forensic sciences, and have a vested interest in a specific discipline, that they apply and get involved with the organization. Feel free to subscribe to our newsletter. It's available to all at the OSAC site. There will be a website address coming up here quickly. And of course, the email blasts and standard bulletins is another way we keep our membership and our stakeholders informed. And we are asking all of our stakeholders to encourage and promote the use of standards in the OSAC registry. And we're all here, we wear these nice little name tags. If you have any questions that we haven't answered in our briefings that we give you today. Those of you that will be here all day, I have to fly out of here during the morning, but there will be others here during the week. If you happen to be here all week, feel free to grab one of us, ask us questions. I know I'm ripping through this rather quickly, but the people that are important. I'm just the chairman, the people that are important are gonna be coming up and speaking next. And they're the ones that are gonna be able to to give you a better idea of the specific things that are going on within their stacks and subcommittees. So that's why I want to get through my limited piece and give these people a chance to speak. So there's your website, www.nist.gov/osac. It's pretty easy to remember. I also have an opportunity to hand out some flyers today that will involve not only what I discussed but what everybody else will be discussing, some hard copy documents. If you wanna take something with you and throw it in your luggage. It's always fun to take stuff home. So without further ado, I'll turn the mic back over to Mark. I appreciate your time and attention. Thank you very much. >> [APPLAUSE] >> Thanks very much. Our next speaker is Chairman of the Human Factors Subcommittee, I'm sorry, the Human Factors Resource Committee. And Professor William Johnson who was made professor at the University of California, and [INAUDIBLE] also a witness, expert witness, [NOISE] criminal cases as well as author of an extensive list, impressive list of references. So without any further ado. >> [INAUDIBLE] >> Thank you very much. >> [INAUDIBLE] All right, a quick review of the Human Factors Committee and what it does in OSAC. I guess I'll stand back here. We're a resource committee. We give advice and suggestions to the Forensic Science Standards Board. And we work with the various subcommittees as they're creating standards trying to give them the benefit of a human-factors point of view. These are our members, a lot of cognitive scientists and psychologists, quite a distinguished group, I think. Tom Albright is a leading cognitive neuroscientist, member of the National Academy of Sciences. Probably knows more than anybody in the world about what's going on in your brain when you're thinking hard about an evidence analysis problem. Hal Arkes from Ohio State, a psychologist who's done extensive research on medical decision-making. And has done a lot of great work on helping physicians make better diagnostic judgments and treatment decisions. Deborah Davis has published a lot in decision-making legal system, and does a lot of judicial training on science issues. John Holloway runs the Quattrone Center for the Improvement of Justice at the University of Pennsylvania Law school. A group that has extensive experience doing root cause analysis of organizational failures and other errors and problems. Rick Lempert ran the Social Science Programs at the National Science Foundation for a number of years. So he's an expert on evaluating social science. He's also a faculty member in sociology and law at Michigan. Erin Morris, a psychologist, did a dissertation on how juries interpret and understand DNA evidence. Michael Risinger's written extensively about validation of forensic science. He's a law professor and about bias in forensic science. Dan Simon, professor of psychology and law at USC has written a famous book about decision-making in the legal system. So we've got a lot of people who study cognitive science issues who I think have something to contribute. Now, I know there's some concern about this. I overheard one OSAC member when I was not listening, saying why do we need a bunch of shrinks to tell us what to do? >> [LAUGH] >> And I was thinking, well first of all, we're not really shrinks. I know we think of ourselves as researchers. And we don't have the power to tell you what to do. I mean, we can make suggestions and make advice. I guess we now have one vote on the standards board. But we're not really telling people what to do, we're trying to help people. What we are really doing is what the late Bryan Found in his rightness called cognitive forensics. Bryan Found, an ETL drawer who's here today, and others have been writing academically about the ways in which cognitive science can contribute to the improvement of the scientific basis of forensic science. The rationale basically has to do with understanding and making better use of the primary instrument of forensic science. And what is that instrument? What's the most important instrument? Well, this is a slide I borrowed from ETL drawer. It's the human brain. Decision-making by human beings is critical to many forensic science functions and talking to people who study that might be useful. At least that's the idea between the human factors committee, and we try to be of help. We have several special concerns where we think we have a special contribution to make to OSAC. One is reducing the potential for bias, and so I'll talk about that bit in a moment. The other area that we're spending a lot of time on is improving assessment of examiner performance. How do we measure how well people are doing in their forensic science work? And how do we help them get better? How do we give them feedback? How do we assess? How do we choose the right people? Those kinds of issues. And the third area where we think we have a lot to contribute has to do with improving communication of findings to a lay audience, and a lot of the members of the human factors committee. We also have a number of affiliates who are actually engaged in research on communication of science to lay audiences. And I think our being involved in OSAC, I think helps direct our research in ways that are beneficial to forensic science. And the research itself I think is gonna be helpful to forensic science. So let me talk first about bias, and what we're doing with regard to bias. When we look at standards coming out of the subcommittees, we often think about, well, is there a potential for bias here? And if there is, what could we do to reduce and mitigate that? So identifying risks of bias is part of what we think about and do. And we spend a lot of time talking to subcommittees encouraging them to think about mitigation procedures for bias. Is it really necessary that examiners be exposed to what we think is potentially biasing information? Can that exposure be delayed until after they make critical decisions and so on? So talking about how to create procedures to minimize bias is a lot of what we do. And again, we're not dictating to anybody what they have to do. We're trying to make suggestions for what might work. And we spend a lot of time actually sharing examples of good practice. So when we find labs and people who are doing things that we think are good from a cognitive science or psychological point of view, we try to let people know about that. A lot of members of the committee are engaged in academic writing on these issues. And I think by talking to forensic scientists, it also enriches our understanding of the problems. One of the big issues when we talk about bias is this question of task-relevance. I mean, bias typically is defined as being influenced by some factor that's not supposed to influence you by an improper factor. Well, in order to talk about bias intelligently, in forensic science, you have to then think about what sort of factors should and should not be influencing a forensic examiner when [INAUDIBLE] particular kind of evaluation? What is task relevant and what is not? And in some disciplines, that's pretty obvious. The fingerprint examiner's supposed to be drawing conclusions from the fingerprint, not based upon other forensic evidence or the suspect's past record, but in other disciplines, it becomes a very tricky issue. We've had really interesting discussions with fire scene investigators, forensic pathologist, blood pattern, examiners about what is and is not task relevant. It depends on how you define the task. And how you define the task often involves really important issues about what rule we expect the examiner to play in the Criminal Justice System in the legal process. All of which, Are issues that forensic scientists need to think carefully about before they write standards about how to do these tests. So we think this kind of discussion is helpful, and as I said we've been engaging with a number of forensic labs on implementing bias reduction procedures, and sharing those results. A lot of the first reaction when you talk about bias, and bias reduction for many people is to say, we can't do that, that's not practical. But people who claim that there's no practical solution to the problem of bias are being answered fairly quickly by people who are trying out solutions. So we think we're having some impact there. Guidance on Testing Examiner Performance is another issue on which we're trying to be helpful. This whole question of, how do forensic scientists, as a field, how do we know how well we're doing? How can we improve how well we're doing? There has to be some measurement of performance, particularly, those areas of performance that depend heavily on the judgement of the human being. And there are lots of those in forensic science. So, testing examiner performance and how to do it is an important issue for validation of procedures, for quality assurance, for training even personnel selection. Part of what we've been trying to do in this area is provide tips on design, methods of analysis, and reporting. We've had lots of discussions on these issues. I think Steve mentioned we've been trying to compile a lot of our advice into a technical publication for OSAC. Guidelines on how to do validation studies in forensic science that we think will be helpful. I think some of the studies that have been done by forensic scientists we think have not been done in the.most optimal way, and somebody want to make suggestions. Again, none of this is mandatory, but we think it might be helpful for forensic scientists to have a group of experts, like to say, here's how we think you should do studies on various issues. Rather than say, go read these 20 textbooks and try to figure it out yourself, here's our suggestions specifically for you. So we've engaged on this with a lot of people in OSAC, and we have in process a proposed technical publication. We've been getting a lot of feedback from different subcommittees about how to make it more helpful. Finally, we've been involved pretty heavily in the question of reporting standards. How do forensic scientists talk about their conclusions in reports, and in testimony? In particular, there's been widespread discussion in OSAC, there's been a task group on reporting of source conclusions. When an examiner compares two items and reaches an opinion about whether they have the same source or different source. What's the best way to communicate those conclusions? Should you use the term match or identification, should you do likelihood ratios, should you do something else? And a lot of difficult issues come up there. It's difficult to resolve questions about how to communicate findings, because they involve a couple different issues. One is what kinds of conclusions are justified logically and empirically? And the other is, what kinds of conclusions will be understood by a lay audience? And we actually think the human factors perspective is sort of helpful on both time. We can comment on how well we think the quality of the empirical assessment of examiner performance. Can we really say that examiners are exactly as they claim? But I think mostly we a lot of say about how different kinds of statements will be understood and sometimes misunderstood by the target audience. And I have to say the interactions that our group has had through OSAC have been very stimulating for us. So a number of people who our human factors members and affiliates have been, it's been sort of inspired in the engagement research on some of these issues. So, several of us are connected to CSafe, which has NIST funding to do work on these issues. And so I think the Human Factors Committee has created sort of a nexus of interaction between forensic scientists on the academic community about these issues of communication that I think is helpful. So far we've been commenting on different proposals of that, trying to identify strengths and weaknesses of different reporting methods. And as I've said, we've tried to promote research interaction with some of these other groups to learn more about these questions. So those are our three main issues, kind of bias, performance testing, and reporting. A number of other issues keep coming up. These are issues that are sort of on our radar screen but we haven't had a lot of time to pursue, but I'm hoping that we can pursue in the next year or two. Things like better advice on personnel selection. How to assess stress, fatigue. How that affects performance. How lab directors and managers should deal with these issues. Ergonomics, we had one person for a while, we don't get a lot of call human ergonomics, but there's some issues about how to do it work. Vicarious trauma is of concern. The fact that forensic scientists are often exposed to really disturbing events, and how that affects people psychologically. And we've been trying, not very successfully, we've been trying to get some experts on that to work with us. And beyond that, you tell us. There may be other issues where you think psychology or human factors would be of interest to forensic science, could be helpful. And, as I said, we are here to help, and we are trying to do what we can to help forensic science become more rigorous and better grounded. So I'll stop there and I think next here come the judge. >> [APPLAUSE] >> All right good. >> All good. Good Job. >> All right, our next speaker, Judge Christopher Floyd, has the responsibility for chairing the Legal Resource Committee. And LRC also stands for left, right and center, so he has cast to hurry. But in his day job, he has just completed his term as the administrator judge in Imperial County, California, and has now returning to the bench as a senior magistrate. So at this point, I'd like to introduce Chris Floyd. >> All right, thank you. I wanna begin by contrasting my Legal Resource Committee with the Human Factors Committee. They wanna help, they make suggestions. The Legal Resource Committee, we think we are in charge. We wanna tell you what to do but we really know that that's not true. So, essentially we are very delighted to be involved in this scientific process because we are truly the outsiders. We are the outside consumers of forensic science, and we're at an intersection. I call this the intersection of science and the law, and that's a very difficult fit. It's truly taking a round peg and trying to put it in a square hole because our whole thinking and the way that we operate in the legal system is totally different than the scientific process. So this intersection has accidents, it has lots of traffic because you come into our venue and so forth. And the reason there's such a interesting dichotomy between these two disciplines is because the legal system operates on fundamentally different principles than science. So our legal resource committee, after a lot of grinding and pushing, we're sorta getting integrated and trying to help, make suggestions, let you know as we develop these standards and guidelines what things are doing. But we come from an adversarial system that has nothing to do necessarily with scientific processes and so forth, or scientific research. Each side, there are adversaries, they argue with each other, and they argue for argument's sake. And so, you have to understand when you come into the courtroom, you're not dealing with a Necessary level playing field. I'd ok at things a little bit differently now that I'm a judge. I was a defence attorney for 31 years and now I see this sort of happening before my eyes and as Chairman of the Legal Resource Committtee I have to heard the cats. I've got an award and I got a bottle of wine that said the angry bunch on it. And that's really what I have. So I'm gonna tell you a little bit about the Legal Resource Committee, who we are. What we're trying to do, what we hope to do in the future. So essentially, we deal with, when your friends of science either comes into the courtroom, or more importantly most cases get decided based upon reports, they get settled before trial. So we're very interested in report writing, the same things that bill mentioned and so forth and our group is very diverse and this is really our strength. We have judges. We have prosecutors. We have defence attorneys. We have innocence network attorneys. They bring a lot of issues to the table. And then we have academic and government representatives. And when I introduce you quickly to all these people so you can better understand them. Because between all of us, we have the legal perspective that we are able to give you. So what we try to do is we try to, we've invited ourselves into the SAC's and the subcommittees and we're getting fully integrated we have Liaisons that do that, we maintain an open dialogue through these liaisons. And then what we do ultimately is we evaluate standards, but we also help in the process now as the standards are being put together. We're having earlier input that's making a tremendous difference. We just had a very impactful move forward. Just this weekend when I attended the ASTM meetings we were asked to put together a standard on legal issues and training of forensic experts. It's a thing we're vitally interested in. So we're gonna be forward with that in full earnest, and try to give you a standard on legal issues that should be embedded within all forensic scientists to need to know about the law. So I'm the chairman. My people that are on here, academia, we've got Andrea Roth, she is a professor at Berkeley in California David Kaye, a statistician who is a professor at Penn State University. Some of the best people that we could get in academia, I think, are on our group. Andrea Roth is a professor of evidence, and David Kaye teaches a number of legal courses at the law school at Penn State. For the prosecution, we have a Richard Dick Reeve, he's a former assistant US attorney now in the Denver area as a prosecutor, a expert, one of the go-to experts in the United States on digital evidence. Elizabeth Geddis is US attorney from Brooklyn, recently just put on our legal resource committee when we lost one of our other members She is involved in very high level prosecutions in the New York area and so forth. She has got some experience in DNA and a very delightful person and very knowledgeable about the federal issues which are very important for our legal resource committee. Two other judges, my favorite group of course. Because now that I'm a judge I kinda side to them, that's my human factor spies. Ron Reinstein retired from Maricopa county, was a senior judge there for a number and Reveres now he's a judicial consultant to the Arizona supreme court, helps educate and train judges. Not only in Arizona but all through the United States. Kent Cattani he is a de public court judge from Arizona, recently named to the source committee. Very knowledgable, has a lot of background. He used to be in the attorney generals office in Arizona and in charge of capital case prosecutions through out the state. Two government and innocence projects. Lynn Garcia, the staff attorney for the Texas Forensic Science Commission. Barry Scheck involved in the New York Innocence Project. Lynn will never forgive me for putting her on the same side with Barry but nonetheless, that's the way things go. >> [LAUGH] >> Don't tell her. And then we have Jennifer Friedman. She does forensic work for the L.A county public defenders office and works with all of the hundreds and hundreds of lawyers for her office. Working with them to develop forensic Forensic issues and so forth for their litigation. John Ellis is a federal public defender in the San Diego area. Also, an expert in digital evidence. Him and Dick Reed, the prosecutor, the defense attorney, they both work together with the digital people and he's been very impactful for helping us out. In addition to having those members of our legal resource committee, we also have affiliates. I've put two of the latest that we just got Ted Hunt was on the Legal Resource Committee, he's now with the Department of Justice. We're not sure what he does, but we know he does something important. But he's asked to be affiliated and be a liaison with the Department of Justice. I'm delighted that Ted is wanting to do that, because we all do have to work together in that regard. Lindsey Herf, another recent affiliate, she did the Hair Project or worked on the Hair Project with the FBI. And she is now a staff attorney for the Arizona Justice project. So that's our group and essentially what our mission is, is that we want to advise and work with the people that are developing standards and guidelines to let you know what the legal ramifications of those standards are. And we also want to as best we can try to make this standards meaningful and useful to the legal system. And so that is our perspective. You know, when these land, what kind of a third do they have? Or explosion do they have, what benefit do they have. And we're looking to try to make sure that they are beneficial to the legal system. And that's a different issue then many of the forensic scientists. They look at it from a scientific view. We're looking for utility. Particularly, my bias is in favor of how do we get least our cases processed and we get the right result or the best result we can. Unlike science, where you can say, well, if we don't have the right answer we can do more research, we have to decide cases every day on the best information we have today. We can't wait til tomorrow. So we do the best we can with what we have. Standards are gonna help immeasurably. The legal resource committee has a Significant bias. We like standards versus guidelines because they simply have more impact in our system. So what we'd look at, we asked a number of questions about scientific validity. When we look at documents is there a problem. I'm gonna go through these very quickly. And we basically weed through our legal materials, our case law, and say are there any cases on the standard that's being looked at. Do we identify any admissibility issues or foundational issues? We very much have been impacted by the human factors committee, cuz that's really a legal issue as much as a human factors issue. So we look to see whether there are things that require documentation and preservation of how the scientific decision making has been made. We are particularly interested in the human factors issue, and that is documentation of things that scientists are exposed to before they make their critical decisions in the case, cuz that may affect the outcome. All right, so I'm gonna just breeze through these very quickly. These are all gonna be online, but these are the questions that we ask when we start looking at documents. We delve into, we're not scientists, but we all think we are. All of us have a deep experience in forensic science, and we wanna know what that validation is. Because that validation gets talked about in court, and that has to be translated. The scientific process that you go through and make decisions regarding has to be explained to juries, and then that's a fundamental concern of ours. We have to help translate this to the triers of fact. Judges, and, or juries. So we want to know about error rates and uncertainty issues, because it is a critical component of our legal system is that not only do we need to know what opinions or statements that you're making, we have to weigh them. And when you say it's 99% certain versus 50/50, or in all likelihood what Steve Johnson talked about the old days, those were unacceptable. They're not acceptable today. So what weight does the jury give a particular opinion or statement and so forth? But in particular, what I'm most concerned with, and what I emphasize with my group, is we're trying to figure out whether a standard or guideline is going to cause some type of a train wreck. Meaning, a case getting reversed, which is the cataclysmic bad thing that happens in the law. Or you spend months, even up to a year sometimes, in a major case and then it gets reversed because of a forensic science mistake. That has happened. It will continue to happen. But we're trying to minimize that, and we're looking at the documents that you produce to help you better understand how to avoid that from our standpoint. So we know about the use of the standards in our legal system. We're very committed about disclosure obligations. We're going to be publishing a paper about disclosure. We're trying to get an OSAC publication and we're gonna be focusing on that. Getting your information, your data, and getting it to the legal people is very critical, and we want to focus on more electronic means of doing that. And then, finally, what is really the most important thing is to get the information, the standards, the guidelines, and what the OSAC is doing, get that work out to the legal community. And we do that, and we do that in a big way because all of us are out there talking about it. I give lectures to appellate judges, attorneys, groups of that nature. Just recently, David Kay gave a talk to CSCAP regarding issues that he is knowledgeable about, air raids, things of that nature, is that intersection of the law. And we call that the community outreach, that's where we actually get the benefit of our. So with that, thank you. I don't think we have any time for questions cuz we're probably running a little short on time. But I wanna thank everybody for listening to one of the members and the leader of the angry bunch. >> [LAUGH] >> [APPLAUSE] >> Thank you, Chris. The next speaker I have the honor of introducing, Karen Resik, is a program manager at NIST Standards Coordination Office, as a day job. But we also managed to keep her going 22 hours a day with all of the duties that she handles so well for us in OSAC. She is the Chair, Quality Infrastructure Resource Committee, and it is also a very demanding task. And she does an excellent job, and so I'd like to welcome Karen to give her presentation. I also have to say that she helps us so much in OSAC affairs, that she is an informal member of the OSAC Affairs Team. >> Good morning. The QIC has lots of roles. One of them is that we provide input on draft documents in terms of quality assurance and quality control factors. As you heard earlier, we are managing the consolidation of the OSAC lexicon. But most importantly, we draft and manage all of the OSAC processes. Now, I'm not gonna show you a single process map today. >> [LAUGH] >> I bet some of you are disappointed. >> [LAUGH] >> But I am going to show you a map. Welcome to the land of OSAC. OSAC brought over 500 people together to improve the state of forensic science standards. My colleagues in OSAC are hard-working, passionate people. And they all came to the table hoping to make a difference, including the statisticians. >> [LAUGH] >> At the start, we told them they had to follow two rules when they drafted and adopted consensus standards. One, they had to be based on good science. And two, they had to reach some consensus. They didn't have to be unanimous, but there had to be some general agreement. Sounds easy, doesn't it? So as you can see on the map, we have our two major processes. Working with the standards developing organization, and registry approval. Many of the OSAC standards, I went black. [LAUGH] Start up here in apple tree land. So the biological subcommittees and the talk subcommittee started over here. They did pretty well. They got their, they had some consensus hurdles to get over, but they got through there and they went on. I would have to say there's still a lot of OSAC subcommittees sitting here in Apple Tree Land. They're having trouble reaching consensus. Our colleague, Austin Hicklin, undertook the admirable charge of trying to draft a conclusion standard. Much needed, very important document in this community. And as he put his cross-disciplinary sub-committee together, he ran into some problems. There was a lot of apple flinging. There was a complete lack of consensus. There was significant conflict. There were many opposing views. This document was going nowhere fast. Apple flinging. So what did we do? So in the case of this standard, we slowed down, right? We stopped. We started to listen to the concerns, listen to the sometimes minority views. We stopped to gain a balanced perspective. We continued to discuss everybody's concerns, and I think two years later, this document is gaining some traction. And I think you are going to hear more about it today. I need a drink, sorry. >> [INAUDIBLE] >> Water. >> Water. >> [LAUGH] >> Okay. So what happens? So we found out the most from that exercise that reaching consensus requires continual dialogue and takes considerable time. So once a document is ready, it's drafted by our folks. It goes over to the standards development land. It's time to cross that bridge. Some of my colleagues like Craig Butler and John Lantini spent years working in NFPA. Lots of our [INAUDIBLE] and drug and fire folks, Barbara Andre and drug folks like Sandra spent years working in ASTM and international. But for most people this was a pretty unknown territory, really, really scary. Our docking examiner colleagues had the absolute worst experience in ASTM International. They had to sunset that subcommittee. And they made sure they told everybody all about it. People were nervous. But what happens when you get to the Standards Development Organization? Poppies. >> [LAUGH] >> Now I don't know if you remember in the movie. They see the poppies and they see Oz in the distance. And they get excited, and they start running. We are going to write standards. This is great. Everybody goes to sleep. >> [LAUGH] >> Standards development is slow, due process is slow. I remember when I started in the OSAC, a subcommittee chair said to me, we're gonna have ten documents done by the end of this year. I said, really? I said, in the standards world I came from, if you get one document done in three years, that's astounding. Okay, so I think we had a little problem with expectations. The friction ridge subcommittee took their documents to the newly formed academy standards board, and through some disorganization of their consensus body, they didn't get started for a whole year. Same with bio, lots of their documents went to the consensus body but they had to kinda get around there, they had the ballot, they had to resolve comments. Happened with toxicology as well. Had a great document went out for draft and it went out for ballot and it went out for public comment. And they got 150 comments, but they were good comments. And then they went back and they revised the standard. And they re-balloted it, and they sent it back and it got. It takes a lot of time to do this stuff. Likewise with ASTM we see a lot of slowness. Sorry. We had issues with standards where we felt we couldn't even move forward. We had an SEM paint standard we just realized there was not enough research, we couldn't do it. Poppies. So what did we do? We learned the process. We learned how the SDO worked. We learned how to make our commenting more productive. We started to do a good practice of learning technical context in advance if we were gonna vote negative so that we could have time to discuss it. I think most importantly we started to manage our expectations of how long this is actually going to take. Subcommittees started to hold pitch meetings within OSAC so they can explain the standard and address people's comments and concerns. But again most importantly, we learned that reaching census takes continual dialogue, and considerable time. Over here now. So some documents actually come out over here. They're already done. They're past that point of being ready and published. So now we have to cross that registry approval bridge. ASDME3085 standard guide for FTIR spectroscopy in forensic tape examination was put up for registry approval and it went out for an open comment. Should this standard go on the registry, yes or no? This is when the flying monkeys were released. Now, flying monkeys come in many forms. >> [LAUGH] >> So the tray standards >> I swear there hasn't been a single tray standard that's gone out for registry approval that has not received a comment from the legal resource committee, or the statistics task group, or some NIST scientist. And they feel frustrated because they feel they have addressed these comments before. >> Many of the documents that come up for registry approval come, we get comments from other people. So there are a lot of stakeholders who are interested in this process. The document examiner community put up a training standard for registry approval. They sent it out for open comment, and they received 75 comments. That comment fell into the open comment pit never to come out again. They actually to withdraw that standard from the process and now they're reworking it. And in case of another standard the glass standard, ASTME 2926 standard test methods forensic comparison of glass using micro XRS practromotry. The standard subcommittee felt that they had addressed the comments. It went out for open comment period, the document was voted in favor onto the registry by the FSSB. But the commenters in this case were the legal resource committee, and they had a chance to appeal. So before that appeal panel met, we realized that the commenters and the subcommittee had never actually spoken to each other. So we said, you know, would you be willing to have a conversation about this before we sit down to have an appeal? They said sure okay, so we had the conversation. The first comment, we misunderstood, we misunderstood. They withdrew the comment. We do that with five comments, four of them went away. That standard has since made it to the registry. Flying monkeys. So what do we do? We revise the comment adjudication procedure. We realize that these discussions were not really happening. And we made it so they had to talk to the commenter before they could write down that formal adjudication. People just want to be heard. People want their comments to be understood. We asked that the comments become more focused, that they focus in on why it should be on the registry and why it shouldn't be on the registry. And we've started to explore ways to make sure that we got input earlier in the process. And that we learned that consensus requires continual dialogue and takes considerable time. So, let's talk about another standard, ASTME2329 identification of seized drugs. No groan? This was poised to be the first standard ever on the registry. I have to get another drink. Ever, first standard, it went out for registry approval, it received no comments, whatsoever. It got voted up on to the registry by the board. There were no appeals, because there were no comments. Guess who shows up now, the wicked witch. And that witch comes out and puts a statement up there, and says this standard has no technical merit. The wicked witch. So what do we do? We spent two years Two years revising how the OSAC assess technical merit. This was not an easy task. We found out that the difficult conversations were not happening. We found that minority views were being drowned out. This new worksheet requires everybody to have a conversation and to discuss the merits and document that discussion. And document any opposing views so that when the board gets it, they know exactly what everybody felt. As you heard earlier, we added the RC chairs to the board to bring a balanced perspective. We added resource committee representatives to the OSAC to bring a more balanced perspective. And we learned that reaching consensus requires continual dialogue and takes an enormous amount of time. If you haven't read my colleague Bill Thompson's article in the latest issue of the OSAC newsletter, please do. I encourage you to. It's called Why Good Standards Take Time. His message is the same as mine. The road to registry approval is long and slow. It's fraught with road blocks. Drafting and adopting consensus standards based on sound science is not an easy task. Consensus is hard, consensus is slow. OSAC has spent the last couple of years really working together. We've been learning how to talk to each other. We've been learning how to listen to each other. We've been overcoming roadblocks, apple flinging, poppies, flying monkeys and that wicked witch. The OSAC members and affiliate I know are 100% dedicated to improving forensic science standards. They are engaging in these crucial conversations, and they're working on reaching concensus. And I'm confident just like you heard from Steve this morning, there are gonna be more standards on that registry in this coming year and more standards coming out of the standards developing organization. We're gonna continue to tweak the processes, much to everybody's dismay, but we need to as problems are identified and as these roadblocks come up. And I guarantee you we're gonna encounter more roadblocks on this yellow brick road to standards development. I hear the Wicked Witch has a cousin. I wanna thank my committee. This is a group of very, very productive, bright people who really spent a lot of time drafting and managing the OSAC processes. I thank you all for everything you do and continue to do for us. Thank you. >> [APPLAUSE] >> All right, by the way, thank you very much, Karen. The article that you referred to by Bill Thompson in the newsletter is actually available on hardcopy on the desk outside. So be sure to pick that up. And get a signed autograph from Bill Thompson on the article. It's a super honor for me to be able to introduce the next presenter. Professor Karen Kafadar is actually the Chair of our OSAC Statistics Task Group. And she also herds cats. And I wanted to take this time to tell you that in addition to being Chair of the Statistics Department at the University of Virginia, she's also the president elect of the American Statistical Association. So I'd like to give her the control for the slides and to welcome her. >> [APPLAUSE] >> Telling you a little bit more about, thank you, better? Okay, let's see here, okay. Telling you a little bit more about what statistics is and so why it's relevant here for forensic science. I think there are an awful lot of people who think that the only way they've heard statistics is through the media. Our football statisticians has blah blah blah. And it's amazing the number of people, even when I worked on Hewlett Packard, that looked at me and said so what are you doing here at Hewlett Packard? But I think there's a couple of reasons why statisticians haven't been involved more in forensic science. One is the perception of statisticians and the other is why statisticians themselves have not become very involved. And so, I'll have a few more words to say about that later. But, the real mission here for including statisticians in this organization is to provide advice to the different OSAC units on the proper use and explanation of statistical concepts that are used in standards that are posted on the OSAC registry. So all of the members of the statistics task group serve as members on OSAC units in some way. And the reason they do that is because they want to be able to understand the specific discipline, and how statistics will relate to that discipline. But we also have the task group in order to coordinate the advice it's given to the different task groups, the different units. We don't always succeed that way, but we really do try to ensure that there's consistency. Also want to be able to collaborate with forensic practitioners to develop proper validated and practical statistical procedures. And I emphasize practical because all statisticians, even if they have been in universities, have worked on real problems. And we know that there are time constraints, and there are practical cost constraints, and just other limitations. We try to also provide other statistical advice as requested, and service advisers really, not intruders. We're really not here to be obstructionist, but to help. The members, being on the FSSB I chaired the statistics task group and we have four of the five SAC's have statisticians there. As I said, I'm just gonna list their names here because you can go to the website and then look them up. But they're all very productive, very successful in their own disciplines as statisticians as well as having worked closely with their specific areas. We tried to get statisticians on all of the units. Only 15 of the 25 subcommittees have statisticians, and so I've listed them there. I think part of the reason for that is that statisticians are, at the risk of some modesty, we are in demand. And so if someone comes to you and says, I really need your help. And another person is not coming to you and asking for your help, you tend not to go looking for more work than you have already. Sometimes we do get forensic scientists who do want to interact with statisticians. And so it's been a slow process getting statisticians interested enough in some of the problems involved in forensic science that they actually want to serve. So we're still trying to enlarge that pool. We do have a large American Statistical Association, it's about 18 or 19,000 people, so we certainly hope that we can get a few more who are willing to participate. So let me just mention, we also have three affiliates. David Kaye is on the Legal Resource Committee, Bill Thompson from who you just heard, and also Alicia Carriquiry, who used to be on the OSAC as well. So what is the whole raison d'etre of statistics? Probably number one is to help define the problem. So my thesis advisor is known for having said often finding the question is more difficult than finding the answer, and that really has been true. And most importantly draw valid inferences from data. So I often say data without inference are useless. So people are able to collect massive amounts of data, some of which are even relevant. But some may not be terribly relevant to what it is you're trying to do. And so we're trying to take what data are available and to draw inferences from them. Large part of what we do is to try to understand how the process can be measured, identify sources of variability. What influences are affecting the data? Is it the different methods, materials, personnel? And how do you quantify the relative contributions of the sources and variations? Also wanna be able to assign quantitative measures to what might be qualitative outcomes. Sometimes they really are qualitative and there's nothing you can do about it. But you'll think like colors, red, green, blue, yeah, you can make something quantitative out of that by looking at a spectrum. Work with forensic scientists to understand, characterize, and quantify these sources of uncertainty. And most importantly, help to design studies that will identify what sources of variability, what data will help us to identify sources of variability and to quantify their effects. So I know a lot of people feel that unless you can go into the courtroom and say I'm a 100% certain, it's not useful. I think that's the reason that we have the Legal Resource Committee. We do want to somehow communicate that unless you're talking about something like what's the probability of drawing number 55 out of the lottery, I don't think 55 is possible, is it? I don't play the lottery, does anybody know? >> Okay, nobody here plays the lottery then. >> [LAUGH] >> I don't think 55 [LAUGH] is a valid number in the lottery. That one has a probability of zero, okay. But unless you're talking about something like that. Somebody says, you mean you're not 100 percent confident? The answer is nobody's 100 percent. The recording angel knows the answer, but I've never seen a recording angel. So the best you can do is state just how confident your data seem to suggest. And take another dataset, it might be different, but you want to feel that 95% of the time you would have come to the same conclusion. That's the kind of, what we hope, is the kind of statement that ultimately will be translated effectively to laypeople. We also wanna be able to participate. We have commented this, statisticians have commented on proposed standards, and there are just a couple of them. We try to participate in other task groups. Karen Reczek just talked about the Conclusions Task Group. There's another task group now led by Jose Almirall on Interpretations for Testimony. Confer on best practical guidance. And then I'll just also mention that many of our STG members participate widely in forensic statistics activities in other areas, the Center for Excellence, that's in the. There was a statistical program, Statistics And Applied Mathematics Skills, Statistics Institute in North Carolina, that operated a year-long program on research in statistics and forensic science. We serve as journal referees and so forth. So what's the best way to get our attention since we're kind of distracted by many other things? The most important thing, I think, is try to engage the unit statistician. And the unit statisticians, we're not hard to engage, just tell us about the problem. We're like psychologists, you know? Tell us about your problem and what kind of data might be available. And if you have a query to a statistician it's sometimes helpful to copy the statistician above that. If you're on a subcommittee, copy the statistician on the committee and on the SAC. If it's a question for the SAC statistician, copy the FSSB statistician. Just so somebody above there is paying attention, saying did you get back to this person? Please don't hesitate to send us reminders. We're all journal referees, and so we're used to getting reminders from journals. And then the other thing that I think is really helpful, Richard Vander Bruegge did this for the first time, and I found that this is really helpful. He would send me a STAIR and say, Karen, we're really interested in section 4.2, could you look at that one? So I'll scan the whole thing, but at least I know to dedicate my attention to section 4.2. So that might help to get a slightly faster response. And anybody on the SATG will be happy to take any questions. And I think, I realize I'm standing between you and the break. So you're probably all quite anxious to get moving on. >> [APPLAUSE] >> We'd like to ask you to come back in 10 minutes instead of 15 minutes so we are back on time. So it is now 20 minutes after 9, if you come back at 9:30 we'll be right back on time. Thank you so much. Thanks, Karen. >> [INAUDIBLE] >> Okay. Thanks everyone for taking a short ten minute break there. There he is. Come over here. >> All right. >> I was gonna do my best Stolorow impression, but since he's here. All right, thank you very much. We are going to get started now on the next section for this morning. And our next speaker is actually Vice Chair of the Biology/DNA Scientific Area Committee. Our previous chair, George Harrin, decided that he was going to retire and step down. And consequently, we printed in all of the information, George giving this presentation this morning but George is retired, and not here. So very kindly, Kris Cano, who is the Laboratory Director of the Scottsdale Arizona Police Department Crime Laboratory, and also the Vice Chair of the Biology DNA Scientific Area Committee will be giving the presentation. And so at this point, I'd like to welcome Kris Cano. >> [APPLAUSE] >> Can everybody hear me? >> [LAUGH] >> All right yeah, I heard there's no video, just audio, so. >> [LAUGH] >> Excuse me. >> Yeah. >> Get you fixed up here. >> Okay, am I good? >> Yeah. >> Okay, all right. Well thank you Mark and thank you guys for having me today. This is gonna be a little bit of a lengthy presentation. I do have 75 slides, cuz data people, we have to do it up, right? So, please be patient, we do have a lot of information to cover today. As Mark had said, this is the SAC leadership. Doctor Herrin did decide to step down in the 1st of February, and I just wanna recognize him for his efforts of the SAC leadership. He's well know in the forensic community, and was very instrumental in paving the path of the SAC, kind of herding the cats together and getting to started. And he really wanted to stay on to see our first standard get on the registry, so it is our charge now to make sure that that does happen for him. But he will be truly missed, he was a pleasure to work for, and we look forward to Carrying on his mission in his absence. Here's a list of our SAC members. We have academia, statisticians, as well as forensic practitioners that make up the SAC body. And then as Karen Reczek had alluded, we have resource committee liaisons that we call ex-officio members that do sit in our SAC meetings. They do participate heavily and we have that contribution to our group. A little bit about our discipline, I'm sure that you guys have seen this a lot. Our mission is to review and develop standard guidelines relating to DNA and serological techniques used in forensic analysis, human and wildlife. So we have a goal here, that we want standards and guidelines that facilitate the analysis of biological evidence using scientifically rigorous methods that are consistent between and within laboratory. Sound simple enough, right? But that consistency piece has been a challenge not only in our SAC but in our subcommittees. If you take a DNA sample from one lab and you look at another DNA sample from another lab, a lot of times the answer is going to be slightly different. So that consistency has been a real focus for us and will continue in the future. Our strategic plan. We do have the FBI quality assurance standards which are governed by SWGDAM. And so, we wanted to focus on documents outside of that. So that kind of is the framework for our regulation. And we wanted to tackle things outside of that and concentrate on that. Early on, we decided that our standards or guidelines must be based on scientific principles and foundational documents that enhance consistency and compatibility of personnel and laboratory practice. So as I go through the presentation, you will see we have focused a lot of our efforts in the past couple of years on training and validation standards. Those are the fundamental documents that we need to do good science. And then we have chosen ASB as our SDO. And I'll go into how we are making progress with our standards. So basically, for the biology sect, there's five types of standards that we're focusing on. Methods of validation, training and education, methods, meaning how you actually do the analysis, data interpretation, and reporting. And this is a very scaled-down process now of how we get our documents through the process. And I just want to highlight that most of our documents are in the SDO development stage. We did not have documentary standards when we came into this, and so we have been spending a lot of our time creating documents and working with the SDO to get them published. So we often get the question of, where can we find the information? We are placing our draft standards on the NIST website and I've provided the links here for you and we are placing the document on the website once the SDO has voted on them, and the reason for that is that we've run into a couple of snags in the process where the actual scope of the document has changed. So in order for us to provide you accurate standards, we're doing it at that stage. And currently we have 13 standards on our website in draft form from all of the three subcommittees. So as a SAC, what are our next steps? We're gonna continue to review the documents developed by a subcommittee. We're gonna work with ASB, and we're hopefully gonna put some things on the registry. Any questions so far on the stack? All right, as you know, we have three biology subcommittees, one is the Biological Methods Subcommittee, and I like to think of our DNA process as a five step process. So the first three steps extractions, quant, and amplification. As well as serological methods are done by the biology methods subcommittee. Anything that comes off the CER instrument and has to do with data analysis, interpretation of that data, statistical analysis and reporting is done by the biological data committee. And that's for our human aspect. We have a third committee that deals with wildlife forensics. And I just wanted to show you how those three groups are working together for our training standards. We realized early on that we tried to take a big huge training standard for forensic biology and DNA, and it just didn't work. So we had to break them down into things that made sense. Also, when things change in our field, technology changes, requirements from the FBI QAS change. We have to have smaller documents that we can revise that make sense. So it would makes sense to revise a huge training standard if we only needed to revise one component of that. So the three groups work together so you have training standards that will cross over. And when I get to the Wildlife Subcommittee I'll explain why that one is in the red for you. Okay, so now Wildlife, I put them first. I always feel like they're the last ones to go so I'm gonna give them a little bit more stage time here. The Chair is Kim Frazier from Wyoming Game and Fish. Vice Chair Chris O'Brien and Executive Secretary Mary Curtis. It has 12 subcommittee members from various organizations. Federal, university, as well as private industry. And also 13 affiliates that assist them in this standard development process. So Wildlife, they focus on guidelines related to taxonomic identification, geographical origins, non human and biological evidence and morphology and genetic analysis. This is summary of the projects and as I give the talk today I'm really gonna focus on the standards that are in the STO process. Those that went to the ASP or already add our STO had being previously presented. So I'm gonna delve into the ones that we're working on at the sub committee level. So this group has actually five of those at that point and six that have gone to ASP. The first document that they're working on is called geographic assignment. And this method is gonna address DNA techniques that will address the software statistical analysis. And what this is, it determines what population an animal can be from. So the example is given if there is a deer, it may be important to determine whether it's a farm or wild deer. One of the problems that they are addressing when they are working on this standard is getting enough STRs or snips. To be able to genetically differentiate populations and getting enough database samples to determine a source population. And this really comes into play when they talk about endangered species, which could be very difficult to obtain that information. So that's one of the challenges that they're running into when they're working on this standard. The next one is the public sequence validation. Excuse me, and this is more of an awareness document. I found this one was pretty interesting, is that there's sequences that are found in public databases that they actually use. But they're not always vetted or reliable. And this standard will let the analysts know when it's safe to rely on public databases. Where, when it's necessary to run a home grown database. Problems that are rising within their subcommittee is that they're closely related species as I talk to you about in standard one. And they're having difficult deciding if they should make two standards. One for each database, or if there's you create a standard that uses both of this database that are available to them. And then standard three. This is species determination standard. And this is actually using antibody antigen reactions. Mainly they're looking at the deer family at this time. But they would like to expand that to other species. And then reference collection standard, this is the standard for collecting reference collections. And this is difficult because sometimes, the critters aren't willing to give their sample swab. All right, okay, I'm trying make it light here. All right, so they are trying to establish acceptable methods for collection, storage. And right now, they're starting with the morphology reference sample. And trying to collaborate with museums for the other samples, which I though was very interesting. So that's kind of the struggle and the challenge we are facing on this document as well. And then the mitochondrial taxonomy identification standard. This was the one that was in red in the previous slide. And the reason that they couldn't use a human training standard. So the wildlife group, all the training standards that the human group has developed. The wildlife group would consider adopting. Because it's kinda general training standard. But for mitochondrial analysis in the human DNA world, we use mitochondrial analysis for haplotyping. In the wildlife world, they use it for species determination. So they had to develop a separate standard for that. Again, here are the six documents that are at the ASB. The wildlife committee has its own CB, so the CB at ASB is divided and all six of these documents are in the ASB. So what are some future tasks of the wildlife subcommittee? We're gonna be writing protocols and training and validation documents for emerging technologies. So NGS, all the new technologies that are coming on a human forefront. We're gonna look at those applications for how they could be used in wildlife. They're also gonna look at how effectively incorporating emerging technologies. Protocols on specific methods. Validation standards for general and morphology. And then sometechnical reports. They also have some research and development needs, specific species, STR panels and allelic ladders. So currently, there are several labs working on compiling the markers for deer species. And then they have a grad student prepared to take on a desk of developing those allelic ladders. So they're using outside sources in order to get the actual research done. And then applications, as I mentioned, for next gen sequencing. Determine if it's gonna be beneficial for the wildlife community. Any questions on wildlife before I switch gears? They're actually a really neat group, they do a lot of interesting things. The next group is the biological methods subcommittee. It's chaired by Kim Murga who is laboratory director for Las Vegas. Her vice-chair is Margaret Sanger and her executive secretary is Jason Befus. There are 17 members of this group. Representatives from forensics practitioners from small to large organizations, federal, state, as well as the representative from this, some private companies. So it's a well-balanced group of really hardworking 17 individuals. Some affiliates that work with this group are the five listed here. And again, as I alluded to in the beginning of the presentation. This encompasses everything from serology to loading it on the genetic analyzer. And the focus is establishing standards that support molecular and biochemical methods used to analyze evidence, and reference samples. So Kim's subcommittee is divided into four different task groups. The contamination task group, the training and education task group, the validation task group, and then the srology task group. And again, here is an overview of the training standards, from this particular group. And validation standards from this particular group, which is expansive. There's been a lot of work done on the training standards for this group. And it's really broken down to each specific area of DNA testing as you go through the process. So far, this group has submitted five standards to the consensus body for consideration. We actually have voted on and accepted three of them or four of them, excuse me. And one, we actually did not accept at the ASB level. The reason for that, is the best practice document dealt specifically with requirements in the FBI quality assurance sandards. And we felt that was not our space, that was more redem role. So we decided not to move forward with that document in its current state. And here's a summary of ten other documents that the group is working on. Again, that are in the development stages and have not made it to the SDO but are pretty far along in process and they're listed here. Most of them are validation and training standards. Like I said, that was the low hanging fruit that we just started to work on first. Okay, so we have standards for analytical procedures and report writing for serology methods. So, as technology and as the field evolves, laboratories are doing less and less conventional serology in their labs. However, when they are doing it, there isn't a lot of consistency on how that analysis is reported and what tests are being employed. So this standard will provide guidance on how to report when body fluid testing is done. And in this area there's been a lack of consensus within the subcommittee. And hopefully, this document will provide standardization. So if you do, do body fluid testing, we all report it the same way, or the same manner. So we felt like this was a need for the community. Even though, it's not widely used as much anymore, it's still out there, and it needs to be addressed. So the key components of this standard is gonna contain terms and definition. Contamination prevention, and then analytical procedures, how to actually do the testing. So here's kind of where the standard is. We hope to give a pitch meeting of this hopefully in March, and then use that feedback to revise it and kind of go through the process, maybe by summer, July, have it submitted to ASB. So the next one is a guideline for internal validation of human STR profiling on CE platforms. This is actually supplements a standard that's already at ASB. So there's a standard for the actual guideline. What are the requirements for validation? That standard, like I said, is already at ASB. This standard will actually compliment that. So the standard that's at ASB says this is how you should validate something. This standard is more of a meat and potatoes on how you do it. And one of the areas that has been very controversial for this group is actually providing what number, or how many samples you should do. So that's something that they're tossing back and forth. But we understand that there's a need in the community to provide more stringent guidance on how this should be done. So the key components of this, we're going to have reference to that companion document that I mentioned. It's also gonna have some objectives and considerations, some experimental methods, and what to do with the data analysis and results. And it's gonna focus on the internal validation study objectives that are in that parent document, which deal with all of those that are listed in our FBI QAS. So the action plan for this is to present it to the biological subcommittees, which was done this month. And then also to have a pitch meeting in March at our in-person meeting. And then hopefully submit it to ASB in April. So this standard is dealing with contamination, standards for prevention, monitoring, and mitigation of DNA contamination. Contamination is actually our dirty. I'm tying this back to my title, right? It's actually 13 letters, and we don't like to talk about it, although it's there. We also have another dirty word, which is mixtures. But we'll get into that a little bit later in the last subcommittee. But basically, contamination is a word that we don't like to hear in DNA. It happens, but when it happens, many labs can take different approaches to it. So it could be, it´s contamination Tuesday, and they do absolutely nothing. Or it could be the other extreme where they see a slight bit of contamination and they shut down their operation for a month to figure out where it possibly came fro. And so, this standard will hopefully provide guidance in how to consistently feedback, like how to handle that on a daily basis. We have had LRC feedback on this one, and they are in favor of a log. And what history of contamination has happened in the laboratory. So again, this is another area that this group is gonna have to work through to prevent some concrete guidelines on how to manage that. Again, this just goes through the key components of that. We are going to include specific requirements for Rapid DNA. That's a whole different beast. But we are aware of that, and how to manage consumables in the laboratory. Again, the action plan for this is to present it at March at the subcommittee level, and then hopefully to the SAC in April. And then, by summer, again, getting it out to ASB. So standards four through eight that I'm going to go through here, again, training standards. And they all kind of go together. This one is the actual parent standard for the rest of the standards that follow it. And this is an overarching document that is at ASB. It's actually out for public comment right now. It's not? >> [INAUDIBLE] >> I'm sorry. I'm getting ahead of the DNA one. But this one, actually, I guess, is for serology, but it's going to mimic the one that we currently have out at ASB for actual DNA analysis. So this one is going to be kind of in the same realm. It's gonna have terms and definitions, personnel requirements, CT, training and program, and conformance. Again, we're gonna have this at the pitch meeting in March at the in-person OSAC meeting. We're gonna incorporate any comments from that. We're going to have it out for SAC vote in late summer, and hopefully out for the ASB in August. Again, this is a standard training for forensic DNA isolation and purification methods. Again, we're breaking these down for each part of the process. So the previous one was for serology. This deals with DNA extraction or DNA isolation. Again, same key components as standard four. And again, same time line as standard four. This is a training standard for Quant. So now, we've identified the body fluid. We've done the extraction. And now, we're gonna train people on how to do the quant step. So again, this is a very systemic process, looks very similar to the two before. And again, same timeline as those as well. So again, here is a training standard for forensic STR typing methods. PCR amplification and DNA separation and detection. Again, same concepts, it's gonna take those samples that are now quantitated. How do you train people for the amplification and CE set up? Again, same type of components in the standard, and again, same timeline to outcome. And standards eight through ten, we tried, or the group tried, to make the training standards all inclusive. But mitochondrial DNA analysis had to be separate, and so there's going to be three training standards for mito. They're eight through ten, and they deal with amplification sequencing, CE interpretation of haplotypes. So this is an overarching one for mito. It's going to have terms and definition, knowledge-based training, recommended references, conformance requirements and technical components. And again, here's a roadmap for the validation standards that I've been working on in this group. So just like the training standards, the validation standards, as you can see on the top box, will have a parent document. And then it will have guidance, which is the meat and potatoes of how they actually do the analysis, so that the standards for the intervalidation for all of those different methods is going to be the requirement. And then the guidance document is going to be a how to. Okay, research topics to improve analysis of serological evidence. Considerable research as been conducted to improve DNA. That's kind of been where we focused all of our research and upcoming technologies. But very little as changed on the front end of serology. So it'd be beneficial to add methods which decrease serology analysis times on items such as sheets and clothing. So those of you that worked in a DNA lab, we spent a lot of our time actually finding the stain before we can actually get to the DNA analysis. And so we're looking for some efficiency enhancements there. Also, to help us develop and assist in localizing the stains would be a great assistance. So if you've got a large scene with lots of blood stains, it would be easy to have a standard, or some kind of advanced technology to isolate those stains. A lot of times, too, we get the question of how old is the stain, how did the stain get there, those kinds of things. And so, those would be stuff that would be incorporated into this. Any questions on methods? There's a lot going on in methods. All right, the last subcommittee that we have is the interpretation group. Let me just get to my slide. It's chaired by Dr. Robyn Ragsdale, Vice Chairs Dr. Mechthild Prinz, and Executive Secretary Dr. Catherine Grgicak. She has 17 members, as well, which also range from forensic practitioner from state, local, federal, as well as private industry. As well as academia and statistics. There's also 22 affiliates. This group has a had a lot of action, a lot of interest. This group has been working on a couple of standards that have been real hot topics. And so we are getting a lot of affiliates which is great. Again this group focuses on getting that data off of the instrument and actually interpreting it. And writing reports and doing statistical analysis of them. So Robin created her slides a little bit differently. There's actually eight task groups that are within this subcommittee. The first is a training task group. And they've developed six documents. And these are standards that give guidance required for DNA data interpretation and reporting. There's a statistics task group and who's working on one document. And this document actually is supposed to be a general statistics document but moving forward there needs to be some conversation with the SWIGDAM group and the SWIGDAM guidelines on statistical. They're also working on some general statistical guidelines so they'll be working together in the development of this document. They actually started this one a long time ago, but it kind of took a back seat or a back burner to some other pressing items that they were working on. But it's still in the works. And then there's a Mixture Task Group. And these two documents will guide labs on how to validate their internal mixture guidelines as well as how to ensure their laboratory is consistently following their guidelines. One of those standards is at the final stages at ASB. So we'll be coming back to the subcommittee to be considered for registry approval. And then, the Reporting with Contamination/Failed Controls Task Group, this one's pretty interesting. Because it was really kind of the brain child of ATF, who works with minimal samples, or one shot analysis. So, if they run a DNA sample with one hair or contact that cannot be recreated, and there's something that is contaminated within that batch of samples or in the controls. What do they do with that data, because it could have prohibited information in there. So they have developed a task group to work on a document where you cannot repeat the analysis and how that is done. So just like the contamination scenario, that's handled differently in every laboratory. So, they're kind of working on a standard to standardize that. The last four task groups are the reporting conclusion. They're working on one document, and this addresses how to clearly, concisely and properly report DNA results. What Robin had suggested I say here is that everyone just wants to know is it the dude or not? And DNA analysis, we never say it is the dude or not, right? We're gonna apply statistics and weight that averages for inclusions. So anyway this task group has a big document here because I think that that's a huge inconsistency throughout our field. The formulation of proposition task group, they're working on one document, and this is going to explain the proper formulation of likelihood ratios. So, I'm gonna use a likelihood ratio for your statistical analysis. What is that look like and how should it be properly set up? We also have a software validation group that basically wants to provide a document that will give laboratories checks and balances on their software, to make sure that it's still working. And then the last task group is the threshold determination task group. And they're working on one document to explain the methods used to determine analytical and stochastic thresholds. So analytical and stochastic thresholds come into play when doing mixture analysis mainly. And laboratories right now have the ability to establish those on their own. So this document will provide guidance on how to properly do that and hopefully provide consistency within laboratories. So here's where this group is with their documents. The second one down is actually the one that hopefully will be coming out of ASB first. We are very close to finalizing that document. The validation standards for probabilistic Genotyping Systems has taken a little bit longer. ASB received 139 comments on that document. It will probably go out. I'm not gonna say it will go out for a second round of balloting and so that one's taking a little bit longer to get through. Those first two standards, this subcommittee felt that those were the need of the community. If anyone has been listening to the news or keeping up with any kind of what's going on in DNA. We're getting criticized a lot for our mixture interpretation. So the subcommittee felt that these two standards should be the ones that were prioritized and get out of the gate first. And so we're doing our best to get those out there. The other document that there is standard for DNA interpretation comparison protocols. And there are seven new work products that will voted on at our meeting today. Having to do with the training. Six of them are training and then one that we've already voted on. What we don't actually have the document for is the signing of likelihood ratio oppositions. So that's kind of where we are with these documents. She has 11 in progress. And I think I've gone through some of these today. Again, there's six training standards. And then there you can see we start actually getting into the nuts and bolts of this group, which is reporting conclusion, statistical analysis, validation of software and stochastic thresholds. So some future areas that this group would like to address are standards relating to sequencing or next gen sequencing. So that's on the forefront on our community, we wanna make sure that were ahead of that. Standards related to rapid DNA, since now that is something that can be more readily used. Also, standards related to Familial DNA Testing, there's a lot of states that offer Familial DNA Testing. But all the states up to my knowledge, we all do it a little bit differently. So maybe some standardization here on when a state is going to, embark on familial testing. There's some kind of standardization of how that should be done. Standards related to Y STR and mixture interpretation. Right now Y STR is the haploid analysis again. Doesn't really offer itself to consistent mixture interpretation or mixture interpretation at all. So we're gonna be looking at that as well. And then standards related to forensic paternity and relationship testing. Those laboratories that have gone online with genotyping, there's not a good way for us to do kinship analysis, so we'll be looking at that, as well. Robin also asked me to plug, if there's anyone of interest, to contact her or the vice chair Mecki Prinz to be one of 23 and 24 on that affiliate list. Okay, so going back to some identified research topics. For this group, best practice for reporting likelihood ratios or other probabilistics in court. And also accessing DNA background and transfer scenarios in forensic case work. So again, a lot of times when we go to court now, we're not challenged on the actual DNA. We're challenged on the statistics. What does it mean and how did it get there? So this is an area of focus that this group would like to concentrate their efforts in. And then, some challenges and lessons learned. Moving the SDO and OSAC process, there's been learning curve for us. And acceptance that good documentary standards take time to produce. We are learning as we go, and the length of time to actually get the standards produced has been something that we've accepted now. Understanding the role of OSAC, the SDO, SWGDAM and cooperatively working together for a common goal. So I think DNA is a little bit different with having the SWGDAM still in place, and making sure that we're all communicating and talking together, and going towards a common goal. The most effective manner in creating these standards for the biology discipline, was to work with smaller topics, rather than creating a monster standard. So when we first started this journey, everyone wanted to do a huge standard on training, a huge standard on reporting, and a huge standard on analysis. And we quickly realized that the way to be most effective is to break those down into smaller chunks. And there's more standards that are coming, which means more work getting through the SDO process and through registry approval. But it's probably the more appropriate way to go. And as an a discipline we need to prioritize with the needs of the community. I think we were working on a lot of training, and a lot of validation standards. But the mixture issue, and contamination issue, and interpretation issue with stats is probably something that we would need to prioritize better in the future. So again, here's some points, this is kind of from an overall perspective, that familial searching. There really hasn't been a comparative analysis of the Probabilistic Genotyping software programs that are currently out there. So this was one of the things that George wanted to focus on was actually comparing, for example, two allele and STR mix. Assessing how rapid DNA is gonna work, are people going to start employing that in their laboratories and how is that going to affect CODIS? And then again, the consistency in mixture interpretation, making this a real priority for our group. And that is really all I have. It's probably a lot quicker than I thought. Any questions for me? No questions? We actually have 39 standards and process. We have 11 from our wildlife, 14 from the biological methods, and 14 from the DNA interpretation, so we're cruising. So out of those 200 that you spoke about, we've got 40 of them, so couldn't be prouder. Thank you. >> All right. >> [APPLAUSE] >> Thank you, Chris. So we actually have 35 minutes before the next speaker. And because we have them timed, and they are people who are coming in and out for each of the various scientific area committee disciplines. We are tempted but reluctant to start the next presentation, because we also have people who are on webcast. So if the time that we're supposed to start the next presentation is in fact 10:45, we will begin again at 10:45. For any of you who want to play bingo, we will be able to entertain you in the next 35 minutes, but I need to get the wheels. So for those of you who don't enjoy your break, and we'll see you back here at 10:45. Thanks. All right, well, the old clock on the computer says, 1:45 Eastern Time. So it looks like we are ready to begin again. The next introduction is one that I'm particularly happy to share with you. Dr. Gregory G Davis, is university professor at University of Alabama in Birmingham in the medical school. Pathology, has a number of other titles, but he also has the responsibility for being the chief medical examiner in Jefferson County, Alabama. Where he puts to practice the things that he learns and teaches everybody at university. He also has a responsibility for herding cats at OSAC as the Chair of the Scientific Area Committee, which is called Crime Scene and Death Investigation. And there are lots of different stakeholders melded together in that particular scientific area committee, as you'll hear in his presentation. So it goes everywhere from crime scene to courtroom. Following a path through the medical examiner's office in many cases. And he's done a remarkable job. And I appreciate the fact that he gives me an example of what true patience is, long after my patience has evaporated. And Greg is still there pondering thoughtfully hand on chin, and I think at that point I believe it. And he can speak to you about what goes on in the Crime Scene/Death Investigation SAC Thanks. >> Thank you, Mark. Those are pretty kind comments. I wish you'd gone on for about three more minutes, and you would've probably said most everything that I was going to say. >> [LAUGH] >> But I will, and I hadn't planned on saying this at all, but if you want my secret to being patient, cuz I think I am a reasonably patient person. It's that earlier in life, I figured out that impatience really wasn't accomplishing much good for me, so I thought I'd try it the other way. So if that's helpful to you, then I'm glad of it. I am the chair of the crime scene, oops, already had to go in for me, of the Crime Scene/Death Investigation SAC. And it is a bit of an amalgamation as I will mention, but on this title slide, I want to point out something that is important, much of a second. And George Pollard, talked about how this has implications for court and the importance of court. And I agree completely, and I am very proud of the role I am able to play in the justice system. But I wanna point out that death in and of itself, maybe it's a shame, but it's not necessarily a crime. And so, we go and investigate these scenes, and we don't know whether there's a crime that's been committed necessarily or not. And that's part of what it is that we're about. I"m trying to figure out, if things are to go to court, then we would want to do the work that would enable them to go to court and be adjudicated appropriately. If it's not a crime, as occasionally happens, then we would like to be able to determine that it's not a crime and keep from using Lead resources that really aren't even needed, because we can use our medical knowledge to figure out what happened and to explain the death satisfactorily and tie up all the loose ends and show that it's just a natural death, albeit perhaps a particularly unusual or in some cases bloody one. So I want to talk a little bit about the difference between the SAC and the subcommittees. I'll start with the SAC, the crime scene death investigation SAC. Is looking at standards for crime scene/death investigation. But we're really there to approve the source of work and foster the source of work that the sub-committees perform. In a way we're a go between the [INAUDIBLE] FSSB, who sit there at the top of the OSAC structure and the subcommittees who are performing the work that we're really here to talk about today. So the SAC kind of facilitates that work with both of those extremes to make sure that we get some high quality standards and develop new standards as appropriate. A currency death investigation SAC has 16 members. When we were trying to figure out how we would make these presentations this year, compared to the previous years, the though was that we would have more of an overview. If you have questions, I will be happy to answer them. But if it's part of the overview, then we will just have the SAC chair present, rather than having each of the subcommittee chair present. And I wonder, that I was all in favor on that, thinking that that made a lot of sense, somehow it escaped my notice [LAUGH] that I was, in fact, the one that would be doing all the presenting. >> [LAUGH] >> [LAUGH] But here I'am and I'm so be gracious about it. Our committee has 16 members including people with expertise in crime scene investigation, both fire and dog specialist. We have death investigators in various aspects of death investigation. And we have an epidemiologist, and I actually spoke with Karen Kafadar during the break and told her, but there was a change in a position. Jerry McGwynn that she mentioned as being on the medical legal death investigation sub committee, he's actually a member of the SAC. And so all five SACs within OSAC do have statisticians. Jerry's both a statistician and an epidemiologist, which brings a particular application of statistical analysis that's helpful in health matters. So I wanted to correct that idea that Karen had put out earlier. The sub committees, there's seven of them, ours has the most sub committees of any division within OSAC. So we have in my mind, there's two divisions here. There's, those who examine scenes and gather materials that then go on, often to be examined and processed by other individuals on down the stream. And there's those who are they may start the investigation. But they also will be the end examiners of it. So let me cut concrete. Crime scene investigation, dogs and sensors, fire and explosion investigations. All of those entities are to some extent finding materials, collecting the materials. It may be that analysis will stop at that point, but it may well be that other individuals will take those materials that have been collected because of the work of the crime scene investigators, dogs and sensors, personnel and fire and explosion investigators, and perform their own investigations later. Downstream we have anthropology, odontology, of course their concerned with examining bones and then teeth. And then you have the medical legal death investigation where they're examining the body. There's some considerable interaction between anthropologists and odontologists and medical legal death investigation because the medical legal death investigation pulls in the odontologist, anthropologist as needed typically. And then straddling that fence is disaster victim identification where they have some work in scene investigation but also work in trying to identify individuals by odontology, could be anthropology, radiology, examinations, or a fingerprint, whatever DNA, whatever would be the analytical technique used to try to establish identification. So I'm gonna go through now and talk about each of these subcommittees, the work that they're doing, and challenges. The sort of thing that you heard earlier from Chris but perhaps with fewer slides. >> [INAUDIBLE] >> And so it is with great pleasure and considerable- >> [LAUGH] >> The only slot you use. >> [LAUGH] And with considerable pleasure that I surprise Karen Reczek that I put up the process map. >> [LAUGH] >> [LAUGH] >> And I've done it or the flow chart having with it I've done it. Because actually I want to make a very similar point to the point that Karen made and that is, and now I'm actually gonna walk away from the mic, so you might have to live mic. And that is, this is a really complicated process. The process starts up here, and all of these boxes represent where people are talking to each other. And trying to come agreement and consensus. Sometimes they're more likely to be of a like bind at the outset. Sometimes they're very unlikely to have a similar approach and point of view. At the outset of the discussion. So you gotta go from one end all the way through the process down to the other end as you're developing a standard. So I want to show that cuz I think that makes it easy to see why this is a long process. Just making the phone calls would be a long process, even if you didn't talk. >> [LAUGH] >> All right? So it takes a long time. Then you get here, and guess what? You get to send the thing of to an SDO who has their own process map with public comment and all of that. Then the SDO sends back a standard that they've created. And now, and Karen showed us this in her, more colorful now. You start all over again with that standard with what should be a simple question. Is this standard, now that it's been all the way through the process one time. Is it acceptable for inclusion on the OSAC registry. Very involved process. I think it has some worth to it but I'm coming back over. >> Thank you, sir. I think it had some worth, but it takes a very, very long time. So I wanna make sure I mention that, and that's why Steve mentioned the very first talk. There are eight standards on the OSAC registry. More are coming. But I don't want you to wonder why it's taking awhile for all of this to happen. Now, let's talk about anthropology. Thomas Holland is the chair of the Anthropology Committee, and they focus on standards and guidelines that are related to anthropological methods and theory relating to the recovery and analysis of human remains, most of it concerns bones. You can determine amazing things and amazing amount of insight into what caused a person to die or identify the person just from the bones, if that's all that remains. There are 18 members of the anthropology subcommittee. Three of those members are new. I have decided not to list all the names. If you go to the OSAC website and find the lists of the committees and subcommittees, there'll be a page devoted exclusively to anthropology. And you can see all their names and a photograph of them. You've seen some of those earlier. But these anthropology committee is first alphabetically, if not in my heart, and they are also going to be the first that's going to show you how much of what's happening in the SAC is transpiring. The anthropology subcommittee has one document concerning stature estimation from bone measurements that's now with a standards development organization, the Academy Standards Board. The anthropology subcommittee is waiting to see how this process, which is fairly new to us. Most of us are unfamiliar with working within this process. As Karen is painfully aware because she has given us lecture and lecture and lecture over again to try to teach us and get it to stick in our heads. And it's working, so we're gonna see how this stature estimation of standard that they have put forward cut returns and experienced that process of going through the SDO before they put more forward. They have three more documents that are ready. They have gone from the subcommittee to the resource committees. Two of those three have now actually have now come back to the subcommittee for the subcommittee to review the resource committee comments and respond to those. Then there's another document on age estimation that they're ready to send. In addition, they've got 17, the anthropology group existed as a scientific working group before. So they have 17 documents in addition to the ones I've just mentioned that they developed as the scientific working group anthropology, that they have had reviewed and approved by the SAC for hosting on the anthropology web page. So they haven't been approved as standards and added to the OSAC registry. But they are posted as a means, as a reference for what is considered the proper way to perform an analysis, or what's proper training to do the job that's necessary. And in time, you would expect that those documents that are posted on the website will be worked through the process. But they're available now to those who would want to be able to see something that would be useful now, rather than waiting for three years for them to get approved. They also are working on an additional four documents, they are not based on any work that they did at SWIG. And then finally, and this is true for all the subcommittees actually in all of OSAC. They contributed their part to the glossary, I called it, or lexicon, I heard it called today, which we've heard will be posted next week. Anthropology's challenges and lessons learned, really mostly concerned working through the process as they're trying to get accustomed to this new process that they hadn't been so familiar with before. The research needs concern controlled trauma studies in collaboration with engineers so that they can understand fracture patterns in bone. That's easy to say, that's really hard to do, it's really hard to do. It's hard to find some substance that will perfectly mimic human bone, and then apply pressure to it or forces to it and see at what point it begins to crack, or break, or splinter or be crashed. I mean, I have seen all those things happen in real humans who unfortunately died as result to the injuries, but obviously we can't use those bones. We have to find something else that we can we work with and perhaps computer modeling will the proper way. But all those things have to be evaluated and determined if there are appropriate models for human bone in human beings. In any case, the anthropology section's highest priority is to work with the documents that are in the SDO process and get those on through the process. Crime scene investigation is a particularly special subcommittee to me because that's the only subcommittee that now exists that didn't exist when OSAC was created. Crime scene investigation specifically grew out of the very first meeting of this sort which we held, I guess that was four years ago now. And which the audience said, well, you've got this, you've got that, but you don't have anything about crime scene investigation itself. So, we agreed and we created the subcommittee, Maryland Miller is the chair of that subcommittee. And they're focusing on developing standards and guidelines related to investigating crime scenes, in search for documentation of and presentation of evidence associated with a crime scene. And as well as, as maintaining that evidence and in an appropriate way. They have 17 members, three of those are new. If I haven't mentioned, if it's not apparent, OSAC is a lot of work, it has been mentioned it's volunteer. So some people, I think everyone who is volunteered initially to work with OSAC has been eager and willing to do it. And some though have found that the demands were greater than their bosses say were willing to allow, [LAUGH] and so they've had to step back. But there have been others who have been willing to take their place so that's good. The crime scene investigation subcommittee, I promised Marilyn that I would mention this, they're still smarting a little bit because they got a late start. It was about 18 months into the OSAC that they were created, so that means they've only had 55% of the time that everybody else has had, something like that. So they still have a sense of being behind, but I keep reassuring them that as time passes that fraction would be smaller and smaller and soon it won't really count. In any case, the glass really is the work product that they have. They have a several research needs some of these were actually mentioned earlier today. But research for databases, information clearinghouses, documentation technology, what's the best way to document? Is it photographs, is it newer things like trying to create a hologram, there's people at work with that. Is our 3D reconstructions that you would record, so you would put on the goggles, and you can be in the scene, That sort of thing. Human factors in crime scene investigation is actually very important to them. Bill Thompson mentioned earlier how his group was starting to consider things such as fatigue and safety. And it was crime scene investigation that was particularly interested in having that material brought into the human factors discussion. Because it is a hard job, and it can be long hours, and they would like some guidance in what would be appropriate. I mean, I'm too, I know all about girding up your loins and sticking it out, but at a certain point, you can't gird anymore. >> [LAUGH] >> [LAUGH] You just need to let them rest for awhile. I don't know why I say some of the things I do. >> [LAUGH] >> [LAUGH] It's fun though. So their highest priority is to have time to catch up with the others and work toward it. And to that end, they're hoping that they'll get to have an extra meeting some time to catch them up. I better leave that alone and just move on. Disaster victim identification is leading our charge with a number of documents that have gone to a standards development association. Jason Wiersema is the chair of this subcommittee. They focus on standards and guidelines related to fatality management of mass fatality incidents. And particularly the scientific identification, as the name would imply, of human remains. There are 15 members. They've just added one additional member. His particular expertise is in data management. They're very excited to have him join the group. They have six documents with the Academy Standards Board. Those documents are all posted on their web page as drafts, the same sort of thing I was referring to earlier, where you can see them in draft form. They're not on the registry, but you can see what it is that's being worked on, and they're available as resource for those who need them. And they concern things like anthropology, DNA analysis, fingerprint analysis, pathology, all the sorts of disciplines that they use to try to identify individuals. They're working on another four documents for submission. They have their own contributions to the glossary. Their research needs really concern trying, mass disasters are always unexpected, at least I'd hope they would be. If it were expected, it wouldn't be a disaster. I mean, we've averted some disasters, right, by encouraging people to wear seat belts and such. So fatality rates on our highways have decreased as a result. But that was known and we could figure out how to respond to it. Unexpected events, like a tsunami or an earthquake or even a tornado. I know tornadoes are gonna come through my state here in a month or two, but I don't know exactly where they're gonna hit. Those are more difficult to predict and therefore prepare for. So research that would help with at least studying past incidents, and try to predict what sorts of resources might be needed, would be useful. Also, the Disaster Victim Identification Subcommittee points out that there isn't really a means to quantify individualizing characteristics, necessarily. That sort of work is being done in some areas more so than in others, but at what point is it enough? Fingerprints can be very good and have been a standard of excellence for years. DNA is very good. But what if you don't have something to compare those to? Is a tattoo good enough? If it's an ordinary tattoo, and maybe not as good? If it's a really elaborate tattoo, is that better? We don't know. But I can tell you that we use things like that. We do the very best that we can in my own office. I'm a forensic pathologist. But sometimes, the very best that you can do, the strongest evidence that you have might be a tattoo. Or a description of some sort of scar or something like that, or even a wedding ring that's got an inscription on the inside. Is that enough? And that sort of research could be done too. Ken Furton is the Chair of the Dogs and Sensors Subcommittee. I'm very glad that we have some members of the Dogs and Sensors Subcommittee with us. Bill MacCrehan's here, Barbara, aren't you here somewhere? Yeah, yeah, Barbara Weakley-Jones. And I'm very happy that they're here, because as I said, the very first time we met, I actually do not own and have never owned a dog. You can hate me or not, it doesn't really matter, I don't have a dog. [LAUGH] So I have less experience and knowledge of this sort of thing than most people seem to have. There are 19 members on the Dogs and Sensors Subcommittee. Two of them are new and they have six or seven work products that they're working on. The general K9 guidelines are to establish consensus based best practice general guidelines for training, certification, and documentation pertaining to all K9 disciplines. Much of what's going on here concerns things that have been done at the local level. But of course, OSAC is talking about nationwide. And so term that might be used in one jurisdiction might be a little different from terms that are used in a different jurisdiction. So there's work trying to figure out how can we all agree. And of course, everybody wants to use their terms cuz they had them first, right? So trying to work all that out is part of that work that occurs in the process map. There's also a need for the terminology for the dogs and the detectors that they use. Not just the dogs, but the different ways in which they go about detecting. That can be called different things by different people. Tracking and trailing individuals, providing scents to dogs to try to train them, those sound easy enough when you see it done in a movie. But we're actually trying to figure out how to do it in real life, and it can be more problematic. I mean, where do you get a scent that you want to train a dog with? Again, you can't necessarily produce a scent. Or if you do produce a scent, at what point has the scent decreased so much that you need to refresh it? Those are all the sorts of questions that Dogs and Sensors either are working on or have as research needs. And here then is mention too, of the sensors, and the needs for odor training, and analysis of what is involved in odors. And how quickly they, is decay the right term? How quickly the odors decay. Next, I'll talk about fire and explosion investigation. If you haven't figured it out yet, I will not take all of my time, cuz I did want to give an overview. So you'll have a little bit more time for lunch, but there'll certainly be time for questions, too. I'll answer all your questions. As best I can, but then we'll have time for lunch. Craig Byler is the chair of the Fire and Explosion Investigation Committee. Fire and Explosion has an advantage over any of the previous subcommittees I mentioned in that they have for years, and Craig for years has worked with the NFP as a National Fire Protection Association, thank you. And they're a standards developing organization. And Craig has worked with developing standards with them. So they're very familiar with trying to get standards into the standards development organization and getting them passed. Their focus is on standards and guidelines for the investigation of fires and explosions and incidents. They work hand in hand with fellow subcommittee in the chemistry side that is concerned with the chemical analysis of these specimens they collect. But there are other analysis that can be performed that aren't purely chemical. And Craig, actually would support, this lesson is as important to him. And I think important to all of us. It's antithetical to their methodologies to presume a cause of the incident. In other words, they're not told, this is arson, make sure that you figure that out. They're just trying to figure out what happened. And that's true for me too when I approach autopsies. I just try and figure out what it is that caused this person to die? I'm gonna go off on a bit of an aside here for a moment, because I've brought that topic up. But it has to do, this is now touching on things like cognitive bias. And Bill Thompson mentioned this and how much information is appropriate and how much is too much. But I will say this. I don't think it helps if the examiner doesn't know what the question is that you need to be answered. You really need to understand the question that people are asking, or you may not even know to try to address it. So in one way it's very easy for me I know whenever I body comes to me for examination, we're trying to figure out what caused the death. But this is an anecdote I like to tell to people when they try to understand, what is it that makes your job important, or why does it matter? So I'd say most of the time, everything I find is exactly what you would have thought too if you'd gone and looked at it. I may know some technical terms that I can use and put in my report, but if a person's died in a car wreck, and their body is battered and broken from the car wreck, I suspect that you, without any special medical training, would be able to figure that out the same as I can. But to me, it's not that. The really important reason I have my job is for those few cases where it's not obvious but I figure it out because of my training and experience. When I started I thought, so I'm gonna find some homicides, people who've tried to hide it somehow, then I'll uncover it, and someone who tried to get away with murder, literally, we'll catch him. And I've done that about three of four times in my career, at 24 years. But it's very important. But what I didn't understand Is that an equal number of times, three or four times, I would be presented a case and people would say, for whatever reasons, we think this is a homicide. They haven't actually told me it is a homicide, make sure that's what you find, but they present all their evidence as clear. They think it's a homicide and I'm able to figure out that it's something else. A simple example would be, there's a lot of blood at a scene, and it's someone who I later find out is an alcoholic, but they've got liver cirrhosis. They've developed something called esophageal varices, which can bleed profusely if they rupture, and that does happen in alcoholics. And so this is the reason why there's blood all over the person and at the scene at their home because they've bled to death from these varices. And it may look like an awful mess, is an awful mess, but it is an indication that they've somehow been beaten is because of their natural disease finally ending in their death. So I'm actually very proud of that. And I think that that sort of approach would really serve not just the legal system, but the public best. Just try to figure out what happened and then present your reasons for why you believe it. To me, that's really what OSAC is working toward. So I'll come back to Craig and the fire investigations people. 19 members they have, two of those are new. They had a strategic plan for fire and explosion investigation. The next expected publication is later this year. They've put forward a proposal to the NFPA for a standard for fire explosion investigation in units. And that work is to begin later in this year, probably in the autumn sometime. And that's really, I'll skip down to bottom, Craig tells me the highest priority that they have is developing the standards for investigation units. They already have standards for individuals and for examination of scenes, that would pertain to certification of an individual. What they're trying to develop is standards for, they call it, an investigation unit. If you prefer to think of it as an office, what have you, a laboratory, but some entity that could be accredited. That's what they're working toward and are particularly giving their attention to. The NFPA has a five year cycle for its documents, so they just published their large document 921 on fire scene investigation and explosion investigation. But the time has already began that they were renewing consideration for the next update. But they told us of this and so people were able to submit comments and some people in OSAC who have nothing to do with fire and explosion investigation as part of their day jobs, submitted comments and asked for them to reconsidered certain things, the way they were worded. To try to give a more accurate portrait of what it is that they do and the weight that should be given to their work, by jury considering the evidence. And one other thing I skipped over, I'm sorry but I'll come back to it. The components that'll be in this standard for foreign explosion investigation units will include management system requirements, the personnel qualifications, accreditation as I mentioned. The accreditation of the unit. And standard operating guidelines for conducting investigations and standard operating guidelines for providing expert witness testimony. That actually seems like a very ambition project to me and I look forward to seeing how it develops cuz it could be a paradigm for other disciplines within OSEC as well Medicolegal death investigation. This is of course the area that I, myself, practice in. Keith Pinckard is the chair of this subcommittee. Several of the subcommittee members are in here now, as I see and recognize them. There are 15 members total on the subcommittee. They have one document, similar to anthropology, they've got one document with the American Standards Board. And they're gonna see how that goes. Before they submit more. That particular document is a call for competent medicolegal death investigation And refers practitioners to various guidelines that already exist. The NIJ's Publication Guide to Death Investigation, the National Association of Medical Examiner's Forensic Autopsy Performance Standards. The CDC has a publication that provides guidelines for senior investigation of sudden, unexplained deaths in infants. There's also a call for certification by the forensic pathologist who performed the examinations of bodies, including the autopsies, certification of death investigators by the American Board of Medical Legal Death Investigators. Accreditation of offices by name the National Association of Medical Examiners or by the International Association of Coroners and Medical Examiners. And then independence of our ability to investigate, so that we're not under the direction of some other agency saying be sure and find out what this is, and let me remind you that I'm your boss, that we see that as a very poor way to practice. Because if you haven't been paying attention, some deaths take on political overtones, and they can take them on with a vengeance. And suddenly, people who might not care otherwise care passionately about exactly how that death gets classified. And their instances of inappropriate pressure being brought to bear over the years. But we stand firm as death investigators that we need to have independence to try to and figure out what happened and to say so. And you can like it or not, but we'll have our reasons for why we say what we say. We'll be able to present them to you, and you can see for yourself why it is that we have drawn our conclusions. The Medicolegal Death Investigation subcommittee also just put forward and had approved two documents that concern the investigation of sudden deaths in which drug use is suspected. These are not standards that would be on the OSSEC registry. They are though posted on the Medicolegal DeathInvestigation web page, sub committee web page, as documents that are available. And to which individuals can refer say, jurisdictions or health departments, what have you, that would like more guidance. Or at least a reference to show why these standards of practice that we advocate, such as performing toxicology analysis, performing an autopsy in addition to the toxicology analysis, performing more detailed analysis than simply a screen. People can look to these. And I understood from last week that there were already individuals in Congress that were asking for and being referred to those documents on the website. So, what? >> [INAUDIBLE] >> Yeah, so that was good. Keith, I didn't see you. Are you over there? Okay, welcome, hope that all went well. You liked it? Okay, good, good, excellent. [LAUGH] Okay, the last subcommittee I haven't talked about yet is odontology. Bob Barsley is the Chair of that subcommittee. They're focused on standards and guidelines related, of course, to the application of dentistry to various matters of forensic interest, which would include identification, also bite mark analysis, that always gets attention. Age estimation, you can use dental work for that, and then litigation. And there are 15 members of the Odontology subcommittee. They have two documents, and the odontology group, likened to the foreign explosion investigation, the American Dental Association, is a standards-developing organization already. So they have experience working with standards development organizations. I have two documents that have recently gone through the process map here. Well, that was up there a moment ago. Once it's gone through the ADA for their standards development, and now they're ready to come back through the OSAC process map for inclusion on the OSAC registry. One of those is ADA 1058, which concerns uniform nomenclature and other ways to improve having a uniform forensic dental dataset. And also 881088, which provides a comprehensive overview for identification methods using dental records and dental evidence found on a body. Another document concerning estimating age from teeth is in the final drafts. They had their contribution to the glossary. By mark, task group, there's such a thing withIN OSAC, but OSAC is not the only group of dentists that's interested in bite marks. And the American Board of Forensic Odontology, members of that group are here at this meeting. They're meeting with odontologists who are part of the OSAC Odontology subcommittee. And they're working toward and have good hope, they tell me, by the end of this week, of achieving, I don't want to use the word standards, but a rubric that they all agreed to rather than trying for one group to make one rubric and another group, another rubric in how to evaluate these injuries. They wanna have one rubric for all, which makes perfect sense. And they have good hope I'm told of achieving that at this meeting. So perhaps we post these next week, maybe I could update that. That would be nice. That didn't take so long. Does anybody have any questions? Now, how about all the viewers listening at home? Do they have any questions? All right, someone does want to know where these things come into my head from. >> [LAUGH] >> It's my wife, isnt it? >> [LAUGH] >> Mrs Doctor Davis asks, no. So we do have a question. One is, with the ever changing landscape of crime scene investigation, does Dr Davis know if any other subcommittees will be added to this SAC in the next few years. >> The answer is I do not know of any other subcommittees that will be added to this SAC. I do know of interest, and there's a process within OSAC, you might expect there's a process, but I didn't show that map. But there is a process for OSAC to consider whether a discipline should be added to OSAC, so groups can make their recommendation. At the moment, at least, it seems to me that what we have has flexibility to allow addressing things that exist now. You might have to fold something into a group that already exists. >> We do have another question. >> All right. >> I know that medicolegal death investigator workloads can be enormous. I know that workload analysis is a research area of interest. Has there been any analysis done to date? >> I don't know of any. I know that the National Association of Medical Examiners has an upper limit to the number of autopsies that it says is appropriate for one pathologist if the office is going to be accredited, but I don't know. And by extension, you might suppose that that would apply to the death investigation. Death investigators in the office as well if there's a one to one match between them and the pathologist, but beyond that, I do not know. >> That's all that's online. [APPLAUSE] >> Thank you. >> [APPLAUSE] >> All right. We have at this point a break until the end of lunch, which will bring us back for the physics and pattern interpretation scientific area committee presentations by the current Chair, Melissa Gish and previous Chair, Dr. Austin Hickman. I did want to bring to your attention one factoid that when OSAC was first being formulated, and part of the conception occurred here in Seattle, the convention center when the two hour presentation was given to the plenary to a group of more than 500 people, more than 800 people, at the American Academy to hear the initial presentation of a framework for OSAC. And that was four years ago. I had brown hair then. >> [LAUGH] >> I had hair. I had hair then. >> [LAUGH] >> So if all of you return at 1.15 we will be very happy to see all of you. >> [INAUDIBLE]. >> Well- >> [LAUGH] >> Please, please, so thank you I actually was told in the back of the room that ontology late last night did pass a uniform series of guidelines and standards that they agreed to. So now the process for that sort of thing as showed on the board would begin earnest when we learn what they are and various groups talk about them and we go forward. Thank you, enjoy your lunch. >> [APPLAUSE] >> Right, we're done. Calling you out. All right, it's exactly 4:15 PM on the East Coast. So I'm gonna get started. It's okay to start drinking in 45 minutes. Those are the [INAUDIBLE] I'm sorry, Karen isn't, right. Karen isn't here, so. All right. All right, so the next session. Thank you everyone for coming back, and the next session that we have is with the scientific area committee in physics and pattern impression, pattern interpretation. And we have with is today our new SAC chair and our former SAC chair teaming up to give you this next presentation. So, I'd like to begin with our present SAC chair. She's a physical scientist. in latent fingerprints with the federal bureau investigation located in Quantico, Virginia. She drove a long way to be here today. Melissa Gische has a long career in Forensic science with pattern identification in latent fingerprints in particular. So she's going to give you a slide presentation on the status of activities that are going on in the Physics and Pattern interpretation SAC. Then she will turn it over to Dr. Austin Hicklin. Who is going to give you a further update on a interdisciplinary committee that he led during the time that he was actually the chair. Austin is still a member of, still the chair of the committee, so he will be presenting that information to you. And Dr. Hicklin is a senior fellow at Noblis, located in Western Virginia. So without further ado, I'd like to welcome Melissa and- >> [APPLAUSE] >> Thank you. Okay, so update on physics and pattern evidence scientific area committee. As- As was just stated I did take over for Austin, as Chair in October of 2017 for a one year term. But Austin didn't go far. He is still serving as Vice Chair of the Physic and Pattern SAC. Has remained as our Executive Secretary. As far as the subcommittees nothing really has changed here in the sense that we still have the five subcommittees within the physics and patterns SAC being blood stained pattern, firearms and tool marks, footwear and tire track, forensic document and examination as well as friction ridge. For a Bloodstain Pattern and Footwear specifically, those two Chairs remain the same with Toby Wolson and Matt Johnson. Remaining as Chairs of those two sub committees respectively. And then we did also welcome three new sub committee Chairs in this past October with Todd Weller taking over for Andy Smith in Firearms and tool marks. As well as Gerry LaPorte, taking over for Rega Vargus in forensic document examination and then Henry Swofford taking over the position I had previously held for friction ridge. As far as the rest of the physics and patterns SAC members, not too many changes there either. Dave Baldwin, still a member, as well as Ted Burkes, Lesley Hammer, Paul Kish, Hal Stern, David Stoney, and John Vanderkolk. And then we also have our ex-officio members from the various resource committees. This was a relatively new addition, maybe within the last couple of years that each of the resource committees has sent liaisons over to the various scientific area committees. And these folks have been instrumental in terms of helping us establish dialogue with the various resource committees, helping us engage in conversation about our various documents that we're putting forward. And so having Rick Lempert from human factors, having David Kaye and now Lynn Garcia from the legal resource committee, as well as Aaron Henry from the Quality Infrastructure Committee, have really been quite helpful in terms of moving our discussions along within our SAC. As you know, the role of the SAC is basically to provide oversight for the various subcommittees within our Scientific Area Committee. And so we do that by reviewing documents, by providing comments, by interacting with the resource committees and providing that feedback back to the subcommittees, as they're working on their draft documents. We also will put out recommendations if we need to help them prioritize the work that they are doing, as well as providing oversight. When we see areas that are being dealt with across the various disciplines, we can help ensure that all of the subcommittees are working on the same things. And if need to, we can try and draft something that's even overarching for all of those disciplines. What we found, though, so far, is that even though there are a number of areas that are cross cutting, whether it's training, proficiency testing, how we do with conclusions, that sort of thing, is that instead of trying to establish a scientific area committee wide document on those areas, or even something broader across OSAC, is that we've been allowing each of the subcommittees to draft their own documents in those areas. And then as they come up to the PSAC, we're able to adjust and ensure that they are consistent with each other. Austin will be talking here in a little bit about some of the difficulties in trying to obtain consensus across various disciplines when it comes to some of these generalized areas. And so we've been trying to allow each subcommittee to take their own approach while providing that oversight to ensure that we are consistent across the groups. So this is a slide that I stole from Henry on the friction ridge slide set that he provided, but I thought that it applied to all of our sub-committees. And I know that some of the other speakers, I know Karen talked about this earlier this morning, about the difficulties or the time that it takes to get consensus documents through this OSAC process. The slides represent here. The numbers represent number of individuals on each of the subcommittees or each of the task groups that the documents go through. And so if you start in that upper left-hand corner, you'll see that as a document starts, it begins with a subcommittee task group. It's usually about three to five individuals from that subcommittee, that's working on putting forward a draft. That draft then has to go through its own subcommittee. So it's another 20 individuals, where the document now has to achieve consensus amongst that group before it can move forward. And then once it gets through the subcommittee, it gets passed along to its SAC, so another 15 members, as well as the three different resource committees and potentially the statistics task group. So we're talking another 15 members in the SAC as well as another 62 or so members amongst the resource committees. And as you can imagine, when you put together all of the different scientists on the different SACs, all of the lawyers, part of the legal resource committee, all of the scientists and researchers as part of the human factors group, all of the quality folks as far as the QIC, and all of the statisticians that are part of the statistics task group, there are a number of different perspectives and viewpoints that are put forward. And to try and achieve a consensus among all of those groups, particularly on some of these highly contested issues, is quite difficult and takes time, before we can get to something where all stakeholders can agree to move forward on a document. And then once it gets all of those comments, it then gets kicked back to the sub committee to adjudicate those comments, where it then has to go through the same process again, in which case it then gets submitted back to those other groups for another round of review. So we can potentially have several iterations of this review, comment, editing, adjudication period before the document is finally approved as an OSAC approved document and then kicked over to the SDL, where the entire process starts again with a whole new set of people, okay. As far as the Physics and Pattern Scientific Area Committee is concerned, all five of our subcommittees are using the Academy Standards Board, the ASB as our standards development organization. Part of that group, they put out both standards, best practice recommendations as well as technical reports. So you'll see various types of documents that we may be talking about, as the afternoon moves forward. As far as OSAC, we still refer to a lot of documents as guidelines, but once those get to the ASB, they´ll then be converted to best practice recommendations. So just kinda keep those terms synonymous. All right, so let´s talk about what our subcommittees have been up to. So Bloodstain Pattern Analysis, their scope hasn't changed. They still deal and focus on the same issues that they've done since the beginning of OSAC. Their leadership really has remained consistent with Toby,as I said, still remaining on as Chair. Jeff Gurvis serving as Vice Chair, and Holly Latham has remained on Executive Secretary. As far as their specific members, I'm not gonna go through every member. If you guys wanna know more information about the makeups of these various committees, I highly recommend you go out to the website. Lots of good information out there, and as well as their past members. And one of the things that the Blood Stain Pattern Analysis Subcommittee really takes advantage of is their affiliate membership. They have such broad representation from the international community, that they've been able to take on over 50 affiliates to help with various areas, in terms of their subject matter expertise. And so they've done a really nice job of leveraging the ability to work with the affiliates amongst these communities. If you are interested in joining, go to the website. You'll see that any information you need, go to the OSAC website. BPA, blood stain pattern analysis subcommittee, has the honored distinction of being the first group to have an ASB published document. And so they had their >> [APPLAUSE] >> Yay. I'm sorry, we had to spice things up, right? Spice things up, woohoo, okay. >> [LAUGH] >> So BPA, right? They got their termonology document out. They got that over to SDO. It made it through the SDO process, and was the first ASB published document to make it to the website. And so you can go to the ASB website to go check out the BPA terminology document that's out there. They have a number of other documents at ASB, so another four documents that the SDO is currently working on. Report writing, dealing with validation, minimum education and training as well as quality assurance. And you'll see a common theme here. You'll see a number of our subcommittees working on the same types of documents whether it's training, proficiency testing, that sort of thing. All right, so they are actively working on these documents so keep your eyes on the ASB website for when these documents go out for public comment. That will be your opportunity to put in comments for these particular guidelines or best practice recommendations. They also have a number of documents in process. Now we're shifting back to OSAC, working on everything from dealing with their identifications and reconstructions. The language used for conclusion statements in those two areas, dealing with certification, standard operating procedures, proficiency testing, and so on. And then they also have their research needs. I'm gonna talk a little bit more about research needs once we get to the footwear group. But for all of our subcommittees, their research needs are posted on the website. These are great areas particularly for funding organizations who are looking to fund research. But again I'll get to that when we get to footwear. All right, so that's blood stained pattern, right? They've been busy. They've done a great job getting that document through the ASP process. They have a number of documents still pending there. And of course, they continue working to get things through the OSAC group as well. Right, so shifting now over to firearms and toolmarks. So they did have a shift in leadership. Back in October, Andy Smith stepped down. Actually he became vice chair. And Todd Weller became the chair of that group. Wendy Gibson, however, remained on as executive secretary. Just like with the other group, if you wanna know about all the other members, please go check out the website. Their affiliates as well. One of the things firearms wanted me to emphasize was their breakdown in terms of who their membership is. And as you can see, they're pretty well balanced. They have their right around that 70% target mark, for practitioners, as established by the OSAC charter bylaws terms of reference, one of those documents. But then they also have representation from the academic and the research community as well. Their discipline description hasn't changed. That scope has been the same since they created it. All right, and as far as their roadmap or their overview. Back when OSAC started three plus years ago, the Firearms and Toolmarks Subcommittee put out, they basically categorize six different topic areas in which they wanted to focus their work. And those same six topic areas or task groups are still working today. Everything, whether it's examination. Their novel technology task group has been very active. I'll get to that in a minute. On certainty of measurement, training, report writing as well as their criteria for identification. And so they still continue to use those same overarching six topic areas in order to focus their work today. They have in several documents at ASB as well, so these are documents now that have made it through the OSAC process. These are OSAC approved documents or approved to go to SDO, I should say. And as I mentioned before, their novel technology task group put out three documents, topography comparison software, implementation of 3D technologies. As well as 3D measurement systems and measurement quality. All three of those documents were kind of paired together and were approved, and are now over with ASB to go through that process. Firearms and Toolmarks went through and got their barrel and overall length measurement document through the process as well. They continue to work on a number of documents. Again, I'm not gonna go through and read all of these but they're certainly working on all of those things that are necessary to help continue to provide guidance within the firearms and toolmarks field. And again, if you wanna see what these documents are, most of our subcommittees are encouraged to list their active documents on their website. And so you should be able to go to those websites, to see what those various subcommittees are working on. And they also have research needs, so if you're interested in finding out more about what the firearms and toolmarks community believes are their greatest needs when it comes to research, please check out that site on the website. Okay, so it's firearms. Right, so let's talk a little bit now about forensic document examination. This is another group where we saw a change in leadership. Rigo Vargas stepped down to become a member of the subcommittee. And Gerry LaPorte was appointed then to chair this group. John Osborn is Vice Chair, and Thomas Riley is the Executive Secretary. It's a list of their members. They also have a full membership. Their scope as far as what they work on has not changed. Again I encourage you to go to the website. As far as documents, they do have several documents at ASB but I wanna be very clear that these documents did not go through the OSAC approval process. These were legacy ASTM documents that ASB is working on but again these did not go through the OSAC process. But still, again, be aware that these documents may come out for public comment sometime soon. As far as documents that the OSAC subcommittee is working on, they have a number of things. Everything from training, terminology, they're also working on a source conclusions document as well. And so they continue to remain active. Some of the things that they plan on working on in the future. So once they get those other documents done, approved through the OSAC process, kicked out over to ASB then they start working on some of these other things. Whether it is dealing with forensic writing ink, examination of paper and so on. Obviously within forensic document examination, it's a very diverse group. They work on a lot of various subdiscipline type things. And so they're going to be generating documents for all those subdiscipline categories as they continue to work on things. As far as OSAC research needs, yep, they have those too. Again, check them out on the website. Most of our subcommittees, obviously, being from the pattern comparison disciplines. We all need more research in terms of examiner accuracy. Right, we need to look at bias issues, and so you'll see that a number of these topic areas are gonna be the same across all of the pattern comparison disciplines. Okay, footwear and tire track. Right, so Matt Johnson is current chair, has been chair since OSAC started. Rodney Schenck is the vice chair. And Stephen Greene is executive secretary, as well as [INAUDIBLE] expert. So he's been very helpful not just for the footwear and tire track groups. Steve, if you're listening, again, thank you for all of your help over the past six months, and trying to get all of our technical issues sorted out. Membership, again, check out the website. They've got affiliates as well. Their scope has not changed. You guys have seen all this information before, I'm not gonna cover it all again. As far as documents at ASB, they were successful in getting their preparation of test impressions for footwear and tires through the OSAC approval process. So this was a document generated by the subcommittee Went to the SAC, as well as all of the RCs for reviewing comment. They adjudicated those comments, they sent it back to the SAC for approval. It was approved and then sent over to the SDO. They are also working on a number of other documents at various stages. So some of these documents have gone through that review and comment period, have received comments, they're now adjudicating those. Others are still being actively drafted by the subcommittee, so we expect to see a number of these documents coming through the SAC as well. These guys have been very active in terms of pushing documents forward, looking for those reviews and comments. Future documents, they too are going to start working on articulation of conclusions among all of the other things that they've got going on. As far as the research and development needs, these guys have done a great job in terms of drafting the research that is needed within footwear and tire. And in fact, so much so, that a number of their research needs have been picked up, and have been funded, and they have probably six or seven ongoing funded research studies going on right now, through MIJ. Because of the material that they were able to put together, and say hey, this is what our community needs, these are the research projects that we need funded. And then they went out and actually got those things funded, and so we're certainly encouraging all of our sub committees to put forward those research needs. Because we now are seeing actual tangible evidence that these things are being picked up and funded in areas that the pattern comparison disciplines so desperately need that research in. So this has really been a good success story in terms of what our footwear and tire track folks are able to put out and are now subsequently working on. And then finally, the Friction Ridge Subcommittee, what those guys have been doing, as I said, another change in leadership here, Henry Swaford took over In October, Tom Wortman is vice chair, and Maria took over for as Executive Secretary. As far as the membership, formatting's better, it's a little messed up on my screen, looks good on the screen here. So one thing that I've noticed about the membership with Friction Ridge is there's been almost at 50% turnover in the membership since OSAC began. So since the original set of Friction Ridge Subcommittee members, there are now 9 out of 20 are new members. And so one of the criticisms that we used to get as part of the scientific working groups, or the SWGs, was that we didn't have that turnover in membership, right? It was this old boys' club theoretically. And we're certainly not seeing that when it comes to these OSAC subcommittees, particularly the evidence is there with Friction Ridge, with this high turnover rate. In fact they have six new members just since October of 2017. Scope hasn't changed, still working on the same stuff. As far as what their priorities, what their road map is, what they are trying to focus on first is those documents that establish that underlying foundation, as far as testing and examinations process goes. And then as they start to produce those documents, get those documents moving through the process, they're then going to start moving up the pyramid, working on quality assurance documents as well as those dealing with personnel. So trying to focus on those three main topic areas that we see in the ISO Accreditation type of documents, but again, their focus is on testing and examination first, and then will be followed in by these other categories. They have two documents presently over at ASB, the Guideline for Articulation of Source Conclusion, particularly when it comes to identification decisions, and then more recently, they just got their Standard for Training. Was just submitted to ASB and is actively being worked on as well. One of the things that has changed since the Fall, since October, or so, of 2017, is that the FSSB has granted the subcommittees and the SACs, if they approve, then, any documents that have been approved to go to an SDO can be published on the subcommittee's website. The Friction Ridge Subcommittee has decided to take them up on this option and so the two documents that have made it through the OSAC process, that have been approved to go over to ASP. Friction Ridge Subcommittee has posted those documents, all of their content on their website. So if you are curious to see what is actually in those documents, I encourage you to go out to the website, and you can go ahead and download those documents. Yes, they have draft all over them, and yes, there's an OSAC disclaimer and all of that on there. But if you really want to see what they're working on, go ahead and check those out. Now keep in mind, that all of that can be changed by ASB, but at least you can see what OSAC group, the direction that they are heading. Documents in progress by the Friction Ridge Subcommittee. They're dealing source conclusions. In fact, I just got an email within the last couple of days. The source conclusions document has been submitted to the Physics Pattern Shack as well as the RCs for review and comment. So that document is moving along. The subcommittee continues to work on their process map, again, dealing with the examination process, testing methods, that sort of thing and then once they get a little more progress on those, they're gonna move into the quality assurance arena where they're gonna work on verification, technical review, and so on. They also have the research needs. And this is something that as documents come up through the physics and pattern sack, as the comments are coming in, we can identify areas where we need more research, where we don't have answers to questions. This is one of the roles that we have taken on as part of the sack is to then let the subcommittees know, hey, this is clearly an area where there's a gap in our knowledge. Go ahead and put out a research need to help cover that gap. And so that's one of the areas of oversight that we work with the subcommittees on. Some other things that the Friction Ridge subcommittee has posted to their website. They were able to get out several responses to some of the things out in the public domain, whether it is a response to the PCAST Report, or responses to the previously published and posted Department of Justice Uniform language, those sorts of things. So you can see how kinda the Frictional Subcommittee responded to some of those areas. They are published and available on the website. Finally again, go to the website, all right, I can't encourage you more. If you want to find out information about what all of our subcommittees are doing, please go check out the website. All right, so before I turn it over to Austin for the exciting portion of this talk, does anybody have any questions about any of the process, any of the documents, any of the status like that for any of our five subcommittees? I can tell you I'm probably most likely gonna refer them back to our subject matter experts, but yes? >> [INAUDIBLE] So I can tell you for Friction Ridge, absolutely. >> Could you repeat the question?- [CROSSTALK] >> Yes, absolutely. So the question was, are we getting the types of applicants as part of the application pool, that we want, that we are looking for that subject matter expertise to replace and become new members on the various subcommittees. And it varies across the subcommittees, some are getting A lot more applicants than others. Friction Ridge has well over 100 applicants that are in their applicant pool at all times and obviously one of the bigger disciplines. Some of our smaller disciplines are struggling in this area. Now I know it's an issue for blood-stained pattern. I know it's an issue for footwear. In terms of their overall pool of just practitioners in the field is smaller. Therefore, the people that are available to participate is smaller. And so is certainly is a struggle in some of these areas. We would also love to see more applicants from the greater scientific community. Particularly, those folks involved in research and academia. We do not have enough people in those areas that are applying, that are knowledgeable about forensic science that can come in and help immediately. And so, some areas we're doing great, and others, certainly yes, we would love to have more. Judge? >> Yeah, the blood stain document that was approved by the ABS, when is that, where is that in the process of going through the OSAC approval process now that it's been through the STO process. >> So I'm gonna see if Toby will give me some indication here. Toby do you guys intend to put that document through registry approval or not necessarily? So the subcommittee approves it and then goes to the OSAC, and then to the FSSP eventually that's the thought process anyway. >> That's what we're hoping to do for most documents. The terminology we sent straight to the ASP because that's always an area that is controversial [INAUDIBLE] pattern analysis. >> It would be good to get that posted up. If it's, you can end that controversy, that would be a good process, so- >> And we intend on it. It is available to the ASB, however, and we do have a link to it on our OSAC website. >> Okay, very good. >> So, it's available. >> Thank you. >> So I'll repeat the first part of Toby's response there for the webcast. Essentially, the question was, are you going to put that ASB-approved document through the OSAC registry approval process? Toby, Chair of BPA basically responded, that's not something they've discussed yet, but it's certainly something the subcommittee will consider. If they do decide to push that forward, they will have to vote on it. It will then go to the physics and pattern SAC for a vote. And then it'll go through the registry approval process for OSAC, if the subcommittee decides to take it to that point. Any other questions? Okay, so, what I'm gonna do now is turn it over to Austin. As he said before, he has. As previous Chair of this group, he is still the current Chair of Virtual Subcommittee Number #5, which is the interdisciplinary virtual subcommittee from OSAC dealing with source conclusions. So to just discuss some of the trials and tribulations of trying to get. A consensus approved document across multiple disciplines out to the community. So, Austin? >> Working without slides. Hello everybody. For those people who don't know me, I'm Austin Hicklin. So I, as she said I was the Chair of until I stepped down when my term came over and I'm the Vice Chair. But I've been working on a source conclusions document with the pattern sack. It's the background. In 2015, there was a talk at the FSSB. I think it was started off with Laurel and me. Discussing on trying to the value of standardizing terminology across all disciplines. And then when we had our next pattern-check meeting, then went to talk about what can we do, at least within the patter-check of standardizing terminology? And rapidly it became clear that the vast majority of terms that disciplines use don't really have any overlap and most of the ones that have overlap have conflicting definitions. However, there's some core set of terms that are used by different disciplines in different ways and in particular terms associated with conclusions. Identification, individualization, elimination, exclusion, verification. There are some core set of terms, and of the ones that are most complicated to deal with and least likely to be used in exactly the same way or having exactly the same implications from discipline to discipline. A lot of them had to do with conclusions. And so, in summer 2015, which we may notice is two and a half years plus, we started work within the pattern SAC on a draft standard for expressing source conclusions. And, now let me differentiate. Source conclusions, we mean a conclusion where an examiner is examining two or multiple items and try to decide did they come from the same source. For example, did two bullets come from the same firearm or two photographs that show the face of the same person. Or if you have one item and a source and say, this bullet and rifle, did this come from this? And so that's what we mean by source conclusions. There are other types of forensics conclusions that we are not addressing at the moment. Classification conclusions where you are looking at one thing. I'm looking at this shotgun slug. What gauge shotgun did it come from? This blood pattern stain, was it a transfer stain? And so a classification conclusion you are looking at one item of evidence. And trying to make a type determination about it. A different one is also constructive conclusions where you're assessing what happened or the events or actions that resulted. For example, the trajectory of a bullet or the order of events at a crime scene. And so what we're dealing with in this is purely source conclusions. You have multiple pieces of evidence. And you're trying to determine if they come from the same source. So over the course of the last two and half years, I was actually a little bit surprised how rapidly we came back with a somewhat general consensus within pattern. As it comes down to the specific wording, and as you bring in more and more and more disciplines, it gets harder and harder and harder to come back with general consensus. There's some things where it was actually quite easy to come back with consensus. Nobody these days argues for the term individualization or essentially no one. No one argues for the exclusion of all others or 100% certainty or 0 error rate or there's also general agreement on a number of other things that examiners should be stating their conclusions in terms of support for two proposition, for example. You want to say something like the observed characteristics provide extremely strong support for the proposition that came from the same source and extremely weak support the proposition came from different sources. One of the headaches in the past is that people basically look at one-half of that. And they say it had a strong, basically, there's lot of evidence that came from the same person without explicitly stating. And there is next to nothing supporting that they came from different sources. Let's see, of other things people generally agreed on. People, generally, agreed the accuracy and reliability of these conclusions and also the ability of forensic examiners to make these conclusions can and should be validated through empirical studies. And so a lot of the areas of consensus worked out well There are, however, a bunch of areas where it's really quite challenging to get consensus. I mean, one broad one is whether it is practical or even desirable to define conclusions uniformly across a broad variety of disciplines. Many people agree in theory, until it starts to look like they don't get their definition, and then all of a sudden start to back down of, maybe it's not as desirable as we thought it was, if somebody else's definition becomes the standard. That some people are also just as the fundamental, should examiners be reporting that they conclude two items come from the same source? And that should instead they be making more broadly qualitative and never express if it's their expert opinion that they came from the same source, that a number of people do not believe that they should be testifying to that. And every time we have one of these, then the rest of the argument is, are the alternatives any better? The word identification, by itself, I'd say, was at least a quarter of all the discussion. And the word identification is loaded, and there's a lot that a variety of people have written on it. But what's interesting is what everybody seems to have slightly different version of what the implications of what the word identification are. Is does the word identification imply certainty? It's that SWGFAST, gosh probably six, eight years ago, got rid of anything with a certainty to the exclusion of all others etc. But another group are arguing that it innately means certainty, and that it innately means exclusion of all others regardless of what the definition says. And there's also whether the alternatives are better, that every time we come up with, okay, what can we use instead? It's something that we got even less support for any other term to use in its place. Another one that's particularly challenging is, should the forensic examiners be expressing posterior probabilities. How many people off the top of their head are happy with posteriors mean? Okay, so well let's see, for example, I do a footwear comparison and I say there's very strong evidence that this impression was made by this shoe. But then I say, however, this kind of shoe is very unusual in the United States and therefore that the probability that anybody's wearing this shoe is very low. Other than the person who had this particular shoe and therefore I'm going to adjust my conclusion based on it. That was a quick and dirty, but that's a type, but basically I've applied the prior probability to it, and made it into a posterior. And so, much, much, much discussion about when application of priors and expressing posteriors is the role of the forensic examiner. And when that should be done separately, and that's only the role of the court. Let's see. Lots of discussion, also, on likelihood ratios. And a lot of that comes into, is it currently practical? And if you have likelihood ratios that are not based on solid, validated, quantified numbers, are the likelihood ratios implying an inappropriate level of provision and certainty if people are estimating numbers, but then you use the numbers as if they are validated. Is that itself providing a worse solution then with a qualitative results? And so these are the kinds of things where it's been really challenging to get consensus. And so a lot of this is we've had two and a half years working on this and the document is really somewhat in limbo. That we're in the process, Bill Thompson has a draft of reformulating it as more of a discussion as what the various issues are with the hope that it's easier to get people to agree on the benefits and drawbacks of each solution rather than trying to push forward for a standard. But the document has actually been quite beneficial in one aspect. But multiple disciplines have based their discipline specific source conclusions document on the earlier drafts of this and have been moving forward. Fingerprint, I think base recognition and others. So, but some of this sort of comes to a broader question that's sort of an OSAC wide question is standardization, what does it mean to standardize when it is incredibly difficult to get consensus? One example, the document I saw fly through about as fast as anything was the firearm's one on barrel length. And it's something, it's the kind of document that nobody's gonna disagree on because when it comes to it, there is a, okay, you start measuring here. You measure in this way and you're done. It's great to have documents like that go through, but it's not the kind of documents that are the bread and butter of why OSAC was made. The source conclusions, I deliberately picked what I thought was core and probably the most challenging document I could come up with. And it still looks challenging two and a half years later. So some of this is, if you have a variety of groups and achieving consensus across multiple disciplines. When you have very entrenched different perspectives is some place between a challenge and an intractable. And so what does it mean to get consensus? At some point does it mean that somebody has to be dictatorial and make the demands? That gets quite to be a challenge. I think Melissa has a similar kind of thing, that there was an OSAC, when she was in forensic, when she was in the friction ridge committee had a document on articulating conclusions and it went to ASB and do you wanna say something on where that stands? >> Turn me on. Okay, thank you. Still at ASB. So the document that Austin refers to, the guideline for the articulation of the identification, decision in fingerprint comparisons, has been both the document that I am both most proud of and well as most frustrated by. In my entire career as part of SWGFAST and OSAC. It's something that was a SWGFAST draft for at least five years, then was picked up by OSAC, was worked on by the subcommittee, went through the SAC and ORC's got all the comments back. We adjudicated those comments, put it forward, and now it's been stuck with the ASP. Friction ridge consensus body for well over a year now. And so we take so much time and effort in getting it through this OSAC process, and then like I mentioned earlier, everything starts again once it gets to ASB. And so that's certainly been something that's been frustrating. But it is part of this consensus development process and achieving consensus in these highly contentious areas has been quite difficult. And this is just one of those fiction ridge documents that's stuck in that situation. >> And it does sort of open up a question about OSAC in general cuz if ASB Changes it dramatically so that it's no longer acceptable to OSAC. Then it will be an ASB standard, but will make it onto the OSAC registry. And so I do wonder on what the process is when you have things that are highly contentious. Is it something or other where we'll set our sights sort of at a medium level. But there's a certain point where it becomes impractical for us to come back with an overall standard. So that's what I have on the prepared talks, but I'd be happy to open to discussion or commiseration or suggestions. Bruegge? >> Where do you see, what's the path forward given all that you have articulated about the problems? Do you see it coming to a head at some point, or? >> Well, the document was written with the language to be a standard. And as we added more disciplines to it, then it became more difficult to get the disciplines to buy in. Or it was something or other where we had fairly weak approval. And so it's something or other where we would have, we might have been able to get a half hearted support to get it out of OSAC. But given that it's so contentious, then I would be very certain when it got the SB. Or it's actually since it's multi-disciplined, where it would go as a standard piece. It gets complicated, and so what would happen to it, after it left OSAC, becomes problematic in itself. And so we talked among the committee and decided that we wanted to reframe it as a framework. Remove all of the normative shall statements. And then turn it into more of a discussion, and Bill has a point, I'm coming back with the next draft, which is a good start. I think it is the pad of discussing what the issues are is probably what will win. However, having watched how much every single word is argued, I think that it's probably got a long path even at this. Because what people except as valid criticisms or benefits of each side are going to be battled. Bill? >> Yes, I think that the path forward on this is further analysis of the underlying issues. Part of the reason that people are disagreeing is that people have been trained to present their conclusions in certain ways. But that training, it wasn't based on any careful analysis of the strengths and weaknesses of different approaches. Or alternative approaches. People are simply trained to do it a certain way, they're committed to it, they're comfortable with it. They may not even, not everybody even realizes the difficulties of some of the approaches. I've talked to a lot of forensic scientists who present what you are calling posterior probabilities. Without having a good sense that they depend in part on some assumption or evaluation of the prior probability. And so I think that part of the reason that the OSAC members have had difficulty agreeing. Is they come to the problem from different perspectives, and they haven't fully sort of grasped all of. They haven't done a complete analysis of the strengths and weaknesses of different approaches. >> On the other hand, it's, I have never done case work, I'm not an examiner, and so it's, and I think I'm reasonably well familiar. I think that some of it is that, it's not that the only argument in favor of source conclusions are coming from the practitioners. I think also that there are more academic approaches that are being proposed. That have not yet shown that they are appropriate, that they're valid, et cetera. And so I think that it's, and so some of this, I think that there are a number of people who I think have a good grasp of the issues. And it's still challenging. >> Yeah. >> Because in practice, we have a system that's problematic and works. And replacing it with something else that sounds good, but hasn't been validated yet, gets very challenging. And so we, and I think if we come up with something, does that mean ASB will bless it? Or ASB or whichever standards organizations? And one of the other broader thing is since there is no enforcement authority in OSAC. If we come up with something that really changes the discipline, unless you're in Texas, nobody mandates what has to be done. And so if we change things dramatically, will labs even implement it, becomes another issue. >> Well no, I'm not suggesting it's going to be easy. But I'm hoping it will be easier if we take the time to articulate and lay out in some detail. The arguments that have been made in favor of, and against each of the possible positions. In fact, even identifying, creating a taxonomy of possible approaches, right? How might we present source conclusions? Here are the possibilities, now, what has been said in favor or against each of these possibilities? I think once we get all of that in writing, it will help people think about this more critically, and perhaps reach a consensus. Or if we can't reach a consensus, it might be the basis for different subgroups adopting different positions. At least it gives them articulable reasons for it. >> And certainly making sure that everybody understands the strengths and weaknesses of all of them. >> Right, and I think we've got a long way to go even on that point, from my perspective. >> Well, and there's a lot of subtlety of, for example, on the posteriors. Is when is the examiner better situated to have an expert opinion on the application of priors than the court would be? That gets complicated really fast. >> Right and we can't, until we lay out that issue clearly, we can't even have that important discussion. >> Indeed. >> I mean, you're right, it's a very important issue. >> I think this is the path forward, it's not that I don't support the path forward, it's just the path forward will be challenging. >> Yeah, we need additional analysis. We need to think about it more before we decide what the standards can be is basically the answer. >> Also, another thing, I don't know if maybe you could address. Is that some of, I know I think a lot of the subcommittees have been dealing with this notion of having a document that is perfect. Versus a document that at least moves the discipline forward. And so, we've been struggling with getting comments back that kind of deal with hey, let's push towards perfect. Whereas the subcommittees are saying, look, hey, we're at least making improvement here, let's move these things forward. Any comment on that? >> Yeah, I think that's actually a big one, that in particular there are a lot of swig documents out there. And right at the beginning it's like, okay, we've got some notable holes in these old documents, and this is the best that's out there. And so let's rapidly turn around something that patches the holes in the swig documents. So that the record is at least showing something better. Because at the moment, if OSAC doesn't have a document on registry, we go all the way back to the swig documents. And the problem is the OSAC process is slow enough that it doesn't reward a quickly patching holes. It has a, okay, let's send it back for a complete rewrite. And that's where a lot of the slowdown in the OSAC process comes from. Mark? >> From an observer's point of view, watching the Ball go back and forth in a tennis match in our next turning, over two years. First of all, I wanna applaud you for the effort that you put in trying to bring together disparate groups. For the purpose of reaching a consensus, but it also is illustrative in a number of ways. One, there may be a challenge that's associated with bringing disciplines together and have a range of interpretations when they each have a different dialect, even if it's the same mother tongue. And these dialects of saying I like the ranges, but I like the ranges that I'm used to when I was trained in better than the one in the adjacent pattern interpretation section. Illustrates how true the variation is of the fragmentation is from one forensic discipline to another. And also incorporation of an academic point of view, a statistical point of view, a measurement scientist point of view, a dialog that did not take place when the SWIGS were operating independently. So it may be that despite the fact that you had two years of fighting and the debates that wore you out or could have worn you out if you weren't as persistent. It's really to the credit of the structure, the framework of this organization to address it where there's no one else and the community that would address it. The commission wouldn't address this, nor where they equipped to. The individual SWIGS couldn't address it, there really is no other organization that brings together all of the stake holders that are gonna make the contribution. And yes, you didn't get to where you were hoping in consensus the first time. One thing we've learned about OSAC is that if you don't get it right the first time revisit it. Come back and each time it gets a little better, a little better, and this is part of the slow evolution that we're witnessing. So on the one hand, you have given a really artful and articulate description of how we got to where we are. But I'm also looking down the road. And again, as the observer watching this match, at some point we're gonna get to the endgame, and the trophy will be awarded. And we just don't know who's gonna get it yet. Thanks. >> So Austin, I want to reinforce some of the comments that were already made. I think the time and energy you put into this project has been phenomenal, and quite frankly. You know we've been waiting for someone who's been brave enough to take it on, and I'm glad that it was you. >> Brave means unwise in this case. >> [LAUGH] I'm gonna stick with brave. But what I'm curious about is, so, and again, I agree pretty much with all the comments that have been made. I definitely think there's room for additional analysis. And for consensus building, and again, it's gonna be definitely different with the different sub disciplines coming from. In some cases, very unique perspectives on how they discuss their conclusions. So let's just say hypothetically we can come up with a consensus or a scientifically acceptable way. I'm expressing conclusions that relate the relative strength and chance of error and all of that. I'm a little bit more interested in the customer. So there was some initial discussion about what do we come up with? And then how useful is that to the customer? Whether that to be to a law enforcement investigator, an attorney or ultimately a lay juror in a jury trial. How much of the conversation has been on how the customer or in particular how the lay juror is going to interpret these, the articulations of these conclusions? And was a part of the plan somewhere down the road to run any particular studies to say, hey, this is what we want to do. Now let's test it with actual lay jurors and let's figure out the quote unquote get it. If they're interpreting the information correctly. Because I would just make the suggestion that if you're trying to get different disciplines, you're trying to get practitioners from different areas together and to unify bind a common purpose. I think if you came back to them and told them and said listen, this is our proposal, we've tested it and you know what? People get it. When we talk about conclusions this way, investigators, attorneys, judges and in particular lay jurors are getting the correct message. And that might be something that might motivate them to get behind a new or different approach to articulating conclusions. >> Here, I'll start the answer and I'm sure Bill will want to follow on. Is that, this is one of the concerns, cuz also I'm watching from the latent print world went from identification to individualization in the 90's before I was involved in Swig Fest. And then about the time I got into Swig Fest they got rid of individualization or at least made it identical to identification. And the thing is, they told everybody to change from identification to individualization. I don't think that many labs actually changed anything. And so and then, they changed back and I'm sure not all that many labs changed anything. So there is a certain level of when things change here, it's sort of like wiggling a rope and expecting the end to change. I think that the, what happens on the standards, really has to have a good reason for it to make it to the other end. And that's one of the concerns I have on what you're talking about, is that if you change something and it doesn't have any effect on what people do, why did you go to the effort? If it's an academic or semantic exercise, then it's not well worth it. There is the other side of if you change something, what is the effect on the examiner? What is the effect on the lay person? Biltam and Stansam preliminary work at least on what the effects are on reading this. I think one of my concerns I'm going into the, the more quantitative approaches or the posteriors are that it gets very challenging very rapidly. And so I think with some of them, it's gonna be hard for the examiners to understand when things are expressed as posteriors versus weight of evidence much less when you go out to the laypeople. And so I think that one of the problems with what we have is at least, it least professes to be readily interpretable though the implications of what they think they're interpreting may actually be a problem. And changing it may have unforeseen consequences. One of the concerns I have is if we change something, and it doesn't change the interpretation, but the examiner changes. What happens when we change something, but it changes their error rates that that's something where we basically, at least with latent prints, people are pretty good when they say ID however they express it. And so, if we do something where we change it, and they're not as good as that then that is also a problem. But yes, indeed. I think that it is how accurate and reliable are the examiners in making these conclusions. And what are they interpreted as when they are received? Bill, you want to follow on on that? >> Is that. >> I'll say, and I love the question, and the comment. Because I think you're exactly right. That when forensic scientists are thinking about how to present conclusions, and particularly source conclusions, there are clearly at least two issues. I mean one issue is we want the expert to be reporting in reports and testimonies something that's scientifically valid and sound. So we don't want experts saying things that aren't well grounded in science. But second, among the possible justifiable well grounded statements that might be made about a source conclusion. We want experts using ones that are easier for people to understand and that won't be misunderstood and misused in some way. And so there's reason to be concerned about how laypeople, lawyers, and jurors respond to the kinds of statements people make. And there's some tradeoffs. I mean, the kinds of statements that are easiest to understand. It's him, it's an identification. Maybe more difficult to justify on empirical scientific grounds and vice versa. One of the things that I think has been good about the discussion within OSAC of these issues is that it's brought more academics, people like myself, into the issue. CSAFE, Center for Statistical Applications for Forensic Evidence, which is funded by NIST, has allocated some money to doing research along these lines, and there are studies going on. It's funded some of my work at Irvine, it's funded some work at University of Virginia, and some at Carnegie Mellon. Actually, Iowa State as well. So we're starting to see some more research done on these questions. I mean, psychologists have in the past studied people's reactions in general to statistical evidence. But I think the involvement of academics with OSAC, meaning that the researchers are doing more to answer the questions, the exact questions that forensic scientists have and wanna have answered, we're starting to try to address. And the work, it's still early days for this work. But I think we know more now than we did a couple of years ago. I mean, one of the concerns that people had about going from identification to saying something about the strengths and support for example. We're using the likelihood ratio as whether people would just give it no weight at all. I mean, do you have to say identification in order for people to give the evidence, significant weight. And I think the answer to that's pretty clearly no. That people will give weight to these alternative formulations. There are concerns, though. I mean, when you start using numbers likely these ratios, even match probabilities, people sometimes misunderstand what it is at the experts saying in predictable ways. And it's not clear yet, I think, how to deal with that. And whether those problems are sufficiently serious that we need to think about not going down that pathway or not. So I think there is more to be done. But the fact that we're talking about it and that people from the social sciences are getting involved in discussions that clue them into the issues, I think is leading us in a good direction. >> Other questions, comments? >> I just felt a strong desire to clarify this issue of how long it takes to do things. Having been through the SWIG process from the ground floor up, having been through OSAC from the ground floor up, all of these different organizations as they try to do these documents, it takes a minimum of two years before they ever get their footing because they have to learn how to do what they're doing as well as they have to learn how to work together and get to this area of consensus. So when Austin and Melissa and all of us talk about we've been doing this for two years. We've been doing this almost three years now and we're just finally getting things to fruition. That's the norm. This is not something out of the ordinary, it's the norm. The difference with OSAC though, is that we've involved more people who were non-scientists, nonpractitioners in the process than they had in the SWIGs. And that brings in a whole different side of understanding and perspective when it comes to creating these documents. >> Awesome, we have a question that's online. It say, why go to all of this effort to change a system if it is possibly not an improvement? >> That's a very good and valid question. I think some of this is there was variation among different disciplines that they just had different conclusions. That comes up to the if they're different but not bad, is there a benefit in changing them? I think some of this is the, if it were clear, it would not be an improvement than yes, it would be a wasted effort. I think that every single other thing in this, people don't agree on what it should be changed to, whether it should change and whether it's actually an improvement. And so That dancing around enough is an answer? I mean, if it were obvious, that changing it were not an improvement, then yes, it would not be worth it. >> Right, okay. >> Yes, there has been a lot of criticism. >> There was a need for improvement. >> Yes. >> Okay. >> Yes, I think it was [INAUDIBLE] >> I mean, while it's going to him. I think one of the early things it's when was naively thinking that could have a rapid turnaround is just purely at a source conclusion that said, it's not okay to the say to the exclusion of all other zero error rates. It's absolute statements like that. And if you say identification, then it needs to mean this, you can't have an identification where you don't think about the two competing propositions. However, it got more complicated. >> So to respond to the questionnaire, there is a consensus that there is a need for improvement. To fail to even attempt to improve would be an abrogation of our duty. And so that's that. >> Yeah. >> To push back at Karen Kafadar, there is no uncertainty about this issue. >> [LAUGH] >> Yeah. But where to go is the problem. >> Okay, anything else? Thank you. >> [APPLAUSE] >> All right, yeah, I'd like to add one thing to Richard and since I'm at the microphone, I'll call on myself. One of the things that OSAC provides is an opportunity to define who the users are. And I think that one of the things that we've learned is that among forensic science practitioners, there's not consensus on who the user communities are. And we realize that there are multiple user communities where OSAC is going to focus the language of the standards. I think of ultimately will undergo an evolution where we say that forensic practitioners are a stakeholder community for whom these standards are being written. However, we recognize the need for jurors to have a clear understanding of what it is that we've said and to use language which not only defines for forensic science practitioners what they intend is a best approach to a standard. But also that there is enough information for judges and lawyers, and academics, and the public, in general, to be able to benefits from an understanding that we currently don't have. And all of these documents are living documents. And the reason that we adhere to that is that we understand it's evolving. We won't have it perfectly at the beginning, but it continues to improve. And so one of the benefits that OSAC brings is that we have voices from each of the relevant communities to address users. Recognizing that we have a variety of different uses, but that each and every one of these user groups deserves to have language which is understandable to them without losing scientific accuracy. So on that note, we have about until 2:45, we'll have an opportunity here from Dr. Richard Vorder Bruegge, so please come back on time and we will begin right at 2:45. Thank you. >> [APPLAUSE] >> Welcome back for the penultimate presentation of the OSAC subcommittee. I'm sorry Scientific Area Committee presentations, and the next one that we have is digital and multimedia. The Chair is Doctor Richard Vorder Bruegge, who is a senior photographic, Technical analyst. >> Technologist. >> Technologist, or the Federal Bureau of Investigation in Quantico, Virginia. So, Dr Robert, the floor is yours. >> Thank you Mark. >> [APPLAUSE] >> Okay, good afternoon everyone. It is my one real regret that I don't get to be the person that stands between you and beer, because I'm really good on situations like that. So, unfortunately I'm gonna take up all of my time rather than cutting it short. So, briefly I'm going to go over the organization and the status of the DMSAC. Touch on some of our focus areas and challenges, and spend the bulk of our time talking about this issue of this document, The Framework for Harmonizing Forensic Science Practices and Digital/Multimedia Evidence. So, I'm in control. As you see there I'm the Chair, Lam Nguyen of Mandiant Corporation is my Vice Chair. Doug Lacey, of the BEK TEK Corporation is our Executive Secretary. And Doug, I have to say, has been doing a dynamite job, in terms of keeping track of minutes of the meetings, and actually call meetings. We have four very accomplished and capable subcommittee chairs. Jim Darnell leads the Digital Evidence Subcommittee. Laura Sims leads our Facial Identification Subcommittee. Jim Wayman of San Jose State University leads our Speaker Recognition Subcommittee. And our newest subcommittee chair, Julie Carnes, of the Target Corporation, is leading our video and imaging technology analysis, or VITAL subcommittee. Here are the members of our SAC. I'll give you a chance to Take a look at this. As you see on the right hand side we are ably added assisted by our resource committee Exeofishio members. John Holloway of the Human Factors Committee. Dick Reeve of the Legal Resources Committee. And Chris Krouge of the Quality Infrastructure Committee. All of whom are strongly involved in participating in our activities, and we have really enjoyed working with them. Very quickly, I wanna run through some of the standards and some of the documents that we have been working on. The DMSAC, for the most part is affiliated with the ASTM, as our standards development organization. I happen to now be the chair of the E30 Forensic Sciences Committee of the ASTM. And so I look forward to being able to facilitate the process of getting standards from OSAC through the ASTM process. One thing I would also like to make a note of for my SAC chairs, is that it's very important that our subcommittee chairs and the task group chairs understand that there are people who are not in the OSAC, but are affiliated with ASTM. We're interested in participating in the development of our standards. And the FSSB which I am a member being as SAC Chair, made a ruling that the SAC Chairs have the ability to authorize the release of OSAC documents, to individuals that have a vital interest in participating in these standards. And that some subcommittee chairs apparently have not gotten that message. And so, the fact that a document is being developed within OSAC in general, means the document should stay within OSAC. But on those opportunities where there are people outside of the OSAC that can add value to a document. It is absolutely legitimate for the SAC Chair to advise the sub committee chair that no, you can share that document with individuals who are not necessarily in the OSAC. It's important that note also we encourage people like that to become affiliates of OSAC to simplify the process. But if for some reason they choose not to be an OSAC affiliate, there are still people out there who, for whatever reason, have chosen not to join OSAC, but they have a lot to say about these standards. So what you'll see here, the very first three standards on this slide right here are very near and dear to my heart. All relate to facial recognition systems and facial identification. The one that is the third one there, Facial Comparison Methods, is something of a misnomer. Outside of the document I hold in my hand, this one document represents, personally to me one of the single most important standards related document that I have ever been involved in the creation of. And what this is is that it refers to the checklist of features that individuals who are conducting facial identification comparison should be engaged in. One of the primary criticisms that individuals who perform patterned evidence type analyses and facial comparison falls into patterned evidence comparison. One of the greatest criticisms and that is often encountered in court. And I myself have heard this term from three or four different attorneys, none of whom I believe are in the room right now, Judge, is you're just looking at it. And my common refrain response to that question, you're just looking at it, Is that, yes, I'm just looking at this pair of images in the same way that a radiologist looks at an x-ray. Anybody in this room can look at an x-ray, just look at it. But who do you want deciding what type of treatment you are going to get as a result of examination of that radiograph? Do you want anybody in this room to determine if you have cancer? Or if you just have an artifact in your image? Or do you want someone that is trained, that is knowledgable in what it is they're looking at? And this particular standard document, this Facial Comparison Methods checklist. Provides for any individual who is going to engage in the process of one-to-one facial identification. The standard checklist of features that you should be looking at when you are going through that process. And that's the type of work that ASTM is doing and that the OSAC is doing. It is providing public clearly defined standards that allow people to understand what it is we're doing when we are providing our expert witness testimony. Two other documents that are also going through forensic audio laboratory setup and maintenance, you may say forensic audio. I didn't hear that there was a forensic audio subcommittee on the OSAC. For those of you who really wonder about the history here, there never was an audio swig. And the OSAC was effectively established by taking swigs and transferring them over into OSAC subcommittees. Forensic audio, as it is, is an area that is addressed by the scientific working group on digital evidence today. And as a result of that, it was incorporated within the OSAC subcommittee of digital evidence. So forensic audio laboratory setup and maintenance is within the remit of our digital evidence subcommittee. Finally, standard practice for data retrieval from digital CCTV systems. I don't need to tell anybody in this room that video and images have been a fundamental aspect of the investigation of major shooting events in this country over the last five years. The Boston Marathon bombing certainly proved to leadership across all federal agencies in the United States, particularly at the FBI. Of the critical importance of digital CCTV evidence in reconstructing the events that relate to before a crime takes place. And in the immediate aftermath of a crime. Getting that evidence properly secured and retrieving it is of the utmost importance. And having a standard that will instruct the community on how to properly recover that video is only going to become more and more important. As more and more crimes take place, and more and more video is encountered. There are in fact two documents in ASTM that are very close to coming back to the OSAC and ready to be brought into the registry. One is the standard guide for establishing confidence and digital forensic results by error mitigation analysis. The other one on forensic digital image processing. The first one, we could spend the rest of the hour talking about this one. Suffice to say, there are a lot of analyses that experts in across the DMSAC address. Issues of authenticating whether an image, for example, has been manipulated or not. Or whether a file system is accurately recording the events that occurred in that. There are a lot of possibilities associated with digital evidence, digital and multimedia evidence. That because of the variety of situations that can arise. To say nothing of the fact that different changes in technology, upgrading of technology, changes to operating systems. All of these factors make the process of digital and multimedia examination far more complicated than some of our other disciplines across the OSAC. So that you can't necessarily come up with an error rate for a given process, or a statistical basis on which to draw a conclusion. Effectively, you have to rely upon the examiner, and the practices that are being used, to identify- The best possible explanation, while identifying that there are others that could be in play. So this guide basically helps the examiner understand how he or she may go about demonstrating that they are mitigating any potential errors in their analysis. Now among the other things that we have priorities for 2018. You heard from Melissa earlier talking about how the documents getting posted to the OSAC website prior to their going to an SDO. Once they've been approved by the SAC, getting them posted on the website, we have some baseline speaker recognition documents. Which, in contrast to other documents that have been posted on the OSAC website, don't necessarily have any other source from which they came. Basically they came from within the subcommittee itself. And so it has been a challenge getting approval to have those internal documents made posted online. Because Mark said it, others have said it, I take the role of the OSAC to be first and foremost to improve forensic science. I am very much in the camp of, let's make a little improvement. We're not going to achieve perfection right away, so let's make steady, solid progress. To me, to make steady, solid progress, you have to define where you are at the start. And this is the purpose of what all these baseline documents are there for. They are effectively put in place to say, this is where forensic science is before OSAC. Unfortunately, in the discipline and speaker recognition, there's never been documents like that that exist. And so, our speaker recognition subcommittee has done Gilman's work in putting together documents that allow us to get there. And so, I'm trying to shame the members of the FSSB to come into my side of the camp. And agree that these documents should be posted. [COUGH] We also have a process map for speaker recognition. This is another area where our speaker recognition group is really on the cutting edge. Effectively what we're talking about is building this flow chart of how one goes about doing a speaker recognition examination. Flow charts like that can be applied hopefully to all of the disciplines. And I know there is a latent print process map that was relied upon in some part for the speaker recognition one. We're really looking forward to seeing that move forward. Finally, the liaison status. Again, the speaker recognition committee has got a lot of international participation. And in particular there is an ISO/IEC subcommittee 37 that is addressing speaker recognition issues. And we're in the process of trying to make our Speaker Recognition Subcommittee a formal liaison to that subcommittee. So that OSAC although it is a body created of by and for the United States of America, these standards shouldn't necessarily just apply within the boarders of the 50 states and territories. We really want OSAC to be an international leader. And by participating with ISO, I believe we can move that idea forward. Finally, training standard across multiple OSAC disciplines. I'm gonna give a shout out to Scott Oulton, who must have known I was gonna say that because that's why he left the room five minutes ago. There is the idea of developing a training standard that will apply across multiple OSAC disciplines, it's something that can benefit all of us. So [COUGH] our focus areas within the DMSAC are listed here. Accreditation, digital and multimedia disciplines continues to be an issue. There are some issues related to what exactly disciplines certain activities should fall within that we need to address. You heard Austin and Melissa talk about the conclusion scale earlier. Terminology, we also heard about earlier today, both with the 29 terms that were brought up earlier this morning in the OSAC wide lexicon. Error rates through the testing and the examiner's blackbox testing, is certainly something that we have seen from the benefit it provided to the latent print discipline. We in the facial identification community are on tenterhooks because have had a blackbox test. Some of us are aware of the results and we are just waiting for the publication can be accepted, so we can crow about them. And finally, foundations, and I'm gonna come back to that at the end. But lastly, I wanna get into this issue of the scientific paradigm for digital and multimedia sciences. If you go back three years, to the very first time the OSAC presented a meeting like this at the AAFS. There was a telling moment when one of the audience members stood up and basically said, what the hell is digital doing in forensic science? What makes this a forensic science? So that took quite a few of us aback that there were still people that didn't necessarily think about digital and multimedia science as being forensic sciences. In fact, not to reopen old wounds, Mark, I do have my card back there that basically was drawn out. When OSAC was originally formed digital evidence wasn't a part of the OSAC subcommittee, through some arm twisting we got it included though. But still this question remained, what is the scientific basis? And so I tasked the group led by Mark Pollitt, Eoghan Casey of OSAC and some of their academic colleague to develop a response. This task group, the primary authors you see there, Mark, Eoghan, David-Olivier Jaquet-Chiffelle and Pavel Gladyshev. Along with other task group members, Martin Olivier, Mike Piper, Lam Nguyen, Dick Reeve and Marcus Rogers. They prepared this document that I am really, really proud to say is, Now published and on the OSAC website. Copies are available at the back. There will also be additional copies available at the NIST booth on the floor tomorrow. You can also download it quite easily. Although it is listed as the OSAC technical series document number 2, I would say that it's really the first substantive document. Yeah, yeah, that's right. Well, the first one is an annual report, okay? So [COUGH] anyway, this document started as I say in April of 2015. And it has gone through multiple iterations within the DMSAC. Across the resource committees, across the FSSB, until we got the final point here, which basically we're answering the question where is the science? Now the note from this third bullet, it says the work continues both internally and in collaboration with the rest of the OSAC. This is not the definitive answer, science changes, okay? It's a constant move forward, we hope. And so this document is a starting point. Let's try to get into a little more detail. We have some notes of the motivation for this. Because this question didn't just come up at the American Academy, this question comes up at trial. For those of you that aren't familiar, this individual, Johnny Oquendo, who was ultimately convicted, had been tracked by his phone, to and from the scene of the crime. And the evidence was going to be offered. It basically came down to tracking information that Google generated and that was recovered through subpoena with Google. And as it says here, the defendants attorney were looking to see if this is good science. And the judge ended up excluding this evidence. The problem was you had a FBI agent who testified to what was done with the data, to track the location of the individual. You also had an individual from Google that explained the formatting of the data and what the data was. You didn't have a third individual who explained effectively the provenance of the data and where it came from. And so the judge excluded it, because as it says here, it wasn't clear that there was science underlying this, nor that there was general acceptance in the relative community. So the approcher task group took was very much a first principles approach. First, what's science, okay? And how does that apply to forensic science? Then, how does forensic science apply to digital and multimedia forensics? And then, look at how what we do in digital and multimedia forensics applies across our various disciplines. And then, articulate what are the scientific processes that we use within our work? So, this is going to get a little bit philosophical, and this has led many reviews of this paper within the OSAC to say what are you talking about? Okay? You've heard earlier today the fact that OSAC represents the bringing together of stakeholders that have not previously worked together before. I see that as the greatest strength of OSAC, okay? In the past we've had the SWIGS. They would bring in academics, often. Sometimes they would bring in lawyers. But we never had human resource people. We always had quality assurance people, but they weren't necessarily always highlighted there. And statisticians, we didn't have then enough of them. The OSAC brings those people together and so we can address these questions. The idea of what we're trying to do in OSAC is that bottom point. We need to help the decision makers reach an informed decision. And that's what we're trying to do in the forensic sciences. And so, it was recognized pretty early that one of the reasons that digital evidence in particular, is often challenged as being a forensic science, is where it is applied in the process of an investigation. You'll note here that forensic results don't just apply from the moment something is brought into the lab and then move forward. But there are certain other aspects where forensic decisions get made. In an investigation, okay? Forensic results can be used to direct who you're looking for, okay? That doesn't necessarily relate to going into a court and saying I did this or I did that. It also applies obviously within the courtroom, but also it affects how we look at research and where things fall. So, if you think about these three different scenarios, we can think about the scientific process and the application of forensics. So, basically we try to come up with the definition of what is it we are doing in forensic science. And I wanna note that the definition that we use for science, is basically a systematic and coherent study of the universe or nature. Human beings, our technology, are at the most finite level a part of nature. So what we do and the things that we create, are all the subject of this type of systematic and coherent examination. So, in forensic science, we are investigating what we've refer to as traces. And one of those lessons that I have learned working on the terminology within the OSAC, is that when people start using words in ways that are different from the way you as an expert or accustomed to dealing with them, your immediate reaction is to pull back. So that's now how I define that. One of the recommendations that we made to the FFSB based on the 29 terms, and God bless you have been here since eight o'clock this morning, you will recall that Steve mentioned what our recommendations were. One of the fundamental recommendations, people need to be prepared to understand the terms that are being used within the context where they are being used. And so for those of you who might be trace evidence analysts, or might think about other aspects of forensic science where traces is not what your examining, cut me some slack, okay? Evidence gets left behind, we would refer to that as a trace. If something is removed, then it leaves a trace that there was something there. So basically a trace can either be something physical, digital, electronic, what have you, or it could be the absence of that. So, in forensics we are looking for these modifications that are observable which result from an event. So as it says here it can be physical, it can virtual, material or immaterial, and it can be the presence of it or the absence. We also tried to identify, what are the forensic questions we try to answer? And, we came up with this list. So, the systematic and coherent study of traces to address questions for legal context, is how we define forensic science. And it comes to these issues of authentication, identification, classification, reconstruction, and evaluation. I'm gonna get into those. Now, The process of scientific reasoning can follow several different courses. And we use this triangle to try to frame the way we go about trying to answer questions in the investigative process. And sir, I encourage you to download the document, all of these slides are in the document. We can look at abductive reasoning, deductive reasoning, and inductive reasoning. And what you're seeing with these diagrams is basically a triad of perfection in knowledge. And so knowledge, overall, can refer to basically understanding how things work, okay? In the lower left hand corner we have activities, so things happen. And in the bottom right, we have traces. So we know that when things happen, you may get traces left. So if I put my finger on the podium, a friction ridge impression may be left on this material, okay? If all of us came up here and put our finger on this podium, then over time we are going to come to understand that when one places friction ridge against an item, a trace is left, okay? So that constitutes knowledge. We have a way that informs how we have come to know that traces get left, okay? So, in a testimony scenario, a courtroom scenario, the idea that we have knowledge of how things happen, and we have traces, allows us to predict what activities happened, okay? So if I have a crime scene and there is a friction ridge impression on the doorknob, I may, and I ultimately through a source conclusion link that friction ridge impression to a person. As the prosecutor I am going to say, that suspect whose friction ridge impression is on the doorknob, opened that door, okay? In the investigation scenario, if you see a friction ridge on the doorknob, you say, hey, if I find out whose impression that is, I can find them, okay? In research, we use this process of applying fingers to surfaces which create traces and therefore we come to knowledge, aha, when someone does this, it creates that, okay? Deductive reasoning on the other hand, we're basically fact checking. We have the situation where we have this basic knowledge and we have knowledge of activities, and we're basically saying. The suspect put his hand on the doorknob. Therefore, there should be a fingerprint on the doorknob, okay? Fact checking different scenarios. If I in that case know that they put their hand on the doorknob, then I can see that there is a friction ridge that corresponds to that person. And I see the trace allows me to demonstrate that there is a constancy there. Inductive reasoning basically is going back to the very first thing. If enough people put their hands on the desk, we develop this knowledge that when you do that activity, it creates the general theory of transfer friction ridge impressions. Now, that doesn't give us the entire world of possibilities, though. In a real-world scenario, the diagrams that I just showed you were completely on their sides. The triangles were filled-in completely. Here we don't have the triangles filled in completely. Because we don't have complete knowledge. So I was just giving you that example of the person putting their hand on the door knob, and leaving a friction ridge impression. What I didn't offer to you, was the alternative proposition. It's someone could have taken a lift of that persons impression and somehow figure out how to transfer their fingerprints to the doorknob. Okay, that's an alternative proposition that might have to be considered however unlikely it is. God, Bill, I'm glad you're not grading me on this, because I know I'm doing a butcher's job on this. But the idea is, when we have traces and we have a way of understanding how they came to be, we can infer or we can adduce what happens. With deductive reasoning, we have knowledge and we know what activities happened. So, we look to verify whether they are compatible with the traces that we saw. And finally, with inductive reasoning, we have known activities, we have traces, and we establish, wow, we now have the new way things happen. So again, what is the time, Mark? Can I have a, I have 20 minutes, thank you. So now I will proceed to talk about every single word on this chart [LAUGH] >> [LAUGH] >> I'm not gonna do that. We're basically breaking the forensic process down into these core forensic processes that are in the middle of this chart, this reconstruction evaluation classification identification and authentication. Relating them to the scientific reasoning that goes into them and then adding to them all of the different forensic activities that are a part of what we go through. So, if you look at the things on the left side, you'll see that there are things like survey, preservation, documentation, integration and interpretation, these are all things that we do within forensics. But they don't necessarily get to these core forensic processes, they are things that we use. On top of this, we also have other things that we do in forensics that fall outside of these things, and we refer to these as operational techniques. So recovery of evidence is an example of an operational technique. Basically I'm going to image that hard drive so I can preserve the evidence. I'm going to photograph the crime scene so that I can preserve the way that the scene looked and we don't have to keep that crime scene pristine or five years until trial ends. Enhancement techniques and recovery techniques are another piece of this, so I'm going to take that bad, grainy digital video image and improve the qualities so I can see what's going on there. That is not necessarily a forensic analysis in so much as the expertise of the people using the technique to improve it is certainly forensic. But in of itself, making the picture clearer to allow for better determination of what you're seeing Is one of these operational techniques. So quickly running through these five different processes, authentication refers to this decision process that attempts to establish confidence in the truth of a given claim. All of the other processes depend on items of evidence being authenticated in one way or another. So if we have these two photographs on the right we might try to authenticate it by saying, has this photograph been altered? But in other cases it may not matter if these two versions of the photograph. It may only matter that the photograph was taken in Seattle or that it was taken on a given date. Or that they're identical at some level. Okay, all of those are questions of authentication that are gonna depend on the situation. Identification, and I'm gonna use, identification is used in the pattern sack, trying to establish the confidence that two things correspond to some level at a high enough degree to declare that they are from the same source. So it may also include not just this identification, is this person the same person in these two photos? But it can also determine was it really at this particular time, all right? So it incorporates some of these authentication, classification, and other evaluative processes. And it can address questions not only of Is this the same person, but was the same camera used to take these two photos? Classification of course, relates to applying and inserting objects within a taxonomy. Coming to the point, that you have a taxonomy that defines your classes and the ascription of objects to within those classes. Reconstruction, another level, okay. Trying to determine that in this case, the first bullet of functional analysis. Did something work? Okay. Did it work as it was expected to work? Is there a temporal analysis? Okay, what was the sequence of the events. And then are there linkages between people? Okay, so relational analysis is a part of this. Typically again, this is related, achieving a reconstruction can involve these core forensic processes, and it can also support the other core forensic processes. Finally, we have evaluation. So, evaluation is basically that end point where you are taking the traces and the various examinations you have performed and you're trying to evaluate the claims, okay? So if you have two competing claims you're going to evaluate whether the evidence are more likely given one claim or the other, whether there is more support under one scenario than another. So the document goes then to address these questions of talking about whether we are addressing the strength of the evidence On the strength of the hypothesis. So, I can't really put it any better than it says here on the screen. You have to draw the distinction between the examiner expressing the strength of the evidence, versus the strength of the hypothesis. And retake the position that it is the forensic scientist's job to explain, Explain the value of the evidence in supporting one proposition versus another. But not going to the hypothesis of the activities that led to the evidence there. So I mentioned these other supporting activities earlier. If you look at digital and multimedia evidence, [COUGH] We have documented how these different processes that we engage in are common to many of these things. And if you notice, all of the examples that I gave of authentication, reconstruction, authentication, identification, fell into the digital and multimedia discipline. So, within the context of this document, this framework can be applied to digital and multimedia evidence in much the same way that it can be applied to other forensic disciplines across the OSAC. And ever since the OSAC was formed, I have taken it as one of the guiding principles to look back at the National Academy Report And try to say what were the problems they had, and are we in the OSAC addressing the problems that the NAS report identified? And one of the major ones was the disagreement across different disciplines. The lack of coordination across disciplines. And so I take it to be a vital role of the OSAC to do what we can to draw similarities across the different disciplines when we can. And so this document is a major step forward in trying to do that. There were seven major recommendations that came out of this document. Trying to address further the challenges that we face with this particular terminology. All of these terms that I went through I'm sure may have slightly different definitions in your minds, and that's fine, so long as we maintain awareness of what context we're using them in. We wanna further explicate the scientific foundations of all of these things we're doing within forensic sciences. Examine how to minimize bias, improve the characterization and the description of our results, as well as the quality. And this is where we come back to a couple of these terms that I mentioned earlier as being fundamental to our focus areas. The terminology, there are some words that people want to use as a shorthand, and we have to get away from that. Okay, if we're going to talk about bias, what's the context of the bias? Are we talking about confirmation bias? Are we talking about contextual bias? You can't just say bias and think that everybody is going to understand what you're talking about, we have to be specific to the context. Mark ten minutes? Five. 12 minutes, okay. So then we have to get into this question of foundations, and One of the things that we are going to be kicking off, and my colleague Jose Amaral is the Chair of the foundation's task group. We had in 2016, a, what did we call those meetings? I'm having a- >> [INAUDIBLE] >> The OLSS, yeah, the OSAC Leadership Symposium Strategies Session. >> [LAUGH] >> Basically, it wasn't intended as a Kumbaya moment. But it was an idea to bring together folks from across the entire OSAC and say what's not working, and what, It really, there wasn't a whole lot that was really working well. So it was a question of what's not working, what are we trying to do here? And one of the foundational things, and that's why it's foundational, that we identified was that there's a need across all of the forensic sciences, and certainly, there are some forensic disciplines that are doing just great. I'm not gonna name names, you know who you are. >> [LAUGH] >> I'm not a bomb thrower at all. [COUGH] But, There are some disciplines that need to step back, and those of you that are doing really well, take a step back and remember how things were 100 years ago, okay? Some of our disciplines are kinda like just figuring things out. But the idea is, What are we fundamentally trying to do within a given discipline when we are conducting our forensic analysis? What are the questions that we are being asked, and what are the answers that we're giving when we answer them. And then you have to say okay, so how well, how scientifically valid are the answers that you are giving to each of those questions? So you have to go through this process and say, what is the basis for making this assertion to a given claim? Because if we understand how those questions are answered within one discipline, we may be able to apply that same rationale to how they're answered in other disciplines. Or at least explain, so the triers of fact, whether it's jurors, whether it's lawyers, whether it's judges, whether it's police, okay, explain how to correlate one given discipline to another discipline. And explain that the answers that we give you in this discipline as not going to be as good as they are over here, simply because of the facts that our particular discipline has a lot of variables that can't be controlled in ways that certain disciplines can. So, Jeff Saliards, I'm gonna to call you out. Is undertaking on behalf of the FSSB an effort to work on foundational work that Jose and our task group have started with our foundations document, where we are going to address these questions of what are the scientific questions we answer? The digital and multimedia sack offers our organizational structure this framework, as a way to place all of these discussions moving forward as a way of Making If not standard across all of the OSAC, at least common across all of the OSAC, the way we're going to address these questions. So with that, I've left enough time, but, I'm sorry, I- >> [LAUGH] >> No, in all seriousness, this paper, and I'm just gonna read the dedication. This document is dedicated to the memory of Carrie Morgan Whitcomb, who was both a source of ignition and accelerant in the development of digital evidence as a forensic science. As a forensic scientist, leader, and mentor, her contributions to the field will be enduring. For those of you that don't know, Carrie died last month. But those of us in the Digital and Multimedia SAC and in the Digital and Multimedia Sciences Section of the AAFS wouldn't be here without Carrie. And so I would just ask for a moment of silence in memory of Carrie at this time. Okay, thank you. Now, I'll take questions. Judge. >> Yeah, the two documents that you took through ASTM that have now been through that process, where are you at specifically in getting those through the OSAC process? Votes have been taken or are they being voted on, etc.? >> No, so we still have a little bit of clean up to do at ASTM before we're actually going to bring them back. Since they were out of subcommittees, they'll go back to the subcommittees. The subcommittees will bless them. They'll move up through the SAC, and then will come up for the vote through the FSSP. >> Okay, and in that process, do you expect that they'll go back to the resource committees, the statistics group, to get their input on the revised documents? >> I suspect that we will, Judge. Although, I think that both, I mean, all of the resource committees were involved in these documents, yeah. >> Yeah, they were and we wanna now see what the result of our involvement is or is not as the question maybe framed. >> Okay. Yeah, well. >> Might stop now. >> I mean, it seems there are at least two reasons why you might wanna talk about and think about whether digital and multimedia science is a forensic science. When people ask, is this really a science or is it really a forensic science? And one reason is that I think people are curious about whether the conceptual foundations and approaches taken by practitioners in your field, whether they like those that other forensic scientists use and how they might be similar or different. And I think that your frameworks document does a master for job of laying all that out. And I think it's enormously useful to kind of establish how it is that practitioners in your field think about what it is that they're doing and how they approach the general issue. And what are the forms of logic that are used, and I think all of that's great. But the question you put up at the beginning about, well, where somebody is asking the judge, well, judge, is this really a science? I mean, when people ask in court, is this discipline, is it really a science? I think what they're asking is something that's beyond conceptual foundations. I think they're asking, is it trustworthy? >> Yeah. >> Can we believe it? Are their results accurate, right? And so I think, it's important to be clear that although your document is great at addressing the first question. I mean, it shouldn't be put forward as an answer to the second question. I mean, the kind of black box studies you're talking about, really, are needed to address that question, right? >> Yeah, and you're absolutely right. That's the point that I was trying to get at by saying they had the expert who explain how we use the results. And they had the Google technician who basically explained what the results were. But that piece in between that explained how they actually work was missing there. And the reliability. Were there tests done to demonstrate that this is accurate? Did people take phones out? And then track where they were establish the ground truth? And I couldn't agree with you more. And I'm glad you brought up the black box test. There are fundamental differences across different disciplines across the OSAC, where the results really are dependent on what's going on up here. Because we have not yet acquired enough knowledge that would allow us to completely automatically, I guess, measure a similarity or difference in a way that allows us to draw a source conclusion, right? So in the absence of that ground truth data, that foundational data where we are jumping ahead to the conclusions being made, it's always been my point that black-box testing is the fastest way to demonstrate to the courts and to the critics that this stuff really works. And so I am a big proponent, and Austin, don't kill me. I spoke to the PCAST Committee, and that was 90% of what I talked about with them was black-box testing, I believe is really important, so. Other questions? We have plenty of time. This is unbelievable, John Paul. You're usually pulling me off the stage. There's a question here from Joann. >> Just to follow up on the black box. You mentioned that you had a facial, do not take my picture, please. >> [LAUGH] >> I've had more photos taken by OSAC than I have in my entire life. But you mentioned that the facial black box had been done and you're just waiting for the acceptance of publication. Are there plans for a second black box given the recommendations by PCAST? And also as a follow-up, are there plans for black-box testing in the other subcommittee areas under the D/M SAC? >> So I'm gonna address your second question first. It's among the research needs that the Vital Committee put out. Was the need to do black-box testing for vehicle identification, make model identifications. As this time, I'm not aware of there being any funding available for that, but it's on the list of research needs. I really wish Jonathan Phillips was here to talk about the additional black-box testing. I can tell you that I am aware of black-box testing that has been done in individual organizations. Independent of the NIS testing, these black box tests do not necessarily meet all of the check boxes that the PCAST called for. But I believe that there is value in those as adding to the literature and the documentation of this. What is going on right now, follow-up study, is more along the lines of the white-box testing And further, Research into perceptual ability to do matching as opposed to the intrinsic examiner type matching on faces. And by that I mean traditional cognate science research and the ability of human beings to match unknown subjects is a perceptual side of this. That testing is being done in order to correlate individuals perceptual ability with their results in our black box test. >> Thank you, you're now free to go. >> [LAUGH] >> I have to wait for Mark, I think. >> [APPLAUSE] >> Thank you, thank you. Well Richard, you didn't disappoint, it's always a very stimulating presentation when you have an hour to spend. So we're grateful for your help. Before we break, I also wanted to remind Joanne, that we're preparing OSAC comparisons preparing a photographic album of all of your photographs. And we'll send it to you and your children, so you- >> [LAUGH] >> I'm sure you're looking forward to that. Any case, we are gonna be breaking until 4 PM. And then we're gonna be hearing the last presentation by Professor Jose Almirall who will talk to us about the Chemistry SAC. So please come back and join us at 4 o'clock. Thank you. That's me. All right, thank you. Yes, I may have your attention please. The computer says it's 7 PM, and so it's two hours past Miller Time on the East Coast, so we'll get right in to the next presentation. The next presentation is for the Chemistry and Instrumental Analysis Scientific Area Committee, and its Chair is Professor Jose Almirall. And Dr. Almirall is a professor in the Chemistry Department at Florida International University. And we also have the pleasure of having him serve as the Chair of the Chem SAC, and we saved the best for last. I'm sorry, Richard. >> Men. >> [LAUGH] >> Taking your thunder. All right, if you weren't so predictable, we wouldn't be teasing Richard so well, and this is excellent. So, without any further ado, Jose? >> Thank you. >> [APPLAUSE] >> Well good afternoon everybody. It's my pleasure to be here, and talk about what's going in the Chemistry SAC. We were asked to come up with catchy titles for our presentations. And apparently my title wasn't as catchy as some of the others. But the reason I selected this title, Finding Meaning in Forensic Chemistry, is that forensic chemists convert raw data from observations and measurements, into information, and knowledge. That can then be used by the triers and facts, in their decision making process. If we can take good data that we have hope that we will be able to interpret that data into good information and good knowledge that can be put to good use. If the data is not good, if it's bad data, poor resolution, bad sensitivity, not reproducible then there is no hope. Garbage in, garbage out. We have spent the majority of our efforts, and our attention, in the last 20 years, in making sure that we're taking good measurements. And so many of the standards that have been through the ASTM process have been devoted to taking measurements. Some of the more interesting and challenging work that's gong on right now within the chemistry SAC is devoted to the other interpretation of the data. And so this is very difficult, and a younger version of myself, maybe 22 years ago- >> [LAUGH] >> Together with a colleague at Florida International University, Ken Furton wrote an article. And the purpose of this article, I was a member of ASTM. I was a member of and thanks Ed Burdick, for starting up the glass subgroup of back in the early 90s. The purpose of this article was to raise awareness, first, of the power forensic science, and we had DNA emerging. We had things like drug fire, but also about the standards process. The importance of standards and actually a subtitle of this article was The Importance of Standards in Forensic Science even before it was fashionable, okay? And we got in trouble in forensic science because we were overstating. So a lot of the bad press that we got was really due to interpretation issues. And I made some predictions, I was really bold, made some predictions about What was gonna happen in the future back in 1995. Some of that didn't come to fruition. But I did say that forensic science community anticipates an ever increasing role of statistical methods of analysis and data bases will become increasingly available on more evidentiary materials. And one of the beautiful things about OSAC is that it allows for this larger community, lawyers, statisticians, metrologists, human factors people to weigh in. And those people, we weren't talking to before. Even though ASTM was mainly practitioner driven, we had some academics. But that's the beautiful thing about OSAC now, is that we have this expanded participation from these other groups that really ask these important questions. What about interpretations, which we were asking back then, but we didn't get very far. So a lot of the things that I'm gonna talk about really, I'm gonna focus on really highlight today, is about interpretation. And it's happening in almost every single one of the subcommittees within the chemistry set. I'm gonna start with just kind of an overview. We're doing quite a bit Work within chemistry disciplines. Just those documents that are undergoing some sort of SDO evaluation, we have 70 somewhere along the lines in the SDO process. That is an improvement over the last one year's 161. The main thing I want to bring your attention to is some of the subcommittees had already fairly mature structure, some history, SWGMAT, for example, is more than 20 years old. So there were documents that had already made their way through that system and through the ASTM process and fire, debris and explosives analysis is another example, materials. But there are other subcommittees like geological materials, fairly recent development before joining the OSAC. And so there are existing standards, 35 back in 2017. But there are also some new standards. Look at this. 26 in 2017 and some of those from 2017 now are no longer new because they've been incorporated into the OSAC process. One of the things that I want to bring your attention, if you don't read my whole spiel in the document, I want to read to you one sentence. Although innovation and improvements to ASTM's standards are ongoing, the basic science in the existing ASTM methods is sufficiently developed and mature such that laboratories should be using them. I don't know if that's a controversial statement or not but I believe that. All of these documents that have been through the ASTM process are good documents. And the people in your lab should be aware of them, and they should be using them. Part of the purpose of OSAC is not just to develop and recognize standards, but also to implement them. And so we have to be cognizant of that. We have to be, as OSAC members, find ways to make it easy for lab directors to adopt them. And we had a FSSB chair who, he led by example. He said if there's an OSAC standard that makes it to the registry, we're gonna implement that in our SOPs. That's great. So here's just a summary of the existing documents, the new documents that are going through quite a bit. Thanks to the subcommittee chairs, I have a list of all of these and where they are in the SDO process. OSAC approved, we have three, yes. I think I'm very proud of these three because they are the first, really, measurement standards that have been through the entire registry process. And two of them come from the hardest working subcommittee on the OSAC and that's the Seized Drugs Subcommittee. And Sandra who's here in the audience, don't look down, I have to say is the hardest working subcommittee chair, I would say. And it's hard to be first cuz you take all the shots. And she has made it through that process. There is also another document to come out of materials, not to diminish what they have accomplished. The Trace Evidence Materials subcommittee has worked very hard, and they have a lot of irons in the fire. So in addition to, well all this work happens at task group level. So we have task groups that are made up of existing OSAC members, also affiliates and invited guests. So if you're not an OSAC member and would like to get involved, there is a way to do that. You can become an affiliate. You can apply and express your interest to the subcommittee chair or to the SAC chair, and we'll make sure that if you're interested, that we get you involved somehow. So there are more members participating in task groups than there are members of OSAC because of these affiliates. We also have the ability to invite guests, and we do invite at least, I don't know, 18 to 20 guests at each in person meeting for the entire SAC. So just let us know that you're interested, and we'll get you involved. One of the other products from this SAC is a list of those research needs that have been identified by the subcommittee and then approved by the SAC. So the Scientific Area Committee votes, the members vote, on these research needs and they're posted on our website. We're very public and transparent about it. And this is for the benefit of the researcher community to go out and read what is it that we feel at OSAC is needed from the research point of view. It's also useful to those funding agencies who are looking to fund research. They will look at this list and give importance, I hope, to the list that are identified when they're making funding decisions. So you can see that within the Chemistry SAC, we've identified 24 specific research projects and that number is growing all the time. So this is the leadership of the Chemistry SAC. Chris Taylor is the executive secretary, Chris Bommarito. Chris Taylor's here actually, I saw him earlier. There he is, thank you. And Chris Bommarito is the vice chair. We have four new members in the Chemistry SAC. We have turnover of at most one third every year. So out Adam Negruaz is also here, right. Saw him earlier. Adam, is a new member of the Chemistry SAC and some of the other members of the Chemistry SAC are also present. Sue Hetzel is the new chair of the Fire, Debris, and Explosives. Rod Simmons is the new chair of the Gunshot Residue. Diana Wright, this is not Diana Wright's email address, is the new chair of the Material. See it's an imperfect process. >> [LAUGH] >> I'm telling you. But you can reach out to these people, if you're interested to work. So we do have turnover. We are always looking for new members. You can participate as an affiliate even if you're not a voting member, but you can still participate. So let me start with the hardest working subcommittee. This is the Seized Drugs Subcommittee. And they will focus on guidelines related to the examination of controlled substances, drugs and related substances. And here's the makeup of the leadership, Sandra, David and Agnes. And they also have some new members, also in blue here. And you can see the representation is state and local laboratories, private industry, researchers. And there's a fairly good mix, and as well as the affiliates. These are also very hard working members of the subcommittee. So, in 2017, they had four new members. And here's a makeup by employer classification. What have they done? Well, this is quite an accomplishment, two ASTM standards that have made it all the way through. Are they perfect? Maybe not. But they are good, they are so good that you should be using them. You should be aware about them. You should be using them. There's a lot of ongoing work within the SDO process right now at ASTM on certainty assessment in the context of cease drug analysis, and then Clan Lab. So let me talk a little bit more about that. So here's the uncertainty assessment. The key components are in qualitative analysis, quantitative analysis, estimation of measurement uncertainty, reporting of uncertainty. And then with the Clan Labs terminology, safety issues, sampling issues, analysis it self. Yield and capacity calculations, as well as conclusions and reporting. There is a really interesting, and this is what I want to highlight with regards to interpretation. This is a new effort that emerged out of the OSAC process. A standard practice for assessment of GC-MS data for qualitative analysis of seized drugs. This is an interpretation standard and is currently being reviewed by the SAC and then it will eventually make its way through DSTL. Other items of interest within the chemist size drug subcommittee terminology proficiency testing. So these are broader kind of documents and interest that affect the entire sex or their interdisciplinary for the entire sex. Here's the picture of this group. Mark Lebow, chair of the toxicology subcommittee. And they have made great strides in the last two years. And all of these slides were provided by the committee chairs. So thank you for that. Where SWGTOX stopped is where this subcommittee picked up. Their focus has been first on foundational documents, just like Richard has described. And then, taking existing SWGTOX documents and improving them to the point where they can be submitted to an STO. Here's a membership, Mark is the chair and he's here in the group, in the room. And here's the break up by employer classification. You can see mostly state and local practitioners but also researchers, federal practitioners, private, and then R&D. So here are some of the standards that they're working on at the moment at the toxicology subcommittee. Uncertainty of measurement. Standard for mass spectrometry data evaluation. Minimum testing requirements. Identification criteria. And breath alcohol measuring instrument specifications. Now they have selected to submit their documents to the AAFS standards board, the ASB. And you can read the qualifications of this standards board. Here are some of the documents from, excuse me, the tox subcommittee that have been submitted to the ASB. And currently In the process of being approved. Practices for measurement traceability, practices for method validation, practices for quality control program. So here are the minimum requirements for establishing these areas within the discipline. Here's some additional documents that have been forwarded. Reporting, guidelines and opinions and testimony, and the standard breath alcohol measuring instrument calibration. So we've got six documents, right? That have been forwarded to ASB. Now this is the entire OSAC process. You may have seen this this morning, but I'm gonna show you again. This is Mark's slide. Where need is identified and a task group is created and that task group works to develop a draft document. Which is then presented to the entire subcommittee. I'm not gonna go through this, but you can see that this takes time. Eventually it gets to the SDO, now you've lost control over it. Now some other body Is gonna opine us to the suitability and approve it. That SDO in the case of toxicology is ASB. And in the other subcommittees within chemistry SAC it's ASTM. That SDO may provide recommendations for changes. If that's the case, then it has to go back through the whole process again. It gets changed within the subcommittee and it goes through the whole process. Richard mentioned that there are people outside of the OSAC that may want to contribute. That's fine. We welcome that interest. And there may be documents, it's possible, that there may be documents that are created outside the OSAC process, put through an SDO, and then the OSAC considers for taking it to the registry. But even after it's approved, by the SDO, it goes through a additional review process at the SAC level, the resource committees. Now we wanna get resource committee and SAC input as early as possible so Mark actually came up with this idea of pitch meetings. So you present the document very early in the process and seek input from the community, the entire community. The earlier you get input from the community the earlier you can incorporate that and maybe save some time later on. That's the idea. So, I think it's a great idea. We're incorporating this OSAC line At our next in-person meeting in March, we're going to have several pitch meetings within the chemistry sector. One of them is seized drugs, interpretation of DCMS data. So you can see that this process is, if you thought ASTM was slow and painful take a look at this. What about documents on the registry? Nothing yet, but be patient, they will come. There are other standards and guidelines that are being drafted by the OSAC Toxicology Committee. And you can see that these have not reached the SDO yet, and that they're not at ASB. But they're being worked on at different task groups and they're at different levels of maturity. And if you wanna go even further, every subcommittee has been asked to come up with a road map. What's the big picture? Where are we headed? And so the way that the roadmap has been provided by the top subcommittee is basically a list of these documents that eventually will be developed and put through to the SDO process. Some will, maybe some aren't appropriate for SDO, but I believe in the case of toxicology, most will. The tox subcommittee has also identified these research gaps. Emerging drugs of abuse and therapeutic agents, as well as herbal and dietary supplements and plant based toxins. So if you're a researcher and you want to devote some of your time and energy into researching in the toxicology area take a look at this. This has been determined to be important for the OSAC subcommittee toxicology, as well as human factors impacts, as well as post mortem distribution and redistribution. So these are listed on our website, I invite you to go take a look. Gunshot Residue Subcommittee, we have a new Chair, Rod Simmons from Wyoming State Crime Lab. We also have a new Vice Chair, Tom White. I think he's here, I saw him earlier. Emily Weber is the Executive Subcommittee Chair, Executive Secretary. And here's the makeup of the subcommittee. They have quite a number of affiliates. And their mission is to focus on the standards and guidelines related to analyses of evidence that results from deposition of or physical transfer of small or minute quantities of GSR. They have these task groups that are focused on competency and training, methodology and research, organic GSR, report writing and interpretation, testimony and ethics, validation, and performance. Here are their work products that are somewhere in the works within. They haven't identified where exactly. There's only one that I'm familiar with that is at an SDO. But training, there's a training task group that is working to develop a document. There is a report writing task group methodology test with all of these are going to be documents that eventually will make their way through SDO. Challenges that they've experienced? Well, the change in leadership, they lost both the chair and the vice chair. So they've had to make some adjustments in the way that they operate, and because of term limits, and term definitions. They also had additional changes in their subcommittee membership, so they're working their way through that. 1581 is their main document, and that's been around for a long time, and they are revisiting this document. And they've got it through now the SDO or they attend the SDO, and eventually it'll come back to the SAC for a vote. These are some lessons learned, provided by the Chair. Of course, good communications. One thing he identified was succession planning. So looking forward to who are gonna be the new leaders of the subcommittee, and who wants to be involved in this community. So if you're interested, please contact Rod. So the highest priorities are to well hold regular meetings, and revisit this 1588, which was just approved by ASTM. So there's a 17 designation to it. And I think that's what they're gonna focus on during their in-person meeting in March when we meet in Chicago. They've identified these research and development needs, OGSR persistence studies. Characterizing reference materials that can be used by the forensic user community. And then particle formation, so fundamental research in particle formation, and identification of shooters from GSR. So these are their research interests. Materials is a big group, because they have a charge of many different disciplines. Fiber analysis, paint analysis, glass analysis, other miscellaneous materials. So they have to cover many different types of measurements, as well as materials. And Diana Wright is a new Chair of this subcommittee. She replaced Sue Gross, who did an excellent job. And Diana has just picked right up where Sue left off, and I forget here but take and this is their charge. Here is the composition of their leadership, Chantelle Taylor and Kathleen Boone are on the Executive Committee. And they have these task groups set up by material, so fibers, glass, hair, paint, tape. There is a tas group on interpretation that I'll talk a little bit about later. There is a task group on outreach, and I'll talk a little bit about that later. There's a task group on research and then on terminology. Here's the rest of subcommittee members. It's quite a large group. I think they have 25 in total, and here is where they come from geographically. They're distributed throughout the entire country, and they also have a fair number of affiliates. So here's a road map, and this is an example road map. They have other road maps by my material. So here's fibers, there's a training document. There is a general fiber guideline that they will like to revise. There's an existing document that they're going to try to revise. And then from that general kind of an overview document, there is fabric. So standard guide for forensic examination of fabric, that's an ASTN document. Microscopy of fibers, textile fiber days, and ASTM document associated with that. An FTIR document of fibers is an existing document that is going through the ASTM process. Thin layer chromatography for dyes, textile dyes, microspectrophotometry, and paralysis DCMS. And then, there is a separate document that is an interpretation document. So this is what I was talking about when I first started off. This doesn't exist. This is an OSAC product that is new. And we didn't have this before when we were talking about ASTM. We were taking measurements, and we were doing a good job. But then what does it mean? What does this mean? This is the hard stuff. This is why it hasn't been done, cuz it's hard. But they're taking it on. So there is an interpretation report and to that end, there's some research calls and then there's some actions within the committee to do some studies. Surveys, they're taking charge of this. So, we do have an OSAC approved standard, and then we have. Four of them that are in the pipeline that are close. One was just voted on by the chemistry SAC to move to the FSSB for a vote in the very near future. And these other ones are very very close, they are in the process of either public comment or adjudication of the public comment. And then just before we get to the point where they will be voted on by the chemistry tech. And then offered to the FSSB for consideration. Additional documents that are either existing or new that are in the SDO process are listed here. Some of them have been around for a while, but some are new work products of ASTM such as training for examination, forensic examination of human hair. And there are others that are being revised that have been existing documents. And then there are new documents that are being drafted that are new to the discipline that have been be created within the OSAC. So when we ask the subcommittees to provide, what's high priority for you? Interpretation is at the top of the list. So they are working hard to develop a brand new document that is focused on interpretation. And, if you're interested to participate, there is Andrea. You can email her. And if you have ideas on how you do this in your own laboratory, please contact Andrea. So there is a validation study that's under way, initiated by the subcommittee, and Andria is organizing this validation studying. And they're using an interpretation scale. And they wanna see how does this perform? It's not exactly a blind box study, but it is taking into account human factors, issues. There is a human factors representative under subcommittee that is helping them to design the validation study. So another high level priority is what about perception and reality of trace evidence? One of the concerns is that trace evidence is taking a backseat to other disciplines like DNA and others. And this effort is to raise awareness about the value of trace evidence and also to gauge the level of interest, primarily to raise awareness about the value of trace evidence in the forensic science community. Sandy Parent is taking the lead on this outreach initiative. So additionally, they have identified these research needs. They are in the process of putting together a list. They also have some additional resources, some documents that they want to post on their website. Daubert packages, court rulings, articles that may be useful to you if you're a practitioner or researcher, that's in the course of your work. Additional documents that they're interested in are listed here. And their main thing is statistical data interpretation. Evaluates statistical data interpretation and validate whatever it is comes out the other end What are the challenges? Well they have many documents that are coming up for five year review with NTSTM. So they need to be reviewed and that takes a lot of effort. And another challenge that was identified by Diana was that we keep changing the goal posts. We keep changing the rules, we keep changing the way to do things. And that can be frustrating and, We also have different documents or different levels of maturity at different phases. So one of the things that they're still waiting on are some of the foundational documents that Richard was talking about. But again, the bottom line here is evaluating statistical data interpretation and what they have realized is that this is not a one size fits all. They're one of the groups that may not be entirely happy with Austin's source identification that he was talking about, that not everybody is on board. So that's, Fire Debris and Explosives subcommittee, Sue Hetzel, is the new chair. Here's the rest of the leadership for fire, debris and explosives. Here's the membership. They also had a little bit of turnover. And they have quite a number of international affiliate as well. They are tasked with guidelines and resources associated with examination analysis of fire and explosion investigations. Here are some of the documents in progress. A systematic approach to the analysis of identification of ignitable liquids. So they have taken really an honest view of the existing ASTM documents and say are these adequate? And made some decisions about, you know, starting from scratch on some of these things. This is basically their roadmap. They've decided that this is important. They have sent some documents to the SDO, which is ASTM. Terminology for both the fire debris and explosives. They've revised some existing standards and there is a new version of sampling of head space of a personal fire debris. And there is a new one on using absorbent tubes. And here is some documents that are in the registry approval process right now that some of these have been around for awhile. Here are the task groups that have been identified. Their main challenge, their main objective is really to blow up 1618. When I mean blow up, I mean to look at it very carefully. And they have written the position statement regarding 1618, which is their main document for the analysis of fire debris evidence. And they are going to concentrate their efforts on instrumental analysis, ILR classification, interpretation of GC-MS data. This is common to many of the chemists who are working with GC-MS EI data, like the seized drug people. So their pitch meeting will be held together. Fire debris and controlled substances, and seized drugs. And then report writing, so this is taking really a fresh look at a very good document that you should be using, because it's a good document but making it even better. And this is all because of the OSAC process and the conversations that have been taking place. There are other challenges that have been identified by Sue. Some of the practical things, there's just no time. These are volunteers. Every OSAC member who's working on these task groups, they're all volunteer members, so they have other jobs. And so that's a challenge. Increasing the number of in-person meetings really helps. This year we'll have two in-person meetings, and a lot work it's done at those in-person meetings. But a lot of work also gets done outside. So this back and forth between OSAC and ASTM has proven to be frustrating, especially for the fire debris people. The lack of published foundational research and validation studies has also been a slowing things down. And that's because research takes time. There is no money to do research within OSAC. That's up to the funding agencies to recognize the OSAC identified research needs, and then fund that research by in a competitive process, okay? On the positive side, I love this quote by Sue, there are some great scientists out there, and getting to work with so many is inspiring. So what about the path forward? There are some documents within this subcommittee that are approaching, submitting them to SDOs is also a very prolific subcommittee. And you can see they're very ambitious. And here are the highest priorities for them, post blast analysis, validation for fire debris methods, QA for explosives, Daubert packages for both disciplines, supporting terminology documents, training guides, and then report writing. What does it mean? This is another way of looking at the roadmap, and you can see a lot of R's still. Maybe not so much in the areas of evidence control and quality management, but in the analysis instrument and method performance, there still needs further research. Here's an analytical roadmap, it needs further research. And of course, the interpretation. This is why I said this is the most interesting and challenging task that we have before us. Intact explosives, they have a guide, they have a roadmap. Some of which there are existing ASTM documents, very few checks for some of these. And then, they've identified these research and development needs. So you can go on the website. New standards, and these are physical standards, reference materials, not documentary standards. Evaluation and comparison of different adsorption, elution methodology. Thresholds, there's a debate on should we set a threshold for finding an ignitable liquid residue. Should we artificially set below this or is that not really there. There's a debate about that, so there's some research that needs to be done. Source attribution, can we really say this is from one manufacturer, one distributor or another. Explosives related, you can read the research needs that are identified. And here's a picture of the group. And then finally, Geological Materials is kind of the new subcommittee. Here's their picture. Andy Bowen from the US Postal Inspection Service is the Chair. Here's a composition of the subcommittees, and they have affiliates even from outside the United States. The community of practitioners in geological materials is much smaller than,say, in seized drugs and materials. So they have a more difficult time identifying people to join their subcommittee, but they're having some success lately. Here are the areas focus for the geological materials, field collection of soil, geo-sourcing of materials, and then the analysis, the analytical methods used. So they have task groups, associated with microscopy, elemental analysis, X-ray diffraction, fractionation, color analysis, and also terminology, education training, and research. Some of the challenges that they face is that, they have zero SDO documents to work with. So they have to develop all of them from scratch. So that's why it's taking them a little bit longer, and there are relatively few practitioners in this discipline so the pool is limited. And so they need a lot of in-person communication because everything is so new to them. So the lessons learned is to make the most of their in-person meetings. And then also, take advantage of the power of the OSAC, which involves other chemists, other scientists, and as well as the human factors people, the statistician, etc. So they have a priority, high, medium, or these documents that are not yet been submitted to an SDO. But they're in the process of creating documents to submit to them. And here's their roadmap, which is quite ambitious for such a young subcommittee. They have also identified these research and development needs. Determined whether it is possible to anticipate scales of compositional heterogeneity of surface soils. Investigate the effects of different variables, relative abundance and development and evaluation of quantitative methods for soil analysis. That brings me to the end of the presentation. I thank you for your time, and I welcome any questions from anybody in the audience. >> [APPLAUSE] >> No questions? >> You mentioned the pressing need for research in this area, and of course it's across the board. And I don't know if it's a question to you or to Mark. What efforts are being made at the federal level, which has not been overly generous recently in scientific research, to try to get more funding for these areas that are needed to complete these missions? >> I'll take a stab, and then if anybody else wants to add to that. Actually, I think that there is research funds available through several federal agencies like the National Institute of Justice. NIST also funds research in forensic science and maybe somebody from NIST can expand on that. But now the National Science Foundation is also funding forensic science research in a concerted, deliberate, intentional way. And that's the first time, and this is just a recent thing. So I think I believe that there are research funds available. And just a matter of connecting those people who are capable of doing the research, who are the best at doing their research with the proper problems, and making sure that those people are funded. So I dont know that the availability of funds is the problem I think that there is money out there. Mark, do you wanna add to that? >> Just in observation that we have a new administration, and they focus on funding science across the executive branch is still under question, and we have different opinions from the administration, from Congress And we don't have a budget for 18 yet and we just got a proposal for 19. So I think your concern about where the money is coming from and if it's going to be continued and where it's gonna be focused are wonderful questions for which we don't have an answer. Unless Dr. Kavanaugh wants to speak to that issue from a policy question at NIST. He's not standing or waving his arms, so I'm gathering that he's leaving it to us. But I'd also like to say that one of the responsibilities of OSAC, listed in our charter, is to inform the forensic community of research gaps. >> And more than 90 research gaps have been identified. We will continue to do that. But I also think that it's obvious that there is a lot more identification of critical research gaps that will continue to come out from OSAC as it should than the amount of money which is available to fund those. So in answer to your question, ultimately it's going to require prioritization. And I'd like to say that OSAC will set those priorities for the federal government. We don't have that luxury. But we work hand in hand with one of the largest providers of grants, and that is NIJ. And we sit at the table with NIJ on ongoing dialogue. And they're aware of these publications. In fact, they're channeled to these publications. And in the solicitations coming from NIJ, they actually give the website for our list of research. And they give at least psychologically bonus points for raising to their attention the socilitation responses that incorporate the gaps that we've identified on our website. So we've set the table, we're doing everything we can within our power of influence. And [COUGH] the only thing I can report to you are the facts. In terms of where the policy lies with this, I can't tell you. But I don't think that anyone who claims that they know the answer to your question is going to be able to provide information today. We'll just have to see how it evolves. Rich. >> So I'd, I just reconfirm what both people just answered the questions. I think those are complete and accurate answers. The involvemnet of NSF, the NIJ and NDS. This as a sort of where current funds are coming from I think you're watching the federal budget process run its course. And I think, people actually now make the decisions about how much money's gonna get spent in any given area. And the federal budget are your representatives and congresspeople make those decisions. I think the president's interests have been expressed and captured in the documents, he sent to the Hill. And now at least those who are in the federal employment side are waiting to hear what the House and the Senate decide should be distribution >> I offer no official capacity here, but based on my observations, I've just written a book on process. You might just say the process of research, and I had the pleasure or whatever, had the duty of doing the chapter on funding. And yes the NIJ is normally the major sponsor for forensic research. Yes NSF is starting to look at us a lot more. And they sponsored the work I did. Both NIJ and NSF sponsored the work that I did on this book for this workshop I did. But what I wanted to do when I got the opportunity to write this little chapter is to see what the overall picture is. And I reported on the, I believe it was the, 2016 budget, let's just say that sounds about right. And I looked at the overall federal budget. It had been listed in Science Magazine. And it was a good budget. It was during the Obama years, and that administration was very favorable toward science. And they had the best scientific budget in a long time. A really good one. I'm trying to remember the figures now. It was something like It is 3 billion total. That's all the scientists. NIH got about 32 billion, so over two thirds of that. And it went down from there through NASA and all that. So I looked for poverty justice. It wasn't even listed. And I looked at the numbers from NIJ specifically, and I think at that year, that was 30 million. Now, I don't know how you look at dollars and cents for research, but from what we've been talking about, what we need to do here, it's a pittance, really, for what we need. And I didn't come here to become a bearer of bad news, but just to try to look at things realistically. What we're up against and, unfortunately, the administration as has been mentioned is not particularly favorable towards the sciences. So I'm very discouraged at where we can go because I think that there's a great deal, I think we're doing very well. I think, since 2009, I've seen a total turnaround on attitudes for a lot of people. And I think it's very important that we have this research going on, but hopefully, somehow we can get out point through, and that's all I can say. And somehow we just kinda keep having to go at what, whatever, wherever we need to go to get there. Okay, that's basically what I wanted to say. Thank you. >> I think that's a good point, Ed, that what I meant to say, I didn't mean to say that there was enough research funds. There are research funds. There may not be enough. Another area I'm concerned with is providing funds for laboratories to implement these standards once they become registry approved. A laboratory may have to undergo significant changes, acquire instrumentation, train their people, change the way they do things. And that's gonna cost money. So it's not just identifying and funding research needs, but it's also, what happens when these standards become registry approved? And our laboratory has said, we don't have capacity to enforce, but accrediting agencies may in the future or the lab directors themselves might impose these on themselves, just like they did accreditation. Forensic labs it's volunteer, but It is deemed a good thing to do so they might decide we should really adopt these. Well, that's gonna cost money so we should be concerned about facilitating that implementation as well. Any other questions? >> [APPLAUSE] >> All right, well, I'd like to thank everybody for their attendance today and this is the conclusion of the 2018 Organization of Scientific Area Committees for Forensic Science activity update. We're online, and we are in communication with you. And if you have anything to ask of any of the speakers or of any folks in those areas please reach out to us. And these slides will be made available within the next week. Right, and you'll be able to get those at the OSAC website if missed. So thank you very much everybody and thank you for the speakers. >> [APPLAUSE]
write for me writing dissertation in 3 weeks order dissertation philosophique introduction pdf what goes into a dissertation introduction e marketing topics for dissertation mla format unpublished dissertation konstantin von notz dissertation help do my dissertation computer science journalism dissertation examples dissertation on employee engagement pdf muster vorwort dissertation definition comment corriger dissertation akteur netzwerk theorie dissertation help phrase daccroche composition regime totalitaire kosten lektorat dissertation outline thesis topics in human resource management write for me contents page dissertation dissertation title and research question bettina friedrich dissertation definition proquest dissertations and theses search engine dissertation questions for accounting osu dissertation format thesis michael dambier dissertation methode dissertation bac philo paul shirley iowa state statistics dissertations rguhs microbiology dissertation topics how hard is it to defend a dissertation how to write acknowledgements for dissertation dissertation results section introduction university of pittsburgh dissertation write for me dissertation services usa sample thesis on business management comment faire une bonne dissertation dhistoire anaesthetic management of aortic dissection dissertation sur le port de luniforme scolaire joshan dissertation write for me dissertation based on secondary research order competing risks dissertation exemple dintroduction a une dissertation dissertation explicative huis clos wikipedia how to use theoretical framework in thesis dissertation help companies write for me dissertation abstracts international a the humanities and social sciences order revising dissertation into book demystifying dissertation writing pdf documents structure dune dissertation en philo dissertation totalitarisme second guerre mondiale france dissertation format for law order dissertation proposal writing help cheap how to display interview results in dissertation topics dissertation athenes et rome dissertation student meaning dissertations on shared leadership model order dissertation writing discussion online dissertation utopie edexcel epq dissertation word count quality architecture dissertation help dissertations in one sentence describe dissertation lhumanisme et sa diffusion keep calm and finish dissertation real estate dissertation pdf dissertation on customer loyalty exemple de dissertation de la poesie thesis on gender discrimination dissertation based on qualitative research dissertation acknowledgement page dissertation data collection structure how to write a data analysis chapter for dissertation what is a doctoral dissertation dissertation topics finance banking education order leadership dissertation cheap dissertation ideas on serial killers aiish dissertation proposal oxbridge essays dissertation defense writing questionnaire dissertation guerre totale 14 18 dissertation outline how to write conclusion dissertation how many chapters should your dissertation have phrase daccroche dissertation abstract dissertation ma bouche sera la bouche des malheurs dissertation verlag kosten 16000 word dissertation introduction dissertation philosophie politique dissertation the new jim crow laws dissertation topics in construction industry ulrike zetsche dissertation titles do my acknowledgements dissertation cheap devin di dakta masicka dissertation oto faresi dissertation lsu graduate dissertation fellowship dissertation on urban regeneration ch beck verlag dissertation 20 dissertations sur le plaisir de skier mbbs 1st year results 2013 rguhs dissertation dissertation on documentary brian letort dissertation humour noir dissertation proposal example master degree dissertation download free dissertation writing software dissertation table of contents word format thoracic surgery aortic dissection dissertation sur la femme dans le roman where can i get my dissertation bound in edinburgh buy dissertation in a week online dissertation proposal topics management style before you hand in your dissertation architecture thesis examples pdf le transhumanism dissertation sample master thesis database maastricht university cheap writing gliederung dissertation einleitung writing your dissertation derek swetnam pdf creator case presentation of aortic dissection cours sur la dissertation francaise en pdf legal dissertation format write for me computer science dissertation cheap how to conduct research for a dissertation do my dba dissertation order modern concepts on thesis and dissertation writing chapter 4 thesis summary conclusion and recommendation lars nozar dissertation meaning plan dissertation science politique contents page dissertation defense lse ei past dissertations how to write a dissertation in mathematics le bipartisme dissertation sample do my dissertation editing services order dissertation sur la raison et la superstition temple university dissertation defense information about dissertation write for me introduction de dissertation order dissertation these antithese synthese conclusion ucb dissertation layout dissertation philosophie penser cest dire non analysing questionnaires dissertation sample defense dissertation diabetes ppt dissertation sur le terrorisme islamique online mba without dissertation how to write a dissertation glossary small definition of dissertation surviving your dissertation a comprehensive guide to content and process 4th edition dissertation citation harvard dissertation typing services uk distance learning dissertations 2018 example of hypothesis in dissertation arno antlitz dissertations research dissertation guide dissertation writing services illegal order acknowledgement page dissertation example pdf lmu dissertation guide introduction en philosophie dissertations michael licona dissertation uni bonn dissertation online data mining thesis topics order mba dissertation assistance buy qualitative dissertation help cheap poorly written dissertation abstracts jochen lawrenz dissertation abstract write for me masters dissertation writing services for money literature review structure dissertation sample do my dissertation help belfast cheap extinction de laction publique dissertation topics www etudes litteraires com dissertation php tutorial exemple dissertation francais stg hedge zeijlstra dissertation help andreas moodler dissertation topics dissertation adviser for money buy a phd dissertation online buy dissertation binding service near me online dissertation tu berlin dissertation litteraire explicative exemple student does dissertation in 3 days definition of prolific dissertation psychology dissertation writing guide comment reussir la dissertation en philosophie trouver une phrase daccroche daccroche dissertation public health dissertation funding a dissertation on the canon and feudal law 1765 writing dissertation introduction online write for me dissertation writing hints online verzeichnis dissertationen online game 20 dissertations la guerre pdf to excel dissertation abstract definition for science dissertation gratuite sur la guerre du cp4101 b comp dissertation buy business management dissertation for money occupational psychology dissertation topics purdue owl apa dissertation uk university dissertation how to write a good thesis dissertation title dissertation topics in ayurveda dissertation poetry slam good dissertation skills ma history dissertation examples for educational leadership ronny thomale dissertation dissertation proposal example methodology in research dissertationes mathematicae rozprawy matematyczne buy dissertation proposal research methodology music branding dissertation higher education dissertation titles manchester metropolitan university dissertation binding application de larticle 55 de la constitution dissertation dissertation les mots de transition lisa marcinowski dissertation do my uk dissertation writing for money abstract dissertation guidelines william babnes poems of rural life in the dorset dialect with a dissertation and glossary write for me academic dissertation writing m phil computer science thesis free download pdf how to write discussion chapter dissertation referencing a dissertation in apa 6th edition do my introduction of dissertation order dissertation corrige la guerre cpge dissertation topics on depression dissertation quantitative data analysis example texaco chamoiseau dissertation proposal example thesis versus dissertation difference word 2013 dissertation template purdue university theses and dissertations communication design dissertation dissertations on ethical leadership in schools robert vrban dissertation format dissertation in tagalog dissertation culture generale et expression bettge susanne dissertation abstract postgraduate dissertation proposal sample order table of contents for dissertation cheap le roi carolingian dissertation meaning key components of a dissertation aauw dissertation fellowship 2018 corvette business dissertation title ideas for stories whats the difference between aortic dissection and aneurysm how to cite a website in dissertation writing a good dissertation conclusion oguzhan alagoz dissertation titles le retrait et labrogation des actes administratifs unilateraux dissertation dissertation contrat de mandat university of nottingham phd thesis guidelines thomas sauermann dissertations gender thesis ideas ideas for nursing dissertation knowledge management concept paper for dissertation dissertation sur le bonheur stg jura dissertation latex gi dissertations preis 2008 podiatry dissertation examples marketing legalisation of cannabis uk essay dissertations what should go in the discussion section of a dissertation difference between assignment and dissertation dissertation uk online surgery for type a aortic dissection shaker verlag dissertation defense bac de francais dissertation sur le theatre do my empirical dissertation online sociology of health dissertation woodrow wilson dissertation fellowship owl purdue thesis proposal sujet de dissertation philosophie travail dissertation defense presentation tips for students write for me project management dissertation order constructing a good dissertation by erik hofstee pdf dissertation proposal social work vorlage projektskizze dissertation proposal doing a literature based dissertation help dissertation core elements apa format unpublished dissertation dissertation proposal poster slogans proquest dissertations and theses global dissertation today eu competition law dissertation 3 minute thesis history failed dissertation proposal william johnson dissertation un of memphis dissertation plato theaetetus type 1 aortic dissection prognosis dissertation abstract format sample monographische dissertation titles write for me literature review dissertation online quantitative research methodology dissertation examples phd dissertation chapter affection societatis dissertation format umi dissertations publishing 2012 presidential candidates lustige danksagungen dissertation questions to be asked in thesis defense psychology dissertation guide dissertation les competences de l union europeenne gift ideas for dissertation committee members adjunct faculty job satisfaction dissertation conclusion pour une dissertation buy dissertation writing fellowships cheap library based dissertation example dissertation topic ideas for psychology amorce dans la dissertation juridique signe ecg dissection aortique dissertation uni stuttgart chemie periodensystem verteidigung der dissertation proposal destination wedding dissertation michelle obama dissertation la dissertation au lycee dissertation songwriting credits dissertation proposal example topics dissertation philosophie exemple liberte stamp outline worksheet for dissertation dissertation using only secondary research techniques muhs online dissertation 10 key strategic points dissertation economics dissertation help write for me in this dissertation for money est on libre daimer dissertation definition phd dissertation without punctuation how to write a thesis in 1 month do my dissertation writing services org cheap dissertation on homelessness in america theoretical conceptual framework dissertation how to write a dissertation report pdf explication de texte dissertation proquest database of dissertations la mondialisation en fonctionnement dissertation abstract meaning of phd advisor writing questionnaires for dissertations on bullying order dissertation writing schedule template order unibas medizin dissertations example thesis topics in education dissertation sur la negritude pdf kfw preis dissertation dissertation markus clemenzas fort dissertation ethical considerations in counseling ghostwriter dissertation medizin ed leadership dissertation titles what happens if fail dissertation outline dissertation proposal dissertation topic statement example faut il craindre la mort dissertation abstracts types research methods dissertation meaning order uk dissertation services cheap dissertation context section dissertations topics in human resource management ann christine ehlis dissertation definition order dissertation writing process mit thesis cloud computing methodologie de dissertation en histoire dissertation biden innsbruck tourism exeter uni dissertation binding dissertation express ill tetyana galatyuk dissertation proposal lektorat dissertation proposal example teacher efficacy dissertation kcl msc dissertation essay dissertation lart est inutile dissertation abstracts umi dissertation express what to include in dissertation discussion laudition libre dissertation advanced higher history dissertation topics uc berkeley dissertation guide plan dialectique en dissertation tu berlin bibliothek dissertation writing exemple de dissertation sur le commerce international comment faire une citation dans dissertation defense dissertation philosophie je pense donc je suis masters dissertation structure examples dissertations and theses from start to finish psychology and related fields le socialisme en allemagne depuis 1875 dissertation nyu steinhardt dissertation format amorce de dissertation sur le theatre et sa free english literature dissertation examples acknowledgements dissertation boyfriend and girlfriend thesis topics in marketing communication tobias mundry dissertation topics reads dissertation printing abdominal aortic dissection aneurysm regime presidentiel americain dissertation la technique de dissertation historique acheter une dissertation de philosophie acknowledging god in dissertation meaning general surgery dissertation topics buy dissertation contents page cheap mixed tenure sustainable communities dissertation abstracts ideas for tourism dissertations buy dissertation service in uk online comment faire lannonce du plan dans une dissertation outline le theatre dissertation exemple appointment of dissertation committee unm interim report dissertation dissertation example uk zip code methodologie composition histoire geographie terminale s dissertation writing conclusion online paper dissertation lhistorien et les memoires de la guerre dalgerie introduction order dissertation service public et police administrative cheap do my dissertation aims order master dissertation how many references should i list mba hr thesis pdf argosy university student guide to the dissertation process do my anthropology dissertation order what is a business dissertation photojournalism dissertation ideas in psychology digital theses and dissertations chinese dissertation albertine en cinq temps dissertation abstracts powerpoint for thesis proposal 20 dissertations le monde des passions party horntail maplestory proquest digital dissertations mots dintroduction pour dissertation sample dissertation topics for phd duke graduate school dissertation deadlines modele dissertation explicative ann temkin dissertation meaning rechtsvergleichende dissertation gliederung dramenanalyse stuart poyntz dissertation help methode dissertation generale dissertation titles development studies dissertation ideas for ict masters education dissertation ideas for hr best thesis topics in education comment formuler une problematique dans une dissertation these antithese synthese dissertation francais case study thesis structure dissertation explicative sur le horla legal dissertation example types of dissertation questions writing a phd research proposal social sciences dissertationen lmu chemie uetikon impact career technical students organization on dissertations theses andrew phillips dissertation apa dissertation reference list vocabulaire dissertation pdf write results analysis dissertation dissertations from university of phoenix steffen elstner dissertation format dissertation meaning and sentence dissertation sur lapparence what should be in the first chapter of a dissertation hardcover dissertation binding exemple plan dissertation francais dissertation research for money dissertation gratuite compte email order help with dissertations cheap beck verlag veroffentlichung dissertation how long does a college thesis have to be methodologie dissertation philo terminale l thesis front pages non phd doctorates without dissertations comment faire dissertation philo formulating research questions dissertation definition intervention de letat dans leconomie dissertation erik norberg dissertation socialisme en allemagne dissertation sample exemple dintroduction de dissertation sur la mondialisation chicago style cite dissertation ascetoholix dissertation dr gregor gysi speech plan de dissertation francais gs9k dissertation dont know what to do my dissertation on information science dissertations sample qualitative dissertations research structure dissertation guidelines on how to write a dissertation amorce dissertation argumentation vs persuasion dissertation bac es 2016 dissertation innovation portraits research design example for dissertation write for me writing my dissertation for money nation of islam dissertation type a aortic dissection duncker hublot dissertation writing sample phd proposal english literature 2018 literature review example dissertation pdf editor do my dissertation writing boot camp online dissertation printing reading standard dissertation format uk phone e commerce thesis pdf sujet de dissertation francais seconde mi university microfilms international dissertation volker quaschning dissertation ohiolink electronic dissertation writing an applied linguistics thesis or dissertation a guide to presenting business administration master thesis topics dissertation thesis proposal discussion section dissertation examples defended phd dissertation proposal examples london college of fashion dissertations dissertation printing binding manchester dissertation sujet de type 3 janina demeler dissertation abstract que mettre dans la synthese dune dissertation write for me dissertation experts cheap art dissertation title examples of a thesis vietinghoff dissertation outline convaincre et persuader dissertation abstract methodologie composition histoire geographie terminale es dissertation philosophie le pouvoir des mots how to start a dissertation methodology social media and marketing dissertation exemple dissertation mondialisation esperance de vie apres dissection aortique dissertation sur le droit objectif buy dissertation writing services uk reviews for money buy dissertation on writing for money write for me research methodology dissertation cheap apls dissertation awards do my master dissertation order dissertation topics on service quality vak gov ru dissertation do my writing dissertation youtube online dissertation letter credit arbede ecel 2019 dissertation dissertation coach dr keenan research paper 5 chapters of dissertation write for me dissertation design online plagiarism dissertation labelling figures in dissertation defense picture of type a aortic dissection buy doctoral dissertation writing help for money lamwai m phd dissertation do my writing dissertation over summer cheap kosten lektorat dissertation format guidelines for writing a thesis or dissertation wendell kimber dissertation titles buy dissertation writers order summer dissertation fellowship dissertation title page university of ulster magee thesis defense presentation ppt template geography dissertation proposal primary science dissertation ideas for psychology lascenseur social est en panne dissertation help exeter university history dissertation handbook dariusz jonczyk dissertation proposal sujet dissertation libertes publiques how to cross reference in a dissertation buy writing dissertation discussion chapter for money dissertation medizin lmusd phd dissertation acknowledgements example buy dissertation writing services oxford for money umass dissertation latex reussir dissertation bac francais phrases pour faire une dissertation write for me music dissertations dissertation topics on preschool education essay thesis dissertation dissertation printing binding leeds phd thesis topics in english literature in indian funny dissertation acknowledgments 2007 chevy do my hire dissertation writer uk for money dissertation en droit administratif malgache dissertation topics in international trade law definition how do you abbreviate all but dissertation melitensia special collection dissertations and the ses lektorat formatierung dissertation topics dissertation exemple de problematique foi et raison dissertation proposal dissertation student voice soziale kognition dissertation examples dissertation pourquoi se souvenir du pass write for me dissertation writing services legal cheap uk nursing dissertation topics buy methodology section dissertation for money dissertation retreat cheap informs dissertation award library based dissertation introduction elements dissertation erlangen nurnberg dissertation art engage forming a dissertation question symptomes de la dissection aortique malc millions dissertation difference entre note de synthese et dissertation la nullite en droit des societes dissertation medical statistics dissertation dissertation front page sample napier university dissertation guidelines policy dissertation johannes boltze dissertation format apa citation for dissertation dissertation on financial inclusion dissertation explicative therese raquin broadway dissertation define dictionary guide example education studies dissertation dissertation title page generator methodology chapter qualitative dissertation chapters dissertation csr banking beispiel inhaltsverzeichnis dissertation writing sujet de dissertation philo terminale s phrase daccroche heros daccroche dissertation dissertation theorie economique dissertationen online lmu prowl how many words per chapter in a dissertation physical geography dissertation methodology master thesis sample pdf a dissertation on the diaspora of black soul creation monetaire dissertation titles sample dissertation outline bits pilani next day dissertation binding dissertation literature review definition aauw american fellowship dissertation can i write a phd thesis in 6 months comment le travail favorise t il lintegration sociale dissertation how to write a thesis powerpoint presentation florian zickfeld dissertation proposal reussir la dissertation de philosophie dissertation chapter 1 introduction example graphic design dissertation layout dissertation research grants security uga graduate school dissertation announcement uk dissertation examples do my dissertation expert online dissertation sur le cout du travail buy dissertation introduction for money dissertation warwick english marketing degree dissertation memes order phd dissertation defense order literature review for dissertation uk nezz dissertation glasgow university dissertation binding order writing dissertation methodology order path to success write a doctoral dissertations msc nursing dissertation topics in medical surgical nursing owl dissertation dissertation presentation order lse management strategy dissertation rohan amin dissertation abstract dissertation defense cartoon sample powerpoint for dissertation defense le poujadisme dissertation proposal example dissertation table of contents format international marketing strategy dissertation do my dissertation of for money comment rediger une intro de dissertation mba thesis on employee motivation write for me how to write a dissertation cheap bim dissertation topics buy dissertation assistance service berliner wissenschaftsverlag dissertation format structure of postgraduate dissertation do my university microfilms dissertations online sujet de dissertation sur cyrano de bergerac uk dissertation acknowledgements location sample economics dissertation proposal council for european studies pre dissertation award personal reflection dissertation sample do my dissertation writing service usa cheap quantitative research dissertation example entrevista sonia sampayo dissertation introduction composition 1ere guerre mondiale dissertations for sale middlesex university phd management dissertations dissertation innovation foresight pdf dissertation to buy uk how to write acknowledgement for thesis samples how long does it take to write your thesis la conscience philosophie dissertation outline write for me help with dissertation writing cheap apa 6th edition dissertation margins dissertation sur la guerre de coree analysis of a dissertation upon roast pig shortest time to do a dissertation anne bochow dissertation help dieu est mort dissertation errata dissertation example proposal mentally disordered offenders dissertation writing define phd dissertation sample ubi societas ibi jus dissertation abstracts berkeley grad division dissertation guide michele oh land dissertations dissertation topics in blood banking masters dissertation proposal ukm tu berlin bibliothek dissertation help jabref entry type dissertation abstract buy dissertation education dissertation rub bibliothek dortmund dissertation qualitative online apa format for psychology dissertation phenomenology dissertation examples for educational leadership dissertation francais seconde sujet introduction examples for thesis papers business dissertation questions quelle dissertation examples sujet de dissertation sur la guerre froide methode de la dissertation eco dissertation help services in usa buy writing the doctoral dissertation cheap order manual for writers of term papers theses and dissertations dissertation nicole genschina dissertation proposal examples nursing portfolio write for me customer service dissertation questions for money research methods in dissertation ignou dissertation guide onze 20 se vc me dissertation hume dissertation sur les passions text edit download melitensia special collection dissertations online research design dissertation proposal lorenzaccio de musset dissertation defense thesis introduction how many words write for me online theses and dissertations online importance of dissertation project write for me customer service dissertation titles do my online theses and dissertations online methode de la dissertation en philo write for me dissertation help service uk dissertation proposal sample finance thesis unc dissertation library unc graduate school dissertation completion award writing dissertation questionnaire online engineering dissertation discussion step 3 ccs aortic dissection do my dissertation binding service online how to cite a published dissertation in apa format colonel chabert dissertation explicative msc accounting and finance thesis topics first class qualitative dissertation chicago manual style of writing bibliography for dissertation dissertation checking service online dissertation proposal mit background dissertation proposal university of hertfordshire dissertation binding dissertation topics on science education dissertationes mathematicae journalist research design dissertation example buy dissertation writing fellowships history cheap mikael lovkvist dissertations do my dissertation binding service liverpool cheap dissertation proposal template for money publishable quality dissertation meaning sample of phd proposal in international law write for me british dissertation help order how much percentage is dissertation worth banking topic for dissertation abdominal aortic aneurysm dissection radiology how many references in a masters dissertation ssrc dissertation development award law dissertation war crimes in iraq sample thesis statement apa format sample ppt on final dissertation seminar methodologie commentaire capes histoire buy dissertation proposal structure order accounting and financial management dissertation topics dissertation geographie le cacao en cote divoire le president de la republique dissertation proposal example do my dissertation service quality geological mapping dissertation sample dissertations on epistemology disertation on crm relative age effect dissertation proposal ap world history chapter 5 outline dissertation co authored dissertation proposal example scheideck dissertation abstract la romance du vin dissertation abstracts sujet de dissertation sur la famille belier conclusion and discussion in dissertation buy writing dissertation order using microsoft word 2019 mac for your dissertation dissertation business questions answered proquest dissertations and theses password change accroche dissertation ses croissance limpot est il juste dissertation defense teaching degree dissertation ideas ntruhs mds dissertation guidelines how long to complete phd dissertation do my dissertation writing ethics online chiraq gang dissertations how to write a philosophy dissertation proposal dissertation guide successful thesis dr angela howell dissertation at morgan les enjeux du vieillissement de la population francaise dissertation literature review example dissertation pdf file latex dissertation zitieren le personnage de roman sujet de dissertation asking your help for my dissertation questionnaire results dissertation sample formale fehler dissertation titles dissertation gratuite droit constitutionnel francais ed d dissertation titles dissertation sur la motivation scolaire ruth williams dissertation meaning writing research paper apa format lois constitutionnelles de 1875 dissertation titles friedrich balke dissertation sample accounting dissertation topics pavan kumar hanumolu dissertation abstracts order dissertation writing services usa online overview of the current knowledge on etiology natural history and treatment of aortic dissection nils hellrung dissertation outline dissertation help online professional dissertation editors cheap advanced higher dissertation write for me dissertation mba order dissertation chairperson pay les formes de la dissertation korrektorat lektorat dissertation topics mental health nursing dissertation titles dissertation topics in m commerce best books for dissertations how to write chapter 2 of a dissertation shende dissertation help dr lisa lang dissertation methode commentaire de texte philo bac s linterpretation de la loi penale par le juge dissertation social work dissertation examples write for me dissertation abstracts online order how to format a thesis in word 2019 wissenschaftliche abhandlung dissertation definition christopher travis dissertation bettina love dissertation abstracts method section of psychology dissertation do my purchase dissertation online order write for me dissertation writing consultants for money rural geography dissertation dissertation fellowship humanities la revision de la constitution dissertation order writing dissertation last minute for money m ed dissertations in india dissertationen tu darmstadt 2013 dissertation robert vrban dissertation on air pollution surviving the dissertation process corrige dissertation sur le bonheur discuss the format of thesis and dissertation quality assurance in construction dissertation lamorce dissertation dance in education dissertation examples dissertation abstracts international b the sciences and engineering order order professional dissertation writers uk online methode dissertation bts concluding remarks dissertation meaning methodology dissertation uk aortic dissection blood pressure medication aauw international dissertation fellowship pierre mendes france dissertation titles nick sousanis dissertation definition dissertation binding university of ulster methode dissertation bac ses citing dissertation abstract apa dissertation philosophie le travail caroline wootton proquest digital dissertations order top dissertation help where does the dedication go in a dissertation weird dissertation titles sample methode dune dissertation de philosophie psychology dissertation poster university of southern california doctoral dissertations how to reference a focus group in a dissertation dissertationen finden technologies dissertation in in linguistics optimality outstanding reduction theory vowel ruptured proximal aortic dissection sujet de dissertation traites context study dissertation thesis ku leuven ingenieurswetenschappen dissertation ideas physical geography the structure of master thesis dissertation in ma education ub muenchen dissertationen can you change your dissertation supervisor electrical engineering phd dissertation bachelor of business administration dissertation topics structure of a 10000 word dissertation domestic abuse dissertation ideas how to write an evaluation dissertation do my quantitative research dissertation cheap claudia bresgen dissertation titles dissertation introduction and conclusion will anyone read my dissertation literature review dissertation pdf creator nils unger dissertation project report for m tech thesis john biggam dissertation pdf file dissertation plan dialectique pdf best dissertation questions write good dissertation proposal berno bahro dissertation proposal example finance dissertation structure health safety dissertation ideas photojournalism dissertation ideas autism sustainable design dissertation examples about business roy fieldings dissertation on rest dissertation acknowledgements template word 2019 dissertations buy dissertation company for money uk nursing dissertation examples topics for dissertation in hotel management dissertationen uni wien justified amorce du sujet dissertation dissertation en histoire sur les sources de lhistoire a dissertation on roast pig paraphrase mbbs 1st year results 2019 rguhs dissertation dissertation advanced guestbook 2 3 1 sujet de dissertation francais sur le theatre acknowledgement letter for thesis report dissertation zitieren endnote style llm dissertation conclusion sujet de dissertation sur autrui ai dissertation juegos de juliana doctoral dissertations dissertation la gestation pour autrui order dissertation writing vocabulary fine arts dissertation topics proquest dissertations and theses 20111 do my best dissertation writers for money mesenchymale stammzellen dissertation format les droits de lhomme sont ils universels dissertation outline inventory management dissertation pdf dissertation data collection for money dissertation etat unitaire buy best place to buy a dissertation cheap sahnetorten dissertationen und politiken layout of a dissertation methodology marketing dissertation questionnaire a quoi sert lart dissertation home economics dissertation content analysis dissertation methodology section dissertation commentaire article 16 ddhc date comment faire pour rediger une dissertation aera dissertation award 2019 dodge presenting questionnaire results dissertation defense dissertation examples lsbu phrases pour une dissertation dissertation corrigee sur le terrorisme quality function deployment dissertation dissertation on multicultural education la raison philosophie dissertation format met1300 dissertation abstract shortest dissertation buy dissertation service public et contrat administratif online excerpta de dissertations in sacra theology of the body corporate finance dissertations dissertation tartuffe acte 3 scene 2 ruhr uni bochum bibliothek dissertation meaning whats a dissertation order valuation report of dissertation what is a graduate dissertation dissertation topics in maritime law library based dissertation example with tables pleure pas germaine dissertation format buy dissertation editing services reviews order schulthess verlag dissertation vorlage bewerbung ua dissertation dissertation business examples of marketing correction dissertation francais bac 2018 totalitarisme nazi dissertation titles mass effect 3 dark energy dissertation proposal write history dissertation length introductions to dissertations examples order change management dissertation online korreferent dissertation abstracts marketing advertising dissertation ideas for education dissertation notion service public order dissertation report layout preisgeld steuerfrei dissertation abstract le beau et lutile dissertation proposal dissertation zitieren apa past msc dissertations examples dissertation on constitutional law la previsibilite du droit penal dissertation dissertation topics in clinical microbiology jobs dissertation easy topics full dissertation meaning buy doctor dissertation cheap mines ponts 2019 sujets de dissertation palankovski dissertation examples dissertation de culture generale par lexemple dissertation ses bac 2019 arab proquest umi dissertations publishing 2018 movies dissertation bootcamp binding dissertation aberdeen dissertation writers block online lart de la conversation dissertation help forschungsvorhaben dissertation meaning how to write a dedication page for dissertation la dissertation philosophique introduction paragraph hr business partner model dissertation dissertation francais annonce du plan dissertation topics in business it que ce que la dissertation writing psychology dissertation dissertation stress quotes images type a aortic dissection uptodate dissertation topic in nutrition how long should chapter 1 of a dissertation be methodologie de dissertation economique pdf lhomme et son environnement dissertation serve on my dissertation committee members literature review doctoral dissertation chapters nij dissertation fellowship 2015 nfl dissertation sur les bienfaits du sport defended dissertation meaning in hindi assessment dissertation topics dissertation lsu lopposition parlementaire en france dissertation help dissertation cover cheap conclusion dissertation argumentation indirecte bac francais 2019 dissertations buy dissertation writing course cheap trouver une accroche dissertation juridique limitations of study dissertation defense subjectivisation de la cause dissertation dissertation on social media and branding write my phd dissertation proposal examples type b aortic dissection and anticoagulation dissertation topics for primary teaching thoracic aorta dissection icd 10 foto de estrada dissertation dissertation editing services rates do my nsf dissertation improvement grant for money euthanasia pour ou contre dissertation abstract matisses garden samantha friedman dissertation dissertation lallemagne pendant la guerre froide video dissertationen verzeichnis translation herbert utz verlag dissertation writing human resources thesis pdf write for me digital dissertation cheap masters dissertation introductions brian lack dissertation proposal example dissertation knowledge creation naoum 2019 dissertation abstracts dissertation explicative baudelaire orphans dissertation sur la philosophie et science shampoo dissertationsschrift englisch hilfen buy mba dissertation writing for money masters thesis examples pdf katharina beumelburg dissertation outline constitution de 1791 dissertation examples dissertation analyse de pratique cv dissertation committee dissertation uzh medizinfuchs sample apa research paper template ten strategic points of a dissertation order writing dissertation in 10 days order dissertation roman marx deejillz novelist dissertation nicole fitzner dissertation topics write for me hire dissertation writer uk order radiological features of aortic dissection isoletta dissertation warwick dissertation template dissertation topics in innovation management system mediatum dissertationen la terreur 1793 dissertation titles usf dissertation award do my dissertation statistics service for money phd dissertation proposal cheap dissertation topic ideas history of computer markus rauber dissertation abstract methode dissertation philosophie bac stg possible dissertation topics history do my dissertation writing oxford online des territoires inegalement integres a la mondialisation composition terminale s what should i write in thesis introduction declaration page of dissertation dissertation protection de lenvironnement gregor czisch dissertation definition dissertation on after sales service order wissenschaftliche abhandlung dissertation proposal example fiche methode dissertation histoire path to success game walkthrough doctoral dissertation definition labus de droit dissertation en des obligations generales architecture dissertation topics india dissertation credit and dscpt programs dissertation uni salzburg rechtswissenschaften hamburg exemple dune introduction de dissertation en francais order order dissertation methode commentaire de texte philosophie terminale s friedrich ebert foundation pre dissertation advanced graduate fellowships publishing industry dissertation quelques exemples de dissertation philosophique living with type b aortic dissection veera 10 feb 2018 in dissertation buy master dissertation online dissertation master phd thesis le nantissement du fonds de commerce dissertations buy writing a methodology for dissertation for money hypothesis in dissertation proposal order dissertation draft for money soghoian dissertation dissertation writing results section order ancient egypt dissertation carin de vries dissertation financial risk management dissertation topics corrige dissertation 2019 dissertation proposal sample uk visa modele de dissertation historique danksagung dissertation muster station dissertation on innovation and creativity sierke verlag dissertation meaning uk based dissertation writers in ghana do my literature based dissertation methodology for money philosophie dissertation introduction exemple cv labus de position dominante dissertation topics university of manchester dissertation workshop a dissertation upon roast pig wikipedia sample dissertation definition of terms what should be included in a dissertation literature review type b aortic dissection treatment uptodate dissertation sur leducation a lecole good titles dissertations britta jost dissertation do my electronic theses and dissertations online write for me automatic dissertation writer business management degree dissertation topics ucf dissertation proposal dissertation abstract order soas postgraduate dissertation results yoker dissertation dissertation editor interview questions dissertationen verzeichnis photo droit et morale dissertation plan dialectique dissertation on fashion management dr axel stoll dissertation format example dissertation social work autoethnography dissertations gratuites hilfe beim schreiben der dissertation book on how to write a dissertation comment faire lannonce du plan dans une dissertation titles arnolphe personnage comique dissertation help simeon schudy dissertation topics man met dissertation binding dissertation defintion order research dissertation questions dissertation bei emeritierter professor messer dissertation digital communication dissertation topic finder susanne gildehaus dissertation defense how to write discussion of findings in dissertation doktorarbeit medizin schreiben hilfe dissertation photographischer prozess reverse transkription dissertations good gift for phd advisor my dissertation on dissertations in accounting and finance le libraire dissertation proposal dissertation guidelines brown university law dissertation samples types de sujets de dissertation juridique dissertation consensualisme et formalisme phd dissertations uk ucl phd dissertation order engineering dissertation help for money onanisme dissertation topics do my phd dissertation help for money dissertation ruy blas drame romantique dissertation in education philippines dissertation chapters outline dissertation serions nous plus libres sans letat islamique en irak et au levant dissertation la justification des ententes socialisme en allemagne dissertation definition danksagung dissertation pflicht dissertation handbook ames dissertation definition synonymous dissertationen tu darmstadt 2013 chevy introduction de dissertation argumentative usc dissertations what should be in chapter 5 of a dissertation msc dissertation conclusion kcl dissertation german comment faire une dissertation sociologique introduction dissertation peut on se connaitre soi meme order dissertation service militaire cheap list of approved title dissertation dissertation glossary of terms order dissertation sur le service public en droit administratif order methodologie dissertation economie pdf lascenseur social est il en panne dissertation titles write for me history dissertation online credit risk management dissertation pdf free definition aims objectives dissertation michelle prytula dissertations conscient et inconscient dissertation examples buy dissertation writing service projectsdeal cheap dissertation stress meme army 899a bgb dissertation topics order dissertation on value for money for money write for me doctoral dissertations online order economic dissertation order dissertationen rube dissertation nature and purpose thesis on job performance pdf contents page psychology dissertation defense dissertation topic on corporate finance sebastian sonntag dissertation format dissertation year fellowship lsu writing a dissertation proposal ppt johannes wernz dissertation defense business dissertation research questions write for me cheap dissertation for money sqa rmps dissertation what to include in a dissertation rationale do my dissertation results section online dissertation rub bibliothek uni dissertation acknowledgements where to put hummingbird peut on assimiler le vivant a une machine dissertation de philosophie commentaire lorenzaccio acte 1 scene 1 what is all but dissertation dissertationspreis adlershof hot phd dissertation topic in water resources biology dissertation help for money do my motivation dissertation online faut il redouter la mort dissertation defense quellenangabe von dissertationen deutschland chapter 4 dissertation example dissertation gratuite droit constitutionnelle do my leadership dissertation cheap do my dissertation sections for money attachment dissertation ideas business abstract country developing dissertation dissertation topics in md anatomy oguzhan alagoz dissertation defense how to write the methodology section of a thesis le conseil constitutionnel et la constitution dissertation help philosophie dissertation introduction components dissertation topics on retail marketing strategy how to write discussion for dissertation joachim rojahn dissertations dissertation results online sujet de dissertation bac s 2018 buy uk dissertation writing services online dissertation history ict in teaching plan de dissertation sur la loi des 12 tables order dissertation writing services oxford cheap hayal dissertation master dissertation information technology how to write a dissertation executive summary maximilian heitmann dissertation format thesis and dissertation manual for sucs in region vi dissertation word count what is included citation analysis of education dissertations for collection development philosophie dissertation plan progressif writing your dissertation in fifteen minutes a day review masters dissertation length requirements dissection aortique traitement chirurgical le bonheur est il le souverain bien dissertation charles lamb a dissertation upon roast pig sparknotes great sample dissertation on customer relationship management dissertation layout wolverhampton wanderers buy writing dissertation education online dissertation dieter bubeck auto dissertation book of judges proquest dissertations and theses login gmail pierre mendes france dissertation definition legal dissertation questions thesis master degree sample history dissertation topics in zimbabwe undergraduate business dissertation examples methodology define thesis in research methodology composition histoire totalitarisme 1ere s phd thesis on learning organization dissertation prospectus lse portal jan philipp schmidt dissertation definition la france en villes dissertation help dissertation on self service technology for money gliederung juristische dissertation defense dissertation project on digital marketing dissertation francais apologues le conseil constitutionnel en france dissertation examples tobias debiel dissertation format write for me crm dissertation for money dissertation mechanical engineering pdf write questionnaire dissertation paris ville mondiale dissertation how to cite a textbook mla style generator write for me dissertation question cheap la conscience morale dissertation personliche danksagung dissertation amorce de dissertation sur le roman introduction dissertation sur le roman et ses personnages dissertation ne demek frans zca lse msc dissertation fail the ultimate guide to writing a dissertation in business studies a step by step assistance dissertation on organizational commitment how to plan an english literature dissertation order english dissertation online a quoi sert la justice constitutionnelle dissertation order dissertation mba cheap good gift for dissertation committee quantitative dissertation proposal outline dissertation titles in public administration dissertation sur lagriculture dans le monde exploratory case study dissertations uwe dissertation marks online thesis and dissertations online la crise de 1929 dissertation definition jack nasher dissertation proposal example lascenseur social est en panne dissertation format dissertation personnage de roman mediocre techniques de la dissertation cours referencing wikipedia dissertation tate cockrell doctoral dissertation order dissertation essay cheap dissertation topics in knowledge management order published dissertations for money best software to write a dissertation literature review dissertation abstract samples do my dissertation analysis help cheap a quoi sert le droit dissertation do my dissertation proofreading service uk toute oeuvre art est un beau mensonge dissertation durham history dissertation dissertation topics on rural marketing pdf dissertation ideas on childhood obesity dissertation photography ideas dissection aortique type a et b venus khoury ghata dissertation bold dissertation dissertation printing limerick vg wort dissertation 2019 ford methode dissertation bcpst master thesis topics in education pdf teaching english as a foreign language dissertation thesis topics ideas for human resource management sujet de dissertation juridique layout expose dissertation the art of stratifying patients with type b aortic dissection buy literature review for dissertation order buy dissertation writing services dubai online dzk dissertation do my writing your dissertation proposal dissertation topics on talent management titles of dissertations in english literature dissertation pourquoi le public va t il au theatre buy econometrics dissertation help cheap ma management dissertation topics afrikan boy dissertation examples university of nottingham dissertation printing introduction dissertation dialectique exemple lettre tourism dissertation topic dissertationspreis rechtswissenschaft studium faut il toujours avoir raison dissertation do my writing dissertation in 2 weeks accroche dissertation autrui dissertation coach uk sales project proposal dissertation dissertations examples nursing malpractice international business dissertation ideas in school buy dissertation research design buy british dissertation help reviews for money ideas for dissertation topics in anthropology penn state philosophy phd dissertation introduction de dissertation sur lamour dissertation nightmares writing your dissertation fifteen minutes dissertation site selection write for me buy dissertation for money satya sahoo dissertation dissertation proof reading for money sample dissertation case study stanley tsao dissertation proquest order psychology dissertation help order dissertation feedback lse dissertation on motivation of employees pdf editor aortic dissection znaczenie inaugural dissertation help lmu dissertation einreichen dissection aortique bentall secondary analysis dissertation example bac francais 2018 dissertation writing do my bsc dissertation for money dissertation topics on vaccines write for me dissertation for masters degree cheap covering letter for a questionnaire for a dissertation typing dissertation cost law essay thesis aortic dissection radiology stanford thesis and dissertation writing without anguish online gi dissertationspreis 2012 dissertationen uni leipzig tool do my phd dissertation assistance online buy uk dissertation writing citation pour dissertation droit dissertation prospectus vs proposal definition thesis grammar checker free 20 dissertations le plaisir dapprendre phd thesis on water and sanitation ucf graduate studies thesis and dissertation manual dexterity order professional dissertation help for money dissertation books amazon dissertation dhistoire exemple dintroduction gregor czisch dissertation abstracts do my dissertation typing service order art et verite dissertation sample ed d dissertations thesis and dissertation format path to success game walkthrough doctoral dissertation problematique dissertation philo dissertation sur le bonheur philosophie abbreviation page in thesis dissertation les etats unis depuis 1918 influenza library based dissertation methodology help write for me writing a dissertation introduction order what is the function of the dissertation proposal progress made in dissertation thesis binding manchester metropolitan university laryssa chomiak dissertation meaning radicalization dissertation sample ccdmd dissertation explicative acknowledgement template dissertation dissertation sample for mba writing doctoral dissertation uni marburg medizin dissertation defense order ma dissertations ballard and seashore dissertation fellowships write for me doctoral dissertation writing help for money disertacion doctoral dissertation sample dissertation thesis mba dissertation project finance jan mending dissertation writing buy can you write a dissertation in a month online brian lack dissertation examples health and safety in construction dissertation topics dissertation binding tulsa apa style citation unpublished dissertation on school bachelor thesis cloud computing dissertation la poesie doit elle celebrer la vie charles lamb a dissertation upon roast pig analysis quiz dissertation proposal sample psychology resume do my dissertation data analysis online theology dissertation proposal sample write for me dissertation preparation cheap dissertation pierre bourdieu sociology brands dissertation order doctorate dissertation lintroduction de la dissertation en philo names of dissertation konstantin von notz dissertation topics cover letters for dissertation questionnaires sample of thesis acknowledgement pages regent university dissertation handbook rguhs orthodontics thesis topics difference between thesis and dissertation ukulele dissertation lmu bookstore albert mohler dissertation excellent dissertations peter levin free download mellon dissertation research grant write for me film dissertation cheap do my shortest dissertation cheap architecture dissertation methodology dissertation help india review word technique de lentonnoir dissertation writing dissertation uva athletics problem specification dissertation proquest dissertations and theses 2011 gmc guerre totale 14 18 dissertation topics write for me research proposal for dissertation online 23 remix east dissertation tim hucho dissertation meaning procurement thesis pdf do my struggling with dissertation english lit dissertations lektorat dissertation abstracts buy dissertation art do my dissertation proposals order dissertation zur erlangung der doktorwurde dissertation proposal examples business proposals importance of dissertation structure dissertation les effets du divorce ux design dissertation etudes litteraires exemple dissertation juridique literary theory dissertation credit risk management in banks dissertation outline les villes au moyen age dissertations kai wawrzinek dissertation outline faut il croire au bonheur dissertation dissertation layout ucc shooting la france en villes dissertation titles business marketing dissertation ideas lherbe est toujours plus verte ailleurs dissertation defense cheap dissertation help for money art history dissertation fellowships for humanities diantha clark opera dissertation sujet dissertation bac ses 2017 ciep delf a2 sujets de dissertation how long should a dissertation prospectus be dissertation help tutors for money methodologie de la dissertation en philosophie en pdf sample introduction of a dissertation open access thesis and dissertation guide dissertation pages long doctoral dissertation fellowship uconn women lsu thesis and dissertation database attentats du 11 septembre dissertation s j sobolak dissertation dissertation on picture books buy dissertation writing fellowships humanities cheap worldcat dissertations literature review purpose of the study dissertation sample thesis acknowledgement and dedication exemple de dissertation philosophique to pdf path to success walkthrough doctoral dissertation help buy doctoral thesis dissertation order how to start undergraduate dissertation buy dissertation consulting do my mba dissertation writing services uk for money difference between dissertation and literature review ispim wiley dissertation awards dissertation topics services marketing seth yalcin dissertation exemple de plan dissertation philosophique dissertation wikipedia frankenstein business dissertation topics 2013 osborn dissertation roman buckow dissertation titles sami haddadin dissertation writing dissertation medizin uni heidelberg perrine chancerel dissertation format bibliothek rwth aachen dissertationen phd dissertation assistance dissertation verbs dissertation sur phedre pdf michelle priestley proquest digital dissertations sujet dissertation leuthanasie advanced higher english dissertation 2016 order dissertations abstract international online order dissertation writing retreat umn order funny dissertation acknowledgments 2007 chevrolet um rackham dissertation la thesis francais aau electronic library thesis and dissertation accomplished shweta reddy dissertation oregon state ascending aortic dissection flap lektorat dissertation ingenieur definition dissertation editing software cheap undergraduate dissertation chapters sample phd proposal in computer science pdf plan dissertation suis je responsable de ce que write for me dissertation writing gif order dissection aortique cours medecine cuhk thesis and dissertations analysis and interpretation of data sample thesis dissertation cpe 2006 and all but dissertation resume objective cdc grants research dissertation dissertations online australia ethical approval for dissertation buy custom dissertation services gmo dissertation write for me dissertations theses albert schliesser dissertation proposal master thesis topics management publikationspflicht dissertation abstracts adaeze akamigbo iowa dissertation wenner gren dissertation fieldwork application manager apa dissertation title page sample uc berkeley thesis database la philosophie et science dissertations thesis on financial risk management paulomi chakraborty dissertation la quatrieme republique dissertation topics personnages de roman dissertation order project dissertation for money order dissertation abstracts database mechanical engineering dissertation examples oliver baron dissertation proposal dissertation philo amour dissertation printing durham me structural engineering thesis topics do my dissertation question help order 2018 2018 dissertation fellowship dissertation coach uk travel help with data analysis for dissertation for money je pense donc suis descartes dissertation dissertation literature review how many sources of income criminals are born that way dissertation research design sample qualitative dissertations do my dissertation software online corrige dissertation philo 2018 acknowledgement ideas for dissertation do my dissertation proposal structure for money surgery for aortic dissection with intimal tear write for me purchase dissertation online order dissertation requirements cheap do vets have to do a dissertation philosophie dissertation sur le bonheur memphis dissertation de philosophie astuce dissertation proposal sample quantitative reasoning dissertation research definition rguhs dissertation topics 2019 calendar dissertations on online education writing a proposal for your dissertation do my lse dissertation help for money academic article dissertation 8 letters ideas for dissertation topics in fashion order dissertation research proposal cheap ntruhs thesis topics pediatrics dissertation topics on financial risk management domestic violence male victims dissertation sample lera boroditsky dissertation writing kcl dissertation help online apa referencing phd dissertation topics nursing dissertation help uk online buy dissertation essays order dissertation topics on solid waste management exemple dintroduction de dissertation juridique pdf free order masters dissertation writing services order la methodologie de dissertation en francais library dissertation in conservative dentistry and endodontics dissertation example on being a mentoring no et moi dissertation exemple annonce dun plan dans une dissertation thesis and dissertation website ucf advanced higher rmps dissertation examples brookes dissertation binding sacramento masters dissertation examples for educational leadership council anthropology education dissertation award dissertation portsmouth tourism management dissertation ideas dissertation augusten pg dissertation guidelines comment faire une dissertation en francais niveau seconde apa citation of unpublished masters thesis stanford hospital aortic dissection sujet de dissertation histoire du droit l1 structure of a phd dissertation logos verlag berlin dissertations review of metaphysics dissertation essay contest 2019 do my dissertation editing walden university dissertation samples cite thesis dissertation search interesting dissertation topics business my dissertation doesnt make sense directory of doctoral dissertations on sanskrit indian universities pay someone write dissertation order dissertation writing workshops order dissertation synopsis online samurai camara dissertation topics dissertationen uzh webmail newspaper analysis dissertation abstracts order creative dissertation dissertation francais 3 cegep write for me business dissertation help order dissertation sur les causes de la seconde guerre mondiale write for me uf thesis and dissertation order structure de la dissertation explicative petra liedl dissertation examples personal reflection dissertation example buy dissertation tense cheap undergraduate dissertation examples free charlie banacos dissertation format le gaullisme dissertation proposal example dissertationes botanicae 234567890 dissertation topics in kerala la romanization de gaule dissertation proposal methodologie de dissertation culture generale heinrich schliemann allison taylor dissertation dissertation editors in south africa dissertation means in nursing comment rediger une introduction de dissertation lhumanisme et la renaissance dissertation meaning comment faire une dissertation plan dissertation cover gestalten order writing dissertation in 2 days for money susan lepselter dissertation defense bsc dissertation proposals on performance management dissertation doing in politics dissertationsdatenbank uni leipzig medizin buy dissertation help writing cheap claudia preckel dissertation writing common thesis defense questions download criminology dissertation hate crime dissertation topics on cost accounting introduction de partie dissertation titles dissertations on freshman academies phrases conclusion dissertation dissertation nottingham trent doctoral dissertation meaning in tagalog dissertation gold investment dissertation audit quality university of missouri graduate school dissertation ccdmd dissertation dissertation suffrage universal 1848 large dissertation pour ou contre linternet de futur order need dissertation help for money sample dissertation proposal defense examples of dissertation rationales for imperialism johanna uotinen autoethnography dissertation uwe kamenz dissertation definition phd dissertation mathematics publikationspflicht dissertation meaning what is a synonym for the word thesis business plan dissertation pdf converter e dissertation mgr university anthropology dissertation online buy history dissertation proposal for money dissertation school readiness credit risk management in banks thesis fair green restaurant dissertation dissertation research proposal outline dissertation proposal examples business reports art der dissertation kumulativ cite dissertation in apa format union university dissertation handbook citation dissertation apa format social work dissertation examples uk dissertation topic list in rguhs mbbs dissertation literature review editor mehrdad mashayekhi dissertations award in educational christoph zulehner dissertation proposal example nicole becker dissertation examples write for me master dissertation confessions de rousseau dissertation nsf dissertation improvement grant anthropology dissertation complete sur la guerre froide huissier de justice dissertation i need help with my dissertation cheap voir est ce savoir dissertation program evaluation dissertation proposal dissertation schedule plan dissertation topics on trademark law umi dissertation order la dissertation philosophique terminal mac masters dissertation in absentia rochester dissertation projects done in san antonio tx self determination law dissertation research uk masters dissertation dissertation topics for mba hr buy econometrics dissertation help for money presentation of acute aortic dissection primary and secondary data dissertation the ultimate guide to writing a dissertation in business studies bonne dissertation de philo gang dissertation titles in finance what is the meaning of dissertation in english rachna jain dissertation definition what is a dissertation in research dissertation topics economics students mba dissertation proposal pdf example phd dissertation thesis and dissertation in a second language do my custom dissertation help order uk dissertation help for money dissertation site like shodhganga free cover page of phd dissertation law dissertation examples uk presenting questionnaire results dissertation examples roy fielding dissertation proposal example sabine kluwe dissertation sample yasutada sudo dissertation sample disseration online anna babel dissertation lse dissertation international relations northwestern tgs dissertation retrospektive studie dissertation meaning effects of tutoring dissertation auditing dissertation titles order working on dissertation online dissertation verlag naturwissenschaften accents dissertation personal reflection definition aha dissertations in progress writing dissertation in 2 weeks cheap dissertation binding manchester metropolitan buy how to write a good dissertation cheap dissertation le droit et la justice irina lokteva dissertation outline darla deardorff dissertation proposal example what to write your thesis on how many words is a standard dissertation acknowledgements for dissertation samples do my dissertation service innovation online dissertation alexander morlang movie how to collect data for a dissertation clinical presentation of aortic dissection cv sous forme de dissertation rackham dissertation defense questions do my research for dissertation for money comment faire une bonne dissertation philosophique pdf dissertation with only secondary research methods publication dissertation steuer doit on annoncer le plan dans une dissertation dark energy dissertation mass effect 3 endings quotes included word count dissertation dissertation sur la lecture introduction publizieren dissertation format dissertation defense order critical regionalism dissertation format how to defend your thesis presentation buy art dissertation online how to choose a phd thesis title federalisme dissertation dissertation sur la litterature maghrebine dexpression francaise components of a doctoral dissertation noten fur dissertation mc 72 bela dissertation do my philosophy dissertation order chilando try dissertation write for me dissertation marketing order write for me dissertation index uk dissertation examples free non exclusive license to reproduce the ses and dissertations how long does an undergraduate dissertation take leif hommers dissertation sample buy writing methodology for dissertation for money sujet dissertation agregation externe anglais 2019 art design dissertation proposal sample of acknowledgement in thesis paper dissertation topics in obstetrics and gynaecology dissertation meaning in punjabi mother international law dissertation examples free write for me dissertation writing handbook for money writing a literature review for your dissertation pour ou contre limmigration dissertation social work dissertations online order dissertation writing retreat umn for money urgence serveur dissection aortique exemple dissertation philosophie bac dissertation les principes generaux du droit administratif computer science dissertation bsc pjentak dissertation palgrave dissertation book anne patel dissertation outline dissertation volkswagen blues dissertations from karolinska timo mappes dissertation proposal example dissertation topics in education studies accroche dissertation apologies uga grad school thesis template jack london dissertation defense dissertation examples for accounting and finance acknowledgement thesis doctoral dissertation order dissertation word count breakdown cheap do my dissertation assistant cheap mercedes benz sign aortic dissection order dba dissertation online pourquoi parlons nous dissertation arbeitsplan expose dissertations english dissertations online dissertations on translation studies personal development plan for dissertation harvard style of writing dissertation literature examples abstracts dissertations dissertation topics for educational technology non empirical dissertation methodology chapter writing a dissertation checklist gestation pour autrui dissertation definition best powerpoint templates for thesis presentation do my uk dissertation writing services online final year project dissertation example free dissertation proposals amorce dans la dissertation psychopedagogique dissertation title on cv aide a la dissertation introduction houman sebghati dissertation flattening dissertation abstracts media thesis topics list honey bee dissertation do my dissertation on after sales service prof dr karl lauterbach dissertation topics good thesis presentation john f nash dissertation ancient history dissertation questions le diable amoureux cazotte dissertation abstracts dissertation juridique droit constitutionnel pdf creator lauterbach karl dissertation abstract tetyana galatyuk dissertation definition methodologie dissertation arts plastiques dissertation format typing dissertation research grant florida state university proquest dissertation advanced search dissertation writing introduction chapter online dhcp dns dissertation northumbria university thesis binding dissertation philosophique sur les passions de lame uni heidelberg dissertationen online calculator dissertation chapter 4 summary byod dissertation le conseil constitutionnel et la revision de la constitution dissertation intro dissertation mthode just defended dissertation examples buy methodology dissertation online dma dissertation topics relativisme culturelle dissertation outline salford university past dissertations dissertation ideas for sports nutrition sample dissertation about educational management guidelines for preparation of phd phil thesis dissertation buy dissertation help belfast online methodologie de la dissertation en francaise pdf methodologie dissertation exemple editors for doctoral dissertations dissertation help service uk samsung dissertations and theses full text proquest education faire une dissertation traduction en anglais m and s dissertation langoisse du roi salomon dissertation abstracts roman vilimek dissertation dissertation urdu meaning 3 minute thesis competition 2019 proquest dissertations and theses ubc map write for me dissertation service public et droit administratif dissertation auguste escoffier order writing dissertation the night before for money dissertation philosophy apa style for writing dissertation clinical psychology dissertation dissertation writing template dissertation francais le roman masters dissertation timeline example nima etminan dissertation abstracts politics dissertation topic ideas proposals that work a guide for planning dissertations and grant edaa outstanding dissertation award 2012 olympics rutgers university library dissertations online dissertation organization tools 2018 write for me final year dissertation order dissertation acknowledgements format for research write for me research methods dissertation cheap dissertation juridique sur letat unitaire et federal dissertation on hollywood horn players uk dissertation abstracts international section dissertation topics in paediatric nursing best dissertation instagram captions apa generator for dissertation fallait pas faire du droit dissertation defense ferri abolhassan dissertation action research dissertation proposal unoriginal dissertation proposal study design for dissertation dissertation organization cheap oxford format dissertation thesis project report sample literature based research dissertation chapter dissertation length of conclusion paragraph mba dissertation topics in hr dissertation publication verlag meaning of dissertation dictionary how to write a methodology for secondary research dissertation dissertation forum for mba citation of dissertation comment faire une conclusion de dissertation exemple master dissertation proposal sample uk will how many pages should a dissertation chapter be buy mla dissertation order peut on se faire justice soi meme corrige le conseil constitutionnel dissertation juridique exemple ideas for accounting and finance dissertation 12 20 dissertation titles dissertation writing manual cheap dissertation proposal timetable sample dissertation proposal examples nursing notes pr topics for dissertation introduction dune dissertation de droit constitutionnelle diskussion schreiben dissertation meaning information card about defended dissertation la contrainte et letat de necessite dissertation samples of dissertation research methodology masters thesis on cv dissertation ideas musical theatre dissertation today cheap le cumul des mandats dissertation titles library information science dissertations niu dissertation completion fellowship dissertation drucken essen do my sections of a dissertation for money good dissertation tips custom dissertation writing service cheap dissertation powerpoint on inpatient falls buy biology dissertation how to write theoretical framework in thesis pdf emedicine medscape com aortic dissection universitat zurich dissertation medizin thesis type crossword clue write for me creative dissertation dissertation apologue essais definition essay thesis examples prehospital treatment for aortic dissection ruth boehmer dissertation abstract ucl dissertation criteria dissertation on roast pig summary mail boxes etc dissertation binding best tourism thesis topics dissertation vs thesis pdf download sujet de dissertation traite pdf bernd althusmann dissertation examples do my dissertation homework help master dissertation uk example chapters to include in a dissertation significance of the study in thesis definition hedge zeijlstra dissertation defense can i publish my dissertation online the ses and dissertations write for me writing a dissertation proposal order sport economics dissertation topics marketing branding dissertation ideas accounting dissertation les sources orales leeds dissertation binding suis je seul au monde dissertation topics ses dissertation plan mba dissertation viva exemple explication de texte philosophie bac s quinquennat presidentiel dissertation outline la gestion privee du service public dissertation order dissertation consultation services cheap order dissertation marking service dissertation sur la television et la culture taux dinteret et investissement dissertation les ordonnances de larticle 38 dissertation format dissertation ionesco les chaises ketia dissertation explicative la chevelure dissertation grammar dissertation on music festivals teatro glauce dissertation dissertation services in uk for money divorce sans juge dissertation academic paper template consistent with apa style guidelines for dissertations good dissertation literature review outline dissertation of witches using published paper in dissertation dissertation questions on outdoor learning dissertation en premiere l buy nursing dissertation order dissertation ideas education leadership dissertation in a simple sentence dissertation in art and design buy dissertation writing service for money how to write results section in dissertation ogaps tamu dissertation fellowship never finished dissertation sample do my lse dissertation help order methode de la dissertation en litterature dissertation enm 2019 write for me dissertation data analysis dissertation sur l enfer c est les autres dissertationspreis giuliana umlagerungen dissertation format do my dissertation writers in hyderabad zeitplan dissertation literaturwissenschaft philipp misselwitz dissertation abstracts pe dissertation titles in psychology number of pages in a dissertation walden university dissertation prospectus mmu dissertation marking criteria type plan dissertation droit psychology dissertation ideas for money ucsb grad div dissertation abstracts la france un etat unitaire dcentralis dissertation literature dissertations order dissertation lmu psychologie clinique phenomenology dissertation examples for education abusus dissertation titles dissertationen rubiks cube berliner wissenschaftsverlag dissertation proposal winchester dissertation binding lascenseur social est il en panne dissertation topics dissertationspreis dpg performance lse dissertation template concluding remarks dissertation abstracts ma history dissertation examples free buy dissertation precis order methodologie dissertation juridique droit constitutionnel dissertation source du droit ict dissertation projects etre payspan au moyen age dissertation format apa style formatting dissertation topics for dissertation in computer science announcement of dissertation defense epq dissertation length requirements write for me dissertation on value for money online undergraduate dissertation format apa peer mentoring thesis pdf carl heymanns verlag dissertation sample richwine dissertation committee on resume duke ellington dissertation indiana university graduate school dissertation format xu cheng dissertation apa citation dissertation thesis example dissertation posters jurisprudence dissertation topics latex dissertation statt phd thesis format how long does it take to complete a dissertation write for me writing the doctoral dissertation dissertation icp oes methodology definition dissertation abstract methodology chapter qualitative dissertation dissertation topics for criminal law la dissection aortique ppt do my academic dissertation writing cheap do my ljmu dissertation help online dissertationen online tu darmstadt dissertation guidelines muhs dissertation la peche en cote divoire pdf introduction dissertation philosophie sur letat harvard thesis template latex open university phd dissertations in economics write for me dissertation methodology order business thesis proposal le premier ministre anglais dissertation abstract fellowships for dissertation research statement of ethics dissertation abstract article 16 du code civil dissertation meaning how to cite a dissertation asa rhinoceros analysis ionesco dissertation en classe de troisieme methode de dissertation en philosophie introduction central research question dissertation proposal buy dissertation writing google scholar order order dissertation presentation help order les droits de lhomme sont ils universels dissertation examples aau small grants for thesis and dissertation 2016 write for me writing your dissertation in fifteen minutes a day online write for me dissertation service public et droit de la concurrence order friedrich ebert stiftung stipendium dissertation abstract order dissertation writing cpsp cheap la conscience philosophie dissertation topics how to write a literature review dissertation uk dissertation wladimir klitschko anonymous interviews dissertation outline research question for dissertation online owl purdue mla thesis statement dissertation sur la sociologie des organisations thesis oral presentation format order writing introduction for dissertation cheap do my dissertation on marketing online dissertationen ethernet do my dba dissertation online buy methodology of dissertation do my dissertation planning application form for submission of dissertation ed d dissertation titles in education sujet de dissertation francais 3eme senegal dissertation zeitraum dissertation binding service near me cheap dissertation topics on local government kunstchronik dissertations mary beth morrison dissertation correction dissertation philo 2018 dissertation of david carawiay bibliographieren dissertation defense order introduction for dissertation online dissertation quest ce qu un bon debut de roman dissertations on leadership and integrity reworded thesis definition dissertation totalitarisme nazi dissertation topics dissertation example finance lse llm dissertation results primary physical education dissertation ideas for hr research proposal mba dissertation ouverture dissertation pib spud boom yayo dissertation proposal dissertation ppt debdeep jena phd dissertation format dissertation sur le romantisme seconde lamour est une illusion dissertation proposal do my data analysis dissertation marburg dissertation drucken argosy atlanta dissertation program reviews content of dissertation cheap ronal serpas dissertation format research validity dissertation ketose rind dissertation definition dissertation topics for mba operations research sujet de dissertation sur lelargissement du monde jan glasenapp dissertation titles kant inaugural dissertation failed dissertation uk kaemor dissertation aortic dissection echo video social science dissertation ideas order final dissertation table of contents master thesis sample phd dissertation in human resource management exemple intro dissertation economie dissertation littraire cid dissertation committee online gregor schiele dissertation human geography dissertation examples nursing buy aauw dissertation fellowship cheap dissertation report on economic growth zeitlicher arbeitsplan dissertation outline dissertation manuscript robert vrban dissertation sample order change management dissertation order dissertation sale write for me dissertation for dummies for money write for me how to structure a dissertation for money dissertation words dissertation lmu online bewerbung dissertation de philo aide la philosophie est elle une science exacte dissertation sample write for me dissertation writing services illegal for money dissertation topics radiography programs how to write a personal narrative thesis statement dissertation uni innsbruck hannah jopling dissertation definition ideas for fashion dissertation advanced higher history dissertation introduction template analecta biblica dissertationes mathematicae science dissertation titles buy writing dissertation aims and objectives druckkostenzuschuss dissertation bundesministerium dissertation pdf download apa research paper outline template pdf form dissertation harvard sample dissertation chapter 2 dissertation effects in linguistics nasalization neutral opacity outstanding segment dissertation defense powerpoint examples of poor thesis examples for a compare and contrast essay dissertation university of leicester dissertation research proposal sample abady dissertation three minute thesis competition uk dissertation fellowships 2019 17 mlb philosopher cest douter dissertation writing a dissertation uea apa style research paper format pdf undergraduate dissertation examples correction dissertation francais 2019 unique topics for thesis in psychology ict in education thesis topics published dissertations uk daily mail la justice et le droit philosophie dissertation doctoral dissertation po polsku sujet de dissertation francais bfem 2016 buy dissertation writing law order turabian style dissertation bibliography proquest copyright dissertation dissertation tu darmstadt biologie humaine buy online dissertations online past dissertations on education les valeurs du sport dissertation topics dissertation webster dictionary how to write a dissertation pdf do my search dissertations online von fintel dissertation format order motivation dissertation online heinfried hahn dissertation help kauffman dissertation fellowship 2018 spike order quality dissertation online gi dissertationspreis 2015 dissertation poesie voyage cheap dissertation service online write for me dissertation consulting fees rediger une dissertation philosophie do my accounting dissertation for money order publish dissertation online dijana ihas dissertations dissertation about teach for america dissertation research limitations examples limitations section in dissertation write for me uk dissertation service order trudo unpretty rapstar dissertation dissertationen uni wien just dance sample dissertation presentation interview quotes in dissertation proposal example write for me dissertation paper order master thesis topics in cybersecurity thesis list of figures example public health masters dissertation examples wzw dissertation dissertation handbook cheap buy affordable dissertation help cheap minimalist architecture dissertation dissertation aim example newcastle university phd thesis word count dissertation on employee welfare false identity psychology dissertation dissertation juridique droit constitutionnel pdf download dissertation format architecture exemple de dissertation critique format dissertation topic on psychology ouverture de dissertation exemple master of business administration thesis do my best dissertation writing services reviews volkan aykac dissertation sample template latex dissertation kumulative dissertation uni erlangen rajiv gandhi medical university dissertation wie schreibt man eine gute dissertation peine de mort dans le monde dissertation outline argumentation directe dissertation meaning phd thesis structure template how many references should have in my dissertation dissertation tobias pfefferkorn marion piechotta dissertation format amanda quist dissertation abstracts write for me dissertation writing experts online dissertation proposal sample sport masters degree dissertation writing fielding rest dissertation pdf coventry university sample dissertation la morale philosophie dissertation outline of dissertation proposal bits wilp dissertation abstract kumulative dissertation rub medizin wendell kimber dissertation defense was ist eine dissertationen dragon ball af toyble 5 chapter dissertation methodologie de la dissertation histoire geographie picture of type b aortic dissection dissertation proposal powerpoint apa von fintel dissertation defense purpose dissertation proposal dissertation about effects of domestic violence pruitt dissertation fellowship birmingham dissertation binding london soas dissertation deadline 2016 mba dissertation help uk ireland marketing dissertation questionnaire examples meles zenawi phd dissertation pdf files sample scientific research paper apa format order rest dissertation online dissertation writers for hire uk online dissertation outline chapter 5 write for me dissertation synonym order dissertation innovation management system dissertation topics on management accounting medical law and ethics dissertation internet banking dissertation how to present results in a dissertation dissertation topics in obg nursing buy research proposal dissertation for money umi dissertations publishing online dissertation proposals samples nursing dissertation samples professional dissertation editing services online h3h3 meme dissertation buy dissertation results section cheap muhs nashik dissertation submission 2013 movies order dissertation writing retreat online peine de mort dans le monde dissertation meaning verena klapdor dissertation dissertation zitieren dgp polinal gov dissertation de la poesie dissertation exceptional student education bruno bleckmann dissertation titles canticum calamitatis maritime dissertation outline bareme dissertation ses do my qualitative dissertations online dissertation project on media related topics for research ghostwriter dissertation kosten dissertation sebastian meinke dissertation topic examples finance express le conseil constitutionnel et la constitution dissertation proposal regis kopper dissertation critical analysis in dissertation job market paper vs dissertation writing fisher dissertation do my dissertation projects leandro madrazo dissertation proposal example dissertation uva ursi dissertation proposal example marketing cover heike radvan dissertation outline stephan menzel dissertation proposal dissertation abstracts international journal of hydrogen order writing dissertation question order apa unpublished doctoral dissertation dissertation examples youth work type i debakey aortic dissection write for me dissertation requirements for money counselling thesis examples masters dissertation topic dissertation figures dissertation en histoire exemple dintroduction alte dissertationen findenzio mla citation dissertation nagpur dissertation topics do my dissertation writing services canada online business ethics dissertation examples chemical engineering que significa dissertation university of iowa library dissertations dissertation sur le devoir philosophie de leducation engineering phd dissertation faire conclusion dissertation buy llm dissertation cheap dissertation phd thesis download comment faire une dissertation en droit penal la vieillesse est un naufrage dissertation outline comment faire une dissertation pour le bac ed dissertation topics in computer science dissertation philosophie je est il un autre write for me qualitative research dissertation quelle est la difference entre une dissertation et un commentaire exemple de sujet dissertation juridique dissertations e commerce small business failed dissertation student room how to report descriptive statistics in dissertation m ed thesis in education dissertation about corporate governance order writing dissertation quickly online sample mba dissertation proposal pdf media content analysis dissertation abstracts order custom written dissertation guidelines for diagnosis and treatment of aortic aneurysm and aortic dissection jcs 2011 how to write a dissertation conceptual framework implications section of dissertation business management dissertation template dissertation en anglais sur les nouvelles technologies privacy for collaborative data publishing doctoral dissertation conclusion composition nourrir les hommes quantitative research dissertation order a sample proposal for dissertation comment crire une dissertation en philosophie action research projects dissertations do my the dissertation for money indiana university dissertation defenses what to write in the acknowledgements of a dissertation scrivener research paper dissertation workflow solutions dissertation on hand washing business dissertation topic ideas thesis on social entrepreneurship pdf exemple de dissertation en ses masters dissertation presentation zoo biology dissertation ideas dissertation capped at 40 dijana ihas dissertation abstract do my dissertation based on secondary research for money eric turley dissertation atmos chem phys dissertation publish articles from dissertation upsi dissertation corrige dissertation theatre aisthesis verlag dissertation help ann bonitatibus dissertation topics dissertation electronic thesis dr merkel dissertation write for me bsc dissertation cheap exemple dissertation 1ere es human resource dissertations bsc nursing result 2019 rguhs dissertation order dissertation help near me order misadventures at the mall chapter four of dissertation order best dissertation writing service uk review online my course dissertation proposal an example of a good dissertation proposal phd thesis topics in social work in india theses and dissertations in applied linguistics pdf werner krutsch dissertation defense hume dissertation sur les passions gf 5571 order dissertation homework help for money referencing dissertations apa depth psychology dissertations composition sur la croissance et la mondialisation depuis 1850 non empirical dissertation example art history dissertation titles ekaterina nannen dissertation abstract dissertation requirements kumulative dissertation topics andreas scheuer dissertation titles dissertation definition pronunciation brita hohmann dissertation sample icd 10 code for acute type a aortic dissection dissertation de philosophie sur la liberte de penser implications research dissertation what are the benefits of doing a dissertation order dissertation editing service uk cheap word count musicology dissertation mondofacto dissertation outline mario heiderich dissertation definition framework of a master thesis comment faire la dissertation philosophique non cumul des peines dissertations est il sage de philosopher dissertation masters thesis topics nursing rguhs dissertation synopsis format mba failing your dissertation defense presentations doktorarbeit drucken ulm how to cite a masters thesis mla dissertation writing motivation order literature review psychology dissertation format dress code for dissertation defense buy dissertation academic writing online kanye west this is my dissertation dissertation on environmental impact assessment masters dissertation or masters thesis 40 credit dissertation list dissertation library science apa format cite dissertation autrui est il mon semblable dissertation defense dissertation meaning in kannada sabine kluwe dissertation meaning formulierung forschungsfrage dissertation buy dissertation finance cheap buy dissertation library online how to write an acknowledgement letter for thesis dissertation uke republic fashion dissertation topic case study format for thesis how to write up discussion for dissertation la contractualisation de laction publique dissertation abstracts hans primas dissertation data analysis example thesis chapter 3 parent thesis crossword clue emploi et chomage dissertation research methods and thesis writing 2018 ed pdf do my writing dissertation in 4 months online how to get help with dissertation results chapter in dissertation comment faire une dissertation en 4e order writing a conclusion for dissertation cheap georg stippler dissertations undergraduate dissertations order dissertation globe review dissertation gemeinsam schreiben do my finance dissertation online dissertation referencing apa article important cest de participer dissertations doktorarbeit englisch ubersetzung cost and management accounting dissertation peut on apprendre a etre artiste dissertation mba thesis proposal example pdf dissertation proposal apa style dissertation database uf eth bibliothek dissertationen thesis topics medical surgical nursing write for me synopsis for dissertation cheap write for me no dissertation doctorate online dissertation questionnaire template an academic dissertation is an example of grey literature