>> Thank you. Like I said,
- I will be your moderator
for the 2019 webinar series.
Will be hosting three webinars this year. The first one is today.
The second one is on April 3. This will be issues with
the RA.
On September 18. we will be doing the CWM response
[Inaudible]
>> You can keep checking back today's session will be
about an hour 45 minutes. We will take a break in
the middle. With that, I will turn it over to Dr. Jordan Adelson
and Dr. William Corl will be doing the presentation.
We're perfect. We appreciate the intersection.
So, by the title, what
an and I will be presenting, we will be
presenting on the munitions response QAPP
module one. It is presented by the task force.
In this document, -and state regulators.
We are able to pull together a good team approach to represent
what would be needed in a class
[Inaudible]
>> This is just module 1.
We are actually going to be developing module 2 which will be addressing
the RA phase of the investigation. That
will be heavily based on what we put together for class.
We are walking through the same example
for module 1. The other thing that is beneficial it
gives us the opportunity to publish other useful documents or the development
and processes during a response investigation such as a fact sheet
or white paper.
We'll working on a white paper with DSP.
We'll talk about this a little bit as we go through some of this
training.
>> The purpose of today's training is really to focus on
the planning process for putting together a
QAPP. This is very heavily based on this example.
We creatively called it a-
>> What will be presented is we use pieces
from that example to illustrate what we are talking about. We are
focus on the planning of things.
We going to the planning steps.
And when you connect complex.
Its important to note that the example that we base it on
is a complex example that is handled in phases. And and
I say this multiple times, your example, the
actual site could be less complex and you can liberalize, you can
combine some of the planning steps we are going through
and in most cases, it is advisable to zero so. It's
much easier to have a complex example that
you can simplify versus having to take a simple example and a plate
to a complex type. So in the course of going through the planning
session, we'll talk about a number of things on this worksheet
that will be addressed and when they are dressed in the planning
phase. So, you can see this, WS#10 which addresses a conceptual
site model.
Devious love goes through the quality objectives. WS#12 talks about the
measurement performance criteria.
The performance criteria are those quality control and quality assurance
checks. They related to data checks such as if you have a
problem with the measurement performance criteria, it will influence your
ability to use the data and they'll be addressed. We will talk about
WS#17 as well as measurement quality objectives.
There are other quality control criteria that
you can recover from.
And then we will wrap up our talk with the data usability assessment.
Is a key part of this process. That
is where you-we'll talk more about this. This is where he
you look through what you have established and questions you try
to answer, initially, and see if you have
actually answered those questions through the collection of data.
>> Here are highlights of the toolkit.
There is some introductions new terms and
approaches and QA QC procedures applicable to emissions response
projects. Because some of these approaches are different from
existing documents, we are working with the office of the
documents have them updated and it will be consistent within the
MR. This document will be updated.
The terms were talking about, you hear Edin I mention it, we
use HD and LD for high density and low density
areas. As well as non-pick.
Those kinds of terms you'll see.
We will [Inaudible] >>
Was laid out in this document were complex sites.
It could very well be the guy, it will be simpler
and the steps can be combined into a couple
of meetings.
But all of the objectives we laid out, they have to be common.
Where the theme comes together and reaches consensus. And that's critical.
Another point I want to mention, I don't want anyone to
be confused about this, not all of the meetings we are disgusting-they
have to take place face-to-face-tools can be
used as long as you reach consensus.
>> The agenda for the training, Ed will walk
you through the overview of the planning process.
Then we will break it down, step by step. I will walk through
section 1 which talks about assembling the preliminary CSM and defining
objectives and I will talk about the data needs in session 2. Ed
will walk through section three, section 4 and section 5.
[Inaudible] then I will go through section 5 and six
and focus on updating the design HD and
LD calculation and talking through the data usability assessment. With
that, I will turn it over to add to walk us through
that planning process.
>> Okay,
thank you so much, Jordan. Again, thanks to Jess and
Jean for the opportunity. I really appreciate it.
The MR-QAPP worksheet will be completed
in a series of projects sessions. Worksheet nine
is a place where you document, this worksheet is where you will report
participants and decisions.
[ Silence ]
>> [Captioner cannot get audio - lost feed.]
>> We know that is very difficult to get people
in a group because- certainly
we will use as many options as we can to document the DQO
development process. Many people use meeting
minutes as well. This can be included in the appendix of the
QAPP as necessary. This illustrates a phase requiring
multiple planning sessions. For example, this was designed to be
fairly complex. That was to cover as many issues as possible
for the users. So, if you have a site that is not
as complex, some of these steps can be combined or perhaps eliminate
it depending upon your site model.
>> So, here on slide 6. these next few slides
will be an overview of what we have set up. And the project
planning process, Jordan and I will be going into more detail
for each one of these planning sessions and activities that goes on for
each and every one of those. This particular slide applies to screenshot
from a process flowchart, it hopes the
project team to visualize the activities that are performed as part of the
example provided and where you are.
So is a roadmap that makes it easier
to follow the strategy. Each project planning session
or activity is shown, the participants, the general objective or the activity
that occurs in that session, and the output.
The project planning well established separate objectives
for the pulmonary -it includes delaying the high low
density areas. Prior to rising and characterizing the LE areas.
The separate DQO's for the steps are needed because typically the
objective and the data needs and the data collection could be different.
So, in this particular slide we are looking at two planning sessions
in our example, the contractor, as you can see has been selected
yet. These meetings were designed to really discuss the conceptual
site model in a very preliminary steps. We did this as a group and
we did it by design. Often, the planning sessions are necessary
to develop and inform requests for contact.
After the contractor selected, the project team will do steps with
the contract involvement. These particular steps,
we will go steps 1-4. Each project is different. The participants
may be different.
Some planning sessions
may be combined or not even required or there could be different output
which is dependent on your particular site or model.
>> Next, the contractor has been selected. The contractor helps us
do the heavy lifting for the later phases of development
where we have our sampling design, our measurement criteria and are
MQO 's in planning session 3. the
project team will revisit and solidifies the initial DQO output which is
1-4 which was developed in our initial sessions, sessions
1-2. before proceeding to the next steps 5-6 which is wary of
her decision.
In section 4. the project team will agree on the sampling design.
[Inaudible]
>> This will characterize the
MRS. The outputs will include the
MTO's that are needed. In our example, this is finalized by the project
team at the end of project
session number 4. You see there are two additional planning sessions
that are included for the purposes of the discussion of the pulmonary
findings and adapting the sampling design for future phases if needed.
>> So, once the QAPP is established, the process
is outlined in the toolkit. For example, utilizes three steps.
These basic steps. You implement, you do your data collection and
conduct the data evaluation and that is what we call usability
step and we update the site model as necessary and review and
modify our sampling design which is done as necessary for the next
step. The preliminary [Captioner has not been admitted
to chat event; if there has been a change, please contact event organizer
or VITAC at 800-590-4197 or
cc@captionedtext.com] nine characterization, the first
step is to establish a boundary between the low and high density
areas. At the end of the titillation, you are visiting the sampling design,
for the next step which is going to be this. As the
luminary characterization, we perform a interim
DUA, that's the data usability assessment, and we revise the assessment
model. Remember, just because an area has been found to contain
high density anomalies, it doesn't mean it necessarily-
tran12
>> At the end of characterization, our outputs are the PS 2 outputs
and the updated site model Dee Delaney high density and low density areas.
>> So, on slide 9.
there we go. So, for a project
planning session, number 5 is the next iteration of the QAPP.
Now that the HD and LD areas have been delineated, we proceed to the
next step. That is the character is I density
areas. We also place buffer zones at this time as Jordan
mentioned, we have some new terminology that's part of
it. You'll notice at the very beginning, of this toolkit, there is a glossary
and definitions that will help you design this. The buffer zone which
is placed in the LD area is around a high use area and accounts for
uncertainty in determining that a chew a boundary and up
provides a buffer, as the name implies, between and HUI and the adjacent
non-impacted areas, that is the NIA. This is a conservative stuff
which adds questions and confidence in our
delineation.
>> On slide 10. the final project planning
step in the toolkit is to revisit the sampling
site and agree on for the data collection for characterization of the low-density
areas, if that's needed. At the end of planning section number 6.
the project team will revisit the model and insert the
sampling design for the LD characterization on target. We haven't found anything
new based upon earlier investigations
that would change her site model, drastically. In most cases, the
elderly area characterization will be based
upon a pre-existing data. So, in summary, budget planning and the
toolkit example is a step process and although the worksheet
DQO's are established early through project planning,
we revisit that site model and sampling design before we proceed
with each of the sampling events.
That, is again, the HD and LD areas
under delineation and characterizing HD and a
LD area.
The overall theme here is plan, implement,
DUA and affirmation of our site model. Then we proceed to the next
step. Okay. That is an overview of the project planning
step in the toolkit. I will turn it back over to Jordan who will
go through and give you more specific planning steps for 1-2.
>> Perfect, thank you, Ed. Ed went through that large overview and
this is something I don't want to escape anyone's attention, the first
two planning sessions occur
with just the lead agency in the regulator. So,
this is for you to focus on the fact that if you're going to
put out something, this is the information you will need to do that successfully.
Are you going to complete, you see the outputs and worksheets,
10 and 11. they don't have to be completed, completed. But you need
enough information for this information so you can pull the solicitation
together. Having the regulator involved in this phase, prior to solicitation,
set you up for significant success.
Walk-through what you do through planning, think about the fact
this is information you need to pull together so you can effectively
put something out for-as I said, the outputs are worksheets 10
and 11.
The pole Mary CSM. And the DQO steps are sections 1 and 2. If
you're looking us, this conceptual site model should be an overview.
But, it is a working, iterative model depicting
understanding of sources, pathways and receptors. A lot of this
understanding should come from previous investigators. You should have the
information and pull the stuff together, okay? I always look at the entire
project, essentially, as completing this conceptual site model. You
start off the conceptual site model and as you gain information, it
becomes what the information at the site. It is no longer
conceptual and it is not the model, it is information you have for the
site. You'll see the type of thing that you will need is a facility
profile, the site location, the size of facility, previous investigations,
physical profile, what is the geology and topography? Right?
The release profile, right? What was the use,
what kind of distribution you expect to see on the site? Like I said,
the working version of TSM should be updated throughout the process.
As you get information, it feeds into the CSM. You can imagine, if
you don't have this information, hacking put together a solicitation?
You have to do
this prior to regulation which makes complete sense.
>> This is a graphical demonstration taken from the template from
camp example.
This is part of her conceptual site model. You see here that this
shows the number bombing targets.
And a couple of lanes, and making sure
everyone is on the same page and agrees to what is there and what
you are investigating. You are also going to see things in the areas
we have he wrote can easily read the key on this but
those are areas we don't have right of entry. You should
know this try to address this with the regulatory
people in charge and make sure you're in the same page
to make sure that this is a already known gap with information
going forward. What kind of issues and limitations you have as you
investigate? Try to plan around that as you work around it.
>> You will see here, this is an eyechart, I apologize. I won't necessarily
read through this table, but this is table 1
that we have at camp example. We took this from the toolkit and the
type of information listed here, right? It should include information
like the type of ammunition and the current future receptor. This
is how you get the information in this format.
>> That leads us to this point once
you have that CSM information together, this is another step that you need
to accomplish to help put this on
the street for solicitation. That is
defining the problem, defining the question you'll answer with
the investigation. This serve as the basis for what the
solicitation is. We want to do an investigation to answer these questions.
And, you can see through this and read through what we have here.
I won't read through the entire thing but it
is this concept that everyone reaching a regulator and
reaching consensus on what are the goals of the study? What are we
trying to show and prove? The wording choice is very important. The rest
of DQO's and all of the scope of the investigation are paid on this
problem statement. Everything flows from this. This serves
as the foundation. When we talk about the data usability assessment,
you are checking at the very end, did I collect appropriate data?
Will I be able to answer this question?
To really answer the question and the problem that is stated. It
is critical you put this information together.
And, the kind of words you use.
>> The final thing you will be pulling together in planning session 1.
would be DQO step 2. These are identified
the goals in data collection and this is you breaking apart,
essentially the problem statement indigestible pieces. I look at this
like eating an elephant for you eat an elephant? You cut it into
small pieces. You will see, you'll see the principal study
questions, examples like what is the nature and extent,
horizontal and vertical distribution of explosive hazards in MRS A and
MRS B? What are the current threats and potential threats?
What are the alternative actions for mitigating threats. These are
just examples of chopping this up, into smaller pieces, so you can
actually address it. It all stems from-
>> So, planning
session 2. like planning session 1. happens prior to solicitation.
Initially, the two planning sessions generate information that
usually need. These are, these will be worksheets 11.
steps three and four. We will not completely
fill in all the details. All we are doing is triggering
these initial steps that you will need so you can properly prepare
this a solicitation.
>> Step 3 of the DQO process, as
it interfacing every nation need to fill in the gaps in the CSM.
What you have here, here are
some examples, you need to establish President
and absence of MEC, background the city, target area density, type
and distribution. You want information needed to establish exposure potential.
What are the receptors and exposure scenarios? Any information
you can support is necessary for cost-effectiveness,
alternatives, these are things that should be pulled together for step
3.
>> In step 4. what you're doing here is your
specifying the target population.
Your defining the boundaries of the project. You can specify target
population and areas of interest, spatial boundaries,
right? In this example, we have a target population, any ordinance
use, camp example?
Including UXO and DMM. The dark population includes MD which serves
in a regulator of potential MEC hazards and potential munitions
contamination. Table of one list munitions that are known or suspected.
This is an example of things you would want to see. So,
table 11-1 will have
this information. Known or suspected munitions. What the munitions
are, UXO versus DMM, potential hazards
and severity. What kind of
munitions, expected fragmentation distance.
Detection depth. Approximate diameter and approximate length. So, all
of the stuff I talked through, right?
For walking through would you put in for the CSM and
this point of DQO development is what you would
need to move on to point to solicitation. What
we are telling you is this information, having consensus on this from
a regulator, and moving on from here
to the solicitation. It sets you up for success. That moves you into
the next planning session which is planning session 3. I will turn
this over dead. At? We're thank you Jordan Park for those of you
who are following along. I will move on to slide 21. Jordan has
started separating the elephant and moving on to planning session
3. So, we've moved through session one and two were
re-finalized the steps. Those are complete and
the team as agreed on the conceptual site model. The problem statement,
is a very important part of that. And the necessary
inputs for the investigations. The next logical step
is to develop the technical approach and that's
DQO step number five. So, assuming nothing is changed and
DQO steps one and four are still good, we defined our measurement
parameters and establish her decision rules for how the data will be used
in the measurement performance criteria, those are
the MPC, needed for the data to use them in those decision rules
appropriately. These are MPC's that we
document in QAPP worksheet number 12. So, for each step
of the IR FS, the plumber a
MRS characterization, the HD and LD area
of characterization. We list of parameters of the data we will
collect. For our transect surveys, the parameters would be
anomalies is eating a project specific detection threshold,
signal-to-noise ratio, background anomaly density and average target
area density above background. Some of
these are inputs for DSP and if they're confusing to you,
we have definitions at the front of the document. So, the key assumption
is that the DSP inputs we will use in this first up are also going
to be defined here in planning session 3. The type of inference that we
defined here would be defined the HD area, the potential
high use areas, as those were the anomaly density is greater
or equal to a user defined critical density that we
use in DSP and LD areas are those areas having
anomaly density less than that user defined
vertical density. Finally, the decision rules are developed that summarize
how those parameters, those assumptions, and types of inference, will be
used in making a specific decision.
Using the data to delineate the HD and LD areas,
the HUI, the eluate and the NIA, as well.
>> Moving on to slide 23. in our camp example,
phase planning may not be applicable to all sites as Jordan mentioned.
For example, a preliminary MRS characterization may not
always be necessary if your locations of the target several document,
you have a solid SI and is reflected in all of that
work you put into that preliminary conceptual
site model. All available and relevant lines of evidence for dual nation
should be used.
All historical records that are available to you, historical
photos, visual observations and interviews, although should be considered.
If the project team notes in the data gaps
in the conceptual site model, then perhaps you may need to do some
additional investigation to fill those data gaps.
Planning for specific areas that may be needed or conversely the
HD-LD characterizations could be performed together. It
all depends on the conceptual site model and the complexity of your
site.
>> Designation as a low use area or and NIA,
non-impacted area, must be made based on the strength of this
evidence that all of you together put together in the central site
model. If that site model contains evidence of munitions
user presence in LD areas, is
a loose area. If the conceptual site model contains sufficient evidence
that comedic no munitions use in the LD area,
is a non-impacted area. The primary objective in characterizing the
LD area is to differentiate low use areas from non-impacted areas.
For very small sites with a fairly mature conceptual site model, the
project team may not find it necessary to delineate the LD area into a
presumed LUA or presumed LIA. It is going to be a project team decision
and again emphasize how much, how important it is for every single
team to put a lot a work, upfront, during the conceptual
site model development.
>> So, for each HD area identified during
the preliminary MRS characterization, the project
team will develop a data collection approach in DQO step number five
to determine whether it is munitions related
and characterize the various populations of anomalies, for example clusters
of specific types of emissions, that could be present and confirm
whether the area or contains a high use area, a
defined H UA boundary.
>> The determination of the sources of all sample anomalies are not
met, or MD, are consistent with other
uses, well, it indicates the area is lightly
not a H UA. And putting in the data caps,
is critical at this point. The decision rules, if met for an day
or I they are to identified, and the
conceptual site model indicates munitions were used in that area,
the area will be confirmed as a high use area. That would be something
that the team agrees on and the team would characterize and establish
a boundary and buffer zone for that particular area. If MEC or RRD are
identified, the project team will determine if further investigation
should be conducted. If no MEC is found and that
team revisits conceptual site model or investigates the area as a performed
low use area or non-impacted depending on the conceptual site model. This
is a project team decision and the project team should agree on this
upfront before moving onto the next step.
What exactly are the criteria? What
criteria do we need on the data to be able to utilize
them in the decision rules? So, that is a DQO step
number six. This is where the project the reason the measurement
performance criteria.
I'm sure you probably heard MPC's described before,
they are the minimum criteria that specify data need to meet in order
to minimize the possibility of making Ernie's decisions. For example
that's including a high use area is a nonprotected area. You don't
want to make that error. MPC's are sometimes referred to as data quality
indicators. It's the same type of thing.
And it represents the qualitative and quantitative specifications
for things like accuracy,
precision, sensitivity, completeness and comparability.
May have heard this called parks.
Trent drove >>
That guys a moment of the sampling design. It
documents it in worksheet 17. They are also,
as Jordan mentioned, they are critical because they are the criteria
against which are data usability is meant to be evaluated at the
end of the study. That is what we document these. The QAPP output
will be on worksheet 12. As a reminder,
there are instructions for completing
-blue text will give you examples of the types of information needed.
Where applicable, the minimum recommended requirements
contained in the worksheet are presented as
black text. Project teams must provide rationale for changes
to the black text. A convenient and efficient way to do this is
to provide it as an appendix to the logic specific QAPP. You can
attach that to the QAPP itself.
>> So, this is just a screenshot directly
from the toolkit of measurement 4 for camp example. As you can
see, the worksheet has entries for the measurement activity being performed
on the left. And the specific DQO, if you will
being performed.
The performance specification and finally, the assessment activity
that summarizes how the performance is being assessed to the
stated specification. Remember that MPC's are the minimum performance
specifications that are I -
RI FSS needs to meet. They are the
criteria against which are intermediate and final DUA,
ID usability assessments will be conducted and the final DUA
will be documented on worksheet 37 that Jordan will talk about later.
In short, meeting our MPC's injures we have data that can be used for
decision-making. Failure to meet these MPC's could mean a significant
detriment to the success of the project. The final usability assessment
is going to evaluate and document the data quality and decision-making
impact of any failures to meet these criteria. Again, Jordan will review
that in Justin, little while.
>> Now that the next iteration in planning is the
final DQ step 7. using the outputs from the previous
DQO steps. An approach for collection of data
that will meet project specific DQO's and MPC's summarized.
And it describes the sampling design and the project work flow
in detail. And it will involve running the plan transects
in DSP. Where using VSP as a tool here. The VSP outputs are the transect
design for our preliminary characterization.
For this conducted, the version
of worksheet 17 contained in the QAPP at the start
of the investigation will start a detailed sampling plan and as
Jordan mentioned, the approach for the subsidence phases.
Results from the preliminary characterization
will be used to update the CSM and revise the detailed sampling approach
for the HD area characterization if necessary.
>> On slide 28. as previously mentioned, DQO's
are summarized on QAPP number 11.
And, with the final step 7 producing the sampling approach, the completed
DQO allows us to produce the sampling design for each step
and is a mention, the toolkit relies on DSP to
provide critical features for the sampling sign. Now
VSP is a since school planning tool like any toll,
is dependent on input parameters. It is based upon assumptions that
we have agreed upon in step 5. It is important that the project team
understand and agree on those DSP inputs as they do affect the
sampling design. They should be documented on worksheet number 11.
And there are sample tables for the inputs and outputs in our toolkit.
So, we've included a sampling
design work flow diagram in worksheet 17 that will help tie all the inputs,
all of the steps and all the outputs onto one flowchart.
>> So, here on slide 29. again, this is another
eyechart. I apologize for that.
This is a screenshot of the VSP example inputs from our sampling
approach for camp example. You will note, again, these
are blue text. These are examples.
Um, there are two separate areas in
our example. Therefore, the sampling approach is also different.
The training today design directions handed how to
use VSP.
We realize there is new terminology here. Again,
their definitions at the beginning of the toolkit and I believe the
training is being planned this point for VSP. So, you will need estimate,
there are two inputs in the
VSP tool. That's the background anomaly density and the target area
anomaly density. The background and target area anomaly
density values are dependent on the detective
system and the target selection criteria. Background anomaly density
is based upon geophysical data for the area
and the like. If no prior geophysical data have been
collected, the project team will need to agree
on an initial background density.
The initial data collection can be used to evaluate whether planning
estimate was cracked and may be necessary to actually define the
transect as it proves to be inaccurate.
Again, it can be a learning curve with these investigations.
>> Target area anomaly density
should be conservatively selected to represent the smallest or lowest
intensity, that is the intensity use, that is the
lowest reasonable density. That is to be conservative in our approach.
On slide 30. you are looking at a
screenshot of the visual sample plan output.
The VSP is used to create a graph of probability of detection versus
transect spacing. Our Pro -transect spacing has been selected
from this graph which is the maximum spacing which the graph
indicated 100% likelihood of traversing and locating the target
area. In table 11-3. the transect spacing, that
is the VSP output for the different MRS methods are shown here. You
will notice the spacing is much tighter because inputs are different.
>> Moving onto slide 31. so, the sampling design
and project work flow should include the following items. It should include
a map showing the physical boundaries for each
MR S, the work show diagram that defines a separate
activities and the decision path of the sampling along with the outputs
needed for each representative step. He
should include specific descriptions for each definable feature work
including the required records, the documentation and
the deliverables. It also include contingencies in the event the old
conditions affect sampling design are different than what you would've
accepted. For example, if a portion of the site is inseparable or the
anomaly is density. Finally,
the sample design should include points in the process at which
the lead organization, regulator and stakeholder interface
might occur as agreed upon during the project scoping.
>> In slide 32. the session 4 outputs, as previously mentioned,
the project work flow in our toolkit is designed
to illustrate a sampling design for RSS
that will connect spaces due to the complexity of the site we came
up. With different range uses and conditions and even conceptual site
model data gaps which requires multiple mobilization.
Again, we did this intentionally to try and covers many aspects and
challenges that a project team may encounter.
Hopefully, the makes our toolkit more useful to practitioners. The
example incorporates the use of analog and digital act algae.
For less complex sites, the sampling design should be adapted
accordingly. For example, I sites where the location and uses of targets
are well documented during the early phases of and SI and is documented
in the site model, a may not be necessary
to conduct this in phases. The project
team may modify the project work flow to accommodate the project
specific sampling design is needed. Worksheet
17 should include the project work flow which we will describe in the
DFW, and highlight the government that is the lead organization or
regulatory organization. Their inspection, their oversight activities, the
key deliverables on the decision points, again, these are going to
be agree to upfront during project planning. Worksheet 17
should reference SOP 's containing detailed procedures
as applicable.
Moving onto slide 33-complexity, especially in our conceptual site
model, camp example, occurs because
of multiple reasons. We have large ranges with multiple uses,
and you'll note there are some non-range activities we have noted
as well. There's different targets, munition types,
in some instances there's lack of historical information creating
data gaps.
>> Is also challenging topography and geology as well. Early
planning is key.
We call it a conceptual site model for reason. It justifies where a
phased approach is really preferred to complex sites. Because,
we learn as we go. The site model and our formulations involve amateurs
as we go as well. You will see when Jordan goes through the final data
usability assessment, how beneficial it was to use a phased approach
and to deal with any conceptual site model challenges and fill in
data but -at the end of worksheet 17.
a detailed sampling design for the first phase,
but the sample design for the second and third phase describes this
in general.
And I can only perform data usability assessment, the sampling data
-
>> Any needed update to the conceptual site model, the DQO's any changes
to sampling design should always be
agreed to by the project team including our partners. This is during the
planning session and documented as technical memoranda or
placed into the agenda as needed.
The important part is to make sure it is agreed to tawn
-in our document. On site 34. conversely, many sites may not
record a phased approach because a conceptual site model is less
complex or more mature because of a will document history and/or establish
data exists from previous investigations. Make sure
the project team, and then including our partners, agree in the perceptual
site model, the problem statement and the sampling design. The initial
MRS characterization step that we
utilize may not be needed for less complex sites because of primary.
If those are well-established, the next up
isn't necessary.
Also, if there are no target areas or other complications
identified during the QA development, perhaps he LD and
HD characterization can be combined.
That is an option. Be flexible and would encourage the project lead
to be prepared for changes in the site model as a which may required
sessions.
Finally, depending upon the anticipated future land use of the site and
the activity, it may not be necessary to define the spatial locations
of those LUA's and on impacted areas. Again, it depends on the
conceptual site model and the complexity of the site.
I'm now on slide 35.
>> Which you see here is basically a workflow diagram,
17 one specifically for the preliminary
MRS characterization for work show should provide concise descriptions
for each definable feature of the work.
It highlights the inspection oversight, the deliverables and outputs for
the steps and the decision points that were agreed to earlier at planning.
There will be workflow diagrams for the HD characterization that
will be 17-2 and the LD
characterization in figure 17-3 in our toolkit. Here
there are six separate, definable features in the MRS
characterization starting with the initial site preparation
and leading through the set up the system check,
the systems that are being used, the anomaly selected and a transect
survey which leads to the demarcation of the HD and L the survey. Many
of these features are going to be used in other phases of the
HD and LD characterization as well.
Note there are periodic process steps at the data
collection activity to evaluate our measurement nullity objectives.
Have the MQO's been met for that particular sampling design, the
particulars up, that particular data collection
effort? We can come back to these during our data usability assessments,
as well.
>> Here on slide 36. what are the
MQO's and MQO's are basically the
presenters for preparing testing, inspections and quality control
for the field data collection activity and their
documented on this worksheet. So, in a screenshot you'll see we have
examples of field collection activity in the high density
characterization phase. References to the applicable VFW
and the related standard operating procedures are
included here. As Jordan mentioned, failure response-
it should be included and include a root cause analysis
to determine corrective action and who the responsible person
or party is for performing that corrective action. As to be recorded.
Again, as Jordan mentioned, we can recover from this.
As a quick reminder, MQO's are similar to
MPC's but their specific field activities
that can be recovered from Park MPC's are more critical as they
are the minimum performance specifications that
the remedial investigation must meet in order to ensure our collected
data will satisfy that the fuel's documented. They are the criteria
from which the intermediate and the final data usability will be
conducted. Okay, I led you through session number
4. I will turn this back over to Jordan who will take you through
the remaining planning things.
>> Thank you. So, he got you through session 4.
I will lead you through session 5 and 6 and talk about the usability
assessment, worksheet 37. that leads us to slide 37.
So, once again, think about this.
If needed, if your are using a phase are FF,
if you go back to what was laid out
through the entire process, this is where you will go and perform
the HD area characterization. If you have
a lead agency, contractor, and regulars
be involved. This is for the worksheet addendum.
This is what we did it in our example.
You don't have to do it that way.
This is one way to document the information.
>> As I advance a slide, you will see that workflow
diagram that was done, it shows you a very
similar work flow diagram perked right? Once
again, these are the steps you will be performing for you finalize your
sample design and site operation, if needed, you will assemble your
sensor, collect data.
To some anomaly selection and data validation. Make
sure that the issues were cheap. Then, the anomaly source
characteristics and obviously that determination and characteristics
of the data usability.
That brings you in the information that you would need
for the characterization for HD.
>> You'll see that planning section
6. we break this out into a lot of planning session. A lot of the
stuff can be combined. Planning session 6 a for the LD area
characterization. This can get tricky. Once again the same
folks are involved, you will be reviewing the HD project
report essentially including the usability assessment.
But the update to the LD area characterization sample design,
as needed, output.
The reason why this becomes trickier than the HD area is because when
you think about this conceptually, a low-density area, just by
nature, being low-density, it is very difficult to
collect enough data to make a defensible decision. There are fewer
things out there. So, it doesn't really make sense for you to go
out there and collect any information, can you really draw
a defensible decision based on the data collected? In some projects,
they want to do this. I'm not saying you shouldn't.
It is an important question to consider
going forward when doing so.
And when looking at and LD area and what possibly could determine
based on on the data collected. You'll see the workflow
diagram I pulled up. It's quite simple. You really don't go through
with the data collection is, we determine didn't make sense to do
so. Here is where you establish the low use area and the non-impacted
area boundaries perked right?
>> You conduct your final usability assessment
bringing this into the finalizing of the CSM for this phase of the
investigation. So, like I said, your project can determine and
do an investigation in this area.
It is something to consider.
How much data can collect and what use would that
be?
This brings you to the end of the planning phases. And now it will
lead us into discussion for worksheet 37. Some of the stuff,
I am on slide 41. It is things you want to document
right on the worksheet prior to doing
data feasibility assessments in the planning phase. Who would do
what? But it walks through the kind of questions you answer.
I cannot stress enough how important the data usability assessment is.
This is where you are actually making
good decisions if you can actually make the determinations you want.
Did you answer the principal study question? It was in the problem
statement that we laid out in step 1. It is to that question
based on what you've done? Is the data good enough? That is
what this process brings you to.
This is perform at the key members of the project team. You will need
technical expertise to go through this. It is not just the regulator
and the lead agency and the contractor. There are key folks
you may need to help you go through this process. You want
to ensure regulators-this is really we
make the case we completed what we set out
to do. And it is conducted in each phase. To evaluate whether the data
supports the DQO's. We spent a lot of time talking
about the fact that these MPC's are closely tied to
the DQO clerk you can actually fail the and PC,
but it will impact your DPL. It
just means you have to change what the data quality objective is and
with the petrol ability you that -
you zoomed and were hoping to see down
to a lower depth, but you weren't able to based on site conditions,
grant interference or whatever it might be. Doesn't mean that you
can't use the data, but it does that mean the data cannot be used
as you initially tended.
Are the underlying assumptions you made supported? Have sources
of uncertainty been managed properly?
Does the data represent the population of interest? Can the results be
used as attended with acceptable level of comfort
>> I mentioned the key personnel you want to consult. Here is an
example of folks.
The project manager, the project QA manager, the project
geophysicist, the QC geophysicist, the field geophysicist,
all these folks may or may not be necessary. But they probably will
be for some of it.
You will one identify the documents and records required as the data
usability assessment inputs.
How will you put out that data? Will be an addendum?
Will be a separate report? Will be added
to the final report? There are lots of ways it will be documented. Some
people like in a separate report.
So here are the steps of the data usability assessment process.
>> Initially he after a reviews the DQO's those are the foundation
for everything you're doing. The feasibility assessment, you're going
back to the DQO's to ensure, where the underlining assumptions correct?
Review the sampling design, add
the DSP and make some assumptions about background density. All of
that information you put in for DSP, where they valid? They don't
have to be exact to which you initially predicted
when you did the study. You could find a firm impression. Will impact
the data? Is it still representative?
Your assumptions aren't always good to be perfect
but they might have minimal to no impact on the DQO's or they could
have a data fact and you may have to address that. Then you have to
review data. If you do have any QC failures, today
go to the root cause analysis?
Where they affected?
Do data gaps remain? Are the places where you don't have data?
If you have data gaps, can you still make decisions you make with
these gaps in your data? In some cases you can but in some cases
you can't, you may have to re-collect the data.
>> So, that wraps up what the usability assessment
gets you, what it pulls you into. It is
critically important part of the process for you to go to that
data usability assessment and get that
information that you will need to be able to determine if you can
make the things you want to make.
So, some next steps that I thought I would mention per that would be,
not only is this training offered,
we are working on developing a two day
training course on module number one that will be
taken throughout the EPA regions, maybe not all regions,
starting this calendar year.
That process was slow down a little bit
by the government shutdown. The course will be a little more hands-on,
obviously. We will have a webinar I walk through the process and decision-making
as a go through that threshold going forward. It is
I mentioned initially, we are looking to update the
AGC QAPP and
generate module 2 of the toolkit. That that
process was we would update the agency QAPP and include other technologies
but the workgroup felt strongly, it was a wise decision, for us to
update module, the agency QAPP and follow through
some of the examples we walk through in module number one and
bring them into completion in
the RA phase of the investigation. These
are the next steps we are planning out with some of the activities.
So, somehow that's like
got out of order. But step 3 of the feasibility process,
it documents the usability and updates the
CSM. It focuses on if the data can be used as attended? I did data
-it summarizes the lessons learned.
Like I said, it initially can be part of the same document
or a separate document. It offends
I want to break it down. Like I said, it is critical, this
step of updating a CSM and finalizing it in this phase is critical. So,
in summary, project planning,
I know there's been a lot of feedback that some projects
are simpler, some projects don't necessarily need to have
these planning process is broken out. But all of these phases need
to be gone through and these need, they should occur. They can occur
in lots of different ways. Module 1 illustrates,
like I said, it can be shortened for simpler sites
and it is much easier to present an example of a complex way and
allow people to modify to a simpler project than vice versa. The CFM
is, the data usability is tremendously critical for this process. Right?
The CSM is unbelievably valuable for the decision-making
figures should be updated throughout the project.
It has to be agreed during the planning process. When you're done,
it is no longer conceptual, it is no longer modeled.
And the data usability assessment is critical
in the sense that you are actually completing the DQO process.
You have laid out your data quality objectives. At this
phase, you are saying have a met my attended goals? Was I able to
answer the study questions? Was I able to meet my quality objectives?
[ Silence ]
>> The next steps, it talks about the two day
course and the potential updates to
the AGC-tran60.
It will be around May 2020 by time we get module 2 out. We did mention
upfront, I served as the chair of the mobile
data quality workgroup. We have, there are significant interaction
in developing the document. We are continuing implementation of the
DOD's advanced Geo organization program.
Currently we have 12 programs. We will continue to provide
some oversight and assistance with this document going forward.
We will monitor the development of technology.
One thing we are committed to, as we look into the future and will
as a C development, we want to ensure the innovation
will be stifled by the systems were put in place. We want to
make sure that nothing we put in place really causes any
issues. With that, I will turn it over to Jeff to
provide questions.
>> Alright thank you for we have a lot of questions so we
will be going through them. The first question is, are
worksheets 10. 12. 17 and 22 completed prior to selecting
the lead agency contractor?
>> So, I will, the answer is no.
As I went through the initial side, which are completing, what you are
doing, you are populating worksheet 10 and going
through the first few steps of the DQO process. That is really
what you need to complete before the agency. Not even completing
worksheet 11. You not completing worksheet 10.
With worksheet 10 that gives you enough information to have a good
handle on the site model.
Working your way through DQO step four, they don't have to be, they
don't have to be fully completed at this step.
But you need to have consensus on where
you are on those big issues associated with it. Having a problem statement
defined as well as you can prior to solicitation set you up for significant
success. All of these solicitations are sent out as part of the process.
We really want to make sure and give the contractor a fair chance
to understand what they're bidding on and exactly what we are asking
them to do. This is the best way to do this.
>> Are right. Next question, how are munitions addressed in this
QAPP?
>> That is a great question. This document does and addresses.
We feel as though the existing class format addresses chemical contamination,
and that will be
answered in a separate document.
We had to make modifications to the worksheets. This addressed,
sort of, munition constituents, just to munitions investigation.
For that purpose, we think it would be a Herculean task to
make one document to address this comparatively. Beam's right.
Next question. We have a lot of questions. I will roll through
these quick. Are these worksheets available
to download and edit?
>> So, right now, they are not. We don't have them as Word files.
That is something we can make available.
We can make them as Word files to publish. The blank worksheets
I believe are available on EPA 's website as Word files. That is
something we can look into doing.
This actual document, the tool, it
posted on the website.
You can download it as a PDF.
But we can look at making a word version available for people to
utilize and make it easier.
's right. Next question. For EPA G-4.
population includes two sampling units to
be studied and from which samples will be drawn. Based on that, wouldn't
the target population be all anomalies?
>> That is actually a true statement and it comes right
out of G4.
It talks about target populations and you talk about your sampling,
whatever it would be,
we're talking populations. What we find in implementing
the QAPP for guidance, we utilized the target population definition
to not only focus on what media you are sampling but also limiting
you to the target items that you're concerned with. You can
handle that in the target population definitions that three of the,
sorry, the four of the process or you can actually
do find that in step five and go forward if you limit that definition.
We have sort of we have use that definition in a couple ways to him
is in and get agreement at that step in the process is
to really what we are focusing our attention going forward. We're yeah,
obviously, I agree, Jordan.
A big would be would be how the problem
statement is set up.
>> That's correct. We're okay.
Next question.
These are generally conducted the same way. Now that we have the MR
QAPP the process is standardized.
DQO 's steps 1-4 will always be the
same. It is I a realistic example where there would be enough of a
difference in both steps from the toolkit that would warrant an additional
meeting?
>> I guess what I was saying, if that truly is the case, once again,
if you have significant consistently an agreement from the regulators,
in this process is standardized, I think it would be tremendous
success. If you can, for those sites, that are less complex, be able to
do so, I can imagine complex sites where that isn't the case. But if
it is the case, you can follow through with the steps, the meanings can
take place over the phone with you contact the regulator and getting
there Brian. I personally think that
that will be trickier based on the site variation but I could be
proven wrong and it would be win for everybody if it was to be
that consistent. Like I said, all he would have to that situation
is to send out what you think would be the acceptable sort of information
for these four steps and get buy-in from the
regulator to go forward. If this helps in that process, that's terrific.
>> Are right. I think we answered this question.
Where do we find this toolkit?
>> It is posted on the website.
I do believe that it is, the week, the link is in
this website as well. We're okay, based off your slide 34.
what do you mean by characterizing the LD area?
The MR- QAPP doesn't have any
digging in the LD area.
>> Yes, I will take-so, as I mentioned, characterization
of the LD area occurs following their characterization of the HD
area. Boundaries for the high use areas in the buffer zones
are already going to be established.
It is not practical to select any sort of sampling site that would
be large enough to prove that a low
use area or a non-impacted their it is done munitions. So, typically,
the evaluation of the LD area will be
based upon pre-existing information. All of
that information will be gathered together as far as a conceptual
site model. In many cases, the characterization will not involve any type of field
work at all.
>> Okay. Next question.
Which guidance documents will be updated as
a result of the publication of the toolkit?
>> This was mentioned earlier but eating a chance to write it down.
>> So, the main guidance is being utilized would be the EM-200
-1-15 document. We are working closely with the folks in
the core to ensure that there is consistency in what
we have in MR- QAPP on that document.
We've had folks reference module 1 going forward. So
that is really the primary document that will be updated.
>> Okay.
How would module one be used for munitions, OB-OD units and burial
sites? ESP is not applicable to nontarget areas.
>> So, well,
I will allow you
to add on because I probably will not help with a lot of part of this
question. Certainly, the MR-QAPP is a phased investigative process
that can be used before any site. Is based on the conceptual site
model. If you build the problem statement
and go through the data quality objective process, it can be used
for OB and OD units and burial sites. I'm not
a huge expert in DSP so I can't really address the second part of
the question. We're yeah, your answer is,
is almost exactly the same as mine was going to be witches the document
can be modified. I am, once again, I am not convinced that the VSP
does address those. So, you will have
to think about, sort of, what is, what
is the bounds you're trying to determine and go forward. I know that there
is in VSP, there is, you know, you can sample
for determining and identifying hotspots. I would imagine that the
disposal area could be considered that way. But, that is sort of how,
how you have to investigated and look at
the flexibility going forward.
Aright, we have three more questions and we will call it a day. The
next one could, on the approval
page of the QAPP, if the analog approval of the QAPP
is required? Who are those approvers? We're so,
the document does clarify, it does require
if analog technology is utilized, that you need to
get a flag officer or equivalent to sign off on these
analog technologies. The document contains
a white paper that discusses some of the limitations of analog technology.
And, the reason why all of this is here is because, for those who
don't know, back in 2001. Geo D and EPA released a memo that
talked about the limitations and concerns of using
analog technology. Even with that letter, we have not
seen projects shy away from its use where federal
targets exist. The purpose of having
a signature on the document is essentially
to put projects in position for ensuring they can justify the places
and times where they need to use analog technology but also to further
put it in a position where its use is really confined to were
it is only technology that can be utilized.
That is why the signature exists because we have a good handle on
that and the policy is is existing with contracting and justifies
the use of, or justifies why you're not using
AGC on projects going forward. This is in that same vein of trying to,
sort of, but that technology into its appropriate box.
>> Okay. The second one year, we have recipes that were mentioned.
Are there formats that can be used with ESOPs? Back so, yeah. So,
to directly answer that question in terms of format,
there is not. I will emphasize that the,
those folks that are accredited on a program
are accredited according to following specific standard operating
procedures. For that program, it is set up based on
an iso requirement. The Esso requirement does stipulate things that to be
in the SOP. It is not in terms of a standardized format. I will also
emphasize that the SOP should be attach as part of the QAPP. It
is a very important part of the quality insurance aspect
of what we are doing. We have to make sure that the process it and
procedures are being followed correctly.
That is why it is important to have that piece.
>> Okay. What elements would be impacted if new information
shows that additional work is needed after
the planning sessions are largely completed? How would a revised QAPP
be documented?
>> So, you would have to, you could have an addendum and revise,
revise the CSM based on information that has occurred and obviously,
update the relevant worksheets.
If you find out a different munition or different
depth was unexpected. You determine a different use,
that information can be, you can go through the process and
identified. You can make changes.
An example for this would be, in the actual
module 1. example, we have a situation where part of the site,
it ends up being high density where it is unexpected
to be and it's a former mine site.
So, worksheets are updated and we walked through
the process would be for you to essentially go through a modified
data DQO process.
This would address that. You can do that quickly. It doesn't have
to take a long time or be overly exhaustive. Just ensure you're collecting
the data you need to make the determination per click is set, you can look into
the actual document and we walked through that example and have information
and therefore essentially how the data can be
collected and how you can ensure you make those determinations. The
goal would be, we went out there we saw debris on the ground. There
is a former mine shaft opening right here. We see no evidence
of munitions for, can we prove it through another round of taking
samples so we can cross that off our list as something
for concern? And obviously, involving a regulator in the process to make
sure you're collecting the correct information is critical.
>> I would like to add on to that.
I want to remind the group that, the CSM and problem statement,
the problem formulation is the foundation of the DQO process. We've been discussing
this throughout the training that there can be
changes dependent upon what you find, changes to the conceptual
site model. If there are major changes to the conceptual site model, everything
subsequent to that is based on a conceptual site model. If you find
a major change that is occurred, then your DQO steps
that follow that, steps 2-4. your date of
limitations of Maddrick conditions, those may no longer be valid. It
means that the tools we used for the data collection and everything
else may also be no longer be valued and they meant-
meet project objectives. Plans should always revisiting
the conceptual site model relative to new data and information
to see if any new information is a major
change to the conceptual site model or minor change. So, it should be
a team decision and
discuss of the project team and decided in terms of whether it is
minor or major or requires a revisit of your data quality objectives.
>> One thing I wanted to add, when, if the conceptual site
model initially, if we knew everything, we would need to do
this, right? The fact there are unknowns,
it is just a fact. Right?
That is why we are conducting the investigation so we can actually,
to move on to the process. So, I'm not saying I would think to expect
this to be completely wrong, but the fact you could uncover something
that you hadn't anticipated is clearly part
of the reason why the investigation is occurring. And this process allows
for for you to recover from that and make adjustments.
7 okay. I would like to thank everyone
for the presentation. It is very enlightening.
We have about 95-97% attendance.
That is outstanding.
If you look at the slide that I've currently have up,
remember we have a presentation on April 3. lessons learned and
issues with the RA tool. Then on September 18. we have
CWM response. With that, I will conclude our presentation.
I will turn this back over to Jean.
>> Thank you so very much. Let me walk through a few
reminders before we close things out for today's session. I do want
to thank oh thanks for those participants who able to
join us in today's live broadcast.
A number of you were looking for copies of the presentation materials
and I will remind you that content is already posted,
it is available on our seminar homepage.
If you look on the links on the middle right of the screen, you'll
see things labeled that has label similar resources.
That page will open up if you click the link and you can access
the presentation content as well as the link to the
toolkit that was reference.
Contact information for presenters and organizers
can be found off of that seminar homepage.
You are welcome to reach out to us if you have questions or comments.
I will ask each of you to take a brief moment to fill out the online
feedback form. I sit there and read every single one and pass on the
summary to organizers. I encourage you to complete that form to help
continue delivery of these online events. Another motivator for completing
out form, one of the most common questions we get is if we offer
credit hours. We will supply you with the participation
certificate after you fill out the feedback form
just be sure to check the box at the
bottom of the feedback form that certifies you were here for the
entire session and when you submitted, you will have access to a PDF file
you can save or print out for your own records. If you participated
in a group, each one of you can flout the feedback form on your
own whether you registered individually or not and get your own participation
certificate for your own records.
You have the one of the lucky ones replaying the recorded version today's
webinar, you can follow the links on the middle right of the screen
to seminar feedback and full that out based on a reply to get a participation
certificate. As I've noted, the session was recorded and you will
automatically receive an email once an archive is available for on-demand
playback. As Jeff Notus, the seminar series will carry
on with her April through delivery.
The registration is open. I want to thank our organizers and presenters
for taking time out of their busy schedules to offer today's session.
I want to thank those of you who are
through any technical issues to participate in our life broadcast.
And, with that I will go ahead and formally conclude today's live session.
Thanks so very much for joining us.