Methods for Measuring Reproducibility of Data Collection and Analysis in Automated Forensic Firearms Analysis

# Methods for Measuring Reproducibility of Data Collection and Analysis in Automated Forensic Firearms Analysis
### Kiegan Rice
### Iowa State University Department of Statistics
### May 5, 2020

---

<style>

/* colors: #EEB422, #8B0000, #CDC9C9, #00a8cc */
a, a > code {
  color: #EEB422;
  text-decoration: none;
}

.remark-slide-content {
  background-color: #FFFFFF;
  border-top: 80px solid #CDC9C9;
  font-size: 20px;
  font-weight: 300;
  line-height: 1.5;
  padding: 1em 2em 1em 2em
}

.inverse {
  background-color: #CDC9C9;
  border-top: 80px solid #CDC9C9;
  text-shadow: none;
	background-position: 50% 75%;
  background-size: 150px;
}

.remark-slide-content > h1 {
  font-family: 'Goudy Old Style';
  font-weight: normal;
  font-size: 45px;
  margin-top: -95px;
  margin-left: -00px;
  color: #8B0000;
}

.title-slide {
  background-image: url(images/title-background.png);
  background-size: cover;
  border-top: 0px solid #8B0000;
}

.title-slide > h1  {
  color: #FFFFFF;
  font-size: 35px;
  text-shadow: none;
  font-weight: 400;
  text-align: center;
  margin-left: 30px;
  margin-right: 30px;
  padding-top: 180px;
}

.title-slide > h2  {
  margin-top: -25px;
  padding-bottom: -20px;
  color: #FFFFFF;
  text-shadow: none;
  font-weight: 300;
  font-size: 35px;
  text-align: center;
  margin-left: 15px;
}
.title-slide > h3  {
  color: #FFFFFF;
  text-shadow: none;
  font-weight: 300;
  font-size: 20px;
  text-align: center;
  margin-left: 15px;
  margin-bottom: -20px;
}

body {
  font-family: 'Goudy Old Style';
}

.remark-slide-number {
  font-size: 13pt;
  font-family: 'Goudy Old Style';
  color: #272822;
  opacity: 1;
}

.inverse .remark-slide-number {
  font-size: 13pt;
  font-family: 'Goudy Old Style';
  color: #FAFAFA;
  opacity: 1;
}

</style>

.left-code {
  width: 40%;
  height: 92%;
  float: left;
}
.right-plot {
  width: 59%;
  float: right;
  padding-left: 1%;
}

.left-text {
  width: 59%;
  float: left;
}

.right-code {
  color: #777;
  width: 40%;
  height: 92%;
  float: right;
}

.img{
  position: absolute;
  top: 50%;
  margin-right: 15px;
  transform: translateY(-50%);
}

</style>

# Acknowledgments

### Funding statement

This work was partially funded by the Center for Statistics and Applications in Forensic Evidence (CSAFE) through Cooperative Agreement #70NANB15H176 between NIST and Iowa State University, which includes activities carried out at Carnegie Mellon University, University of California Irvine, University of Virginia and Duke University.

### Other assistance

- This work was advised by my major professors, Dr. Heike Hofmann and Dr. Ulrike Genschel  
  - The data used in this work was collected at the Roy J. Carver High Resolution Microscopy Facility at Iowa State University   
    - This study would not be possible without the following individuals: 
      - undergraduate bullet scanning team   
      - Curtis Mosher, Ph.D., lab director

---

# Outline

- Reproducibility in Data Analysis

- Forensic Firearms Analysis

- Study Design and Data Collection

- Modeling Approach

- Results and Conclusions

---

# Defining Reproducibility in Data Analysis

---

# Data Analysis as a Process

Any data analysis can be conceptualized as a **pipeline**1. 
 - linear, sequential actions 
 - raw data to result

---

# Data Analysis as a Process

Any data analysis can be conceptualized as a **pipeline**1. 
 - linear, sequential actions 
 - raw data to result

---

# Variation in Data Analysis

(1) varying input (*variation in data measurement*)  
(2)  
(3)

---

# Variation in Data Analysis

(1) varying input (*variation in data measurement*)  
(2) varying methods (*decisions by the statistician*)  
(3)

---

# Variation in Data Analysis

(1) varying input (*variation in data measurement*)  
(2) varying methods (*decisions by the statistician*)  
(3) varying code (*differences in underlying software packages*)

---

# Process Reproducibility

**Is this process reproducible?**  
  - variability of quantitative output   
  - applies to multiple *types* of variation

--
**Application to large-scale data analysis process** 
 - bullet matching in forensic firearms analysis

---

# Forensic Firearms Analysis

---

# Bullets as Forensic Evidence

`$$\begin{array}{rcl}
\textbf{microimperfections in barrel} & \to & \mbox{engraved patterns on bullet}\\
\mbox{lands} & \to & \mbox{land engraved areas (LEAs)}\\
\mbox{6 lands} & \to & \mbox{6 LEAs}\\
\end{array}$$`

.center[
<img src="images/scanning-stage0.png" width="400px" /><img src="images/bullet-sketch-whitespace.png" width="300px" />
]

---

# Bullets as Forensic Evidence

---

# Criticisms of Forensic Firearms Analysis

**Question of interest**: same source or different source  
**Assumptions**: 
  - consistency: a gun will leave same striation pattern on each bullet over time  
  - uniqueness: no two guns will produce the same striation patterns

**Recent criticisms**2 3: 
 - lack of peer-reviewed scientific research 
 - lack of large-scale studies 
 - quantifiable error rates 
 
**Proposed solution**: image-analysis algorithms for bullet comparison

.footnote[
[2] (National Research Council, 2009)  
[3] (President's Council of Advisors on Science and Technology, 2016)  
]

---

# Automated Bullet Land Comparison

Hare et al. of the Center for Statistics and Applications in Forensic Evidence (CSAFE) developed an automated bullet-matching process which completes pairwise comparisons of bullet LEAs4.

.left-code[
 
The Hare et al. method processes 3D scans by: 
 1. taking a horizontal crosscut 
 2. removing extraneous GEA data 
 3. removing bullet curvature 
 4. smoothing

Result: **2D LEA signature** 
  - represents striation pattern  
]

---

# Bullet Analysis as a Process

Hare et al. process as pipeline:

**Is the process reproducible?**

---

# Measuring Repeatability and Reproducibility

---

# Gauge Repeatability and Reproducibility

**Gauge Repeatability and Reproducibility** studies are used in engineering to test a measurement system.

Gauge R&R studies focus on *repeated measurements*  
  1. **repeatability** of measurements under the same environmental conditions  
    - same object, same operator  
  2. **reproducibility** of measurements under different environmental conditions
    - same object, different operators

---

# Gauge Repeatability and Reproducibility

Let `$y_{ijk}$` be the measured value of part `$i$`, taken by operator `$j$`, at repetition `$k$`.

`$$y_{ijk} = \mu + \alpha_{i} + \beta_{j} + \alpha\beta_{ij} + \epsilon_{ijk}$$`

with fixed, unknown process mean `$\mu$` and random effects

`$$\begin{array}{rl}
\alpha_i & \quad \mbox{for Part}\ i,\ \mbox{following a}\ N(0, \sigma^2_{\alpha}), \\
\beta_j &\quad \mbox{for Operator}\ j,\ \mbox{following a}\ N(0, \sigma^2_{\beta})\\
\alpha\beta_{ij} & \quad \mbox{for Part}\ i\times\mbox{Operator}\ j,\ \mbox{following a}\ N(0, \sigma^2_{\alpha\beta})  \\
\epsilon_{ijk} & \quad \mbox{is measurement error across repetitions},\ \mbox{following a}\ N(0, \sigma^2).
\end{array}$$`

We assume:

- all `$\alpha_i,\ \beta_j,\ \alpha\beta_{ij},\ \epsilon_{ijk}$` are independent random variables  
- `$\sigma^2_{\alpha},\ \sigma^2_{\beta},\ \sigma^2_{\alpha\beta},\ \sigma^2$` are variance components

---

# Gauge Repeatability and Reproducibility

Variance components from the model can be summarized by two quantities:

measurement error for *fixed environmental conditions*.

`$\sigma_{\mbox{reproducibility}} = \sqrt{\sigma^2_{\beta} + \sigma^2_{\alpha\beta}},$`

variability in environmental conditions *for a fixed object*.  
]

---

# Study Design and Data Collection

---

# Three-Factor Study Design

**Parts**: bullets

- fired through the same gun barrel  
- striation mark is pattern we want reproduced   
- LEAs from same barrel *similar*, but not *identical*

**Operators**: microscope operators

- responsible for physical staging

**Devices**: microscopes

**Repetition**

- scans of same barrel-land in same conditions (same bullet, operator, and microscope)  
]

---

# Scale of Study

- **machines**: 2 Sensofar Confocal Light Microscopes 
- **operators**: 8 trained undergraduate microscope operators 
- **bullets**: 9 bullets 
 - 3 bullets from 3 barrels each 
 - each barrel considered separately when modeling

`$$\begin{array}{rccc}
\hline
\mbox{Coded Name}             & \mbox{Test Set Name}   & \mbox{Barrel Type}     & \mbox{Ammunition Details} \\ \hline
\textbf{Barrel Orange} & \mbox{Hamby set 224}    & \mbox{Ruger P-85}     & \mbox{Winchester 9mm copper} \\ 
\textbf{Barrel Pink}   & \mbox{Houston set 3}   & \mbox{Ruger LCP}       & \mbox{American Eagle 124-grain 9mm copper} \\
\textbf{Barrel Blue}   & \mbox{LAPD}            & \mbox{Beretta 92 F/FS} & \mbox{Winchester 115-grain 9mm copper}  \\ \hline
\end{array}$$`

- **repetitions**: 3-5 repetitions for each set of environmental conditions 
 - one round: one scan of each bullet LEA on each machine 
 - operators completed at least 3 rounds

---

# Collected Data Description

Resolution: `$0.645\ \mu m$`/pixel

Equidistant `$\mathbf{x}$` locations `$x_i$`, `$i = 1, \dots, n_L$`

---

# Data Exposition

---

# Data Exposition

.center[
<img src="images/variability/signatures-orange-land2.png" width="800px" /><img src="images/variability/signatures-pink-land4.png" width="800px" />
]

---

# Statistical Modeling Approach

---

# Two stages of modeling

<img src="images/variability/signatures-blue-land6.png" width="600px" />
]

<img src="images/variability/two-sigs-aligned.png" width="600px" />
]

---

# Signature-level model

Three-factor Gauge R&R random effects model:

Let `$z_{jkmn}$` be the measured response for bullet `$j$`, operator `$k$`, machine `$m$`, and repetition `$n$`.

`$$\begin{array}{rl}
z_{jkmn} & = \mu + b_j + o_k + d_m + bo_{jk} + bd_{jm} + od_{km} + bod_{jkm} + e_{jkmn},
\end{array}$$`

with fixed, unknown process mean `$\mu$` and random effects

.medium[
`$$\begin{array}{rl}
b_{j} & \quad \mbox{for Bullet}\ j,\ \mbox{following a}\ N(0, \sigma^2_{b})\\
o_{k} &\quad \mbox{for Operator}\ k,\ \mbox{following a}\ N(0, \sigma^2_{o}) \\
d_{m} & \quad \mbox{for Machine}\ m\,\ \mbox{following a}\ N(0, \sigma^2_{d}) \\
bo_{jk} & \quad \mbox{for Bullet}\ j\times\mbox{Operator}\ k,\ \mbox{following a}\ N(0, \sigma^2_{bo})  \\
bd_{jm} & \quad \mbox{for Bullet}\ j\times\mbox{Machine}\ m,\ \mbox{following a}\ N(0, \sigma^2_{bd})  \\
od_{km} & \quad \mbox{for Operator}\ k\times\mbox{Machine}\ m,\ \mbox{following a}\ N(0, \sigma^2_{od})  \\
bod_{jkm} & \quad \mbox{for Bullet}\ j\times\mbox{Operator}\ k\times\mbox{Machine}\ m,\ \mbox{following a}\ N(0, \sigma^2_{bod})  \\
e_{jkmn} & \quad \mbox{is error across repetitions},\ \mbox{following a}\ N(0, \sigma^2) \\
\end{array}$$`
]

Several adaptations to this model at the LEA signature level:

1. location-based mean structure  
2. accounting for location  
3. removing dependence by subsampling

---

# Signature-level model

**1. location-based mean structure:** `$\mu = \mu_{i}$`

.center[
<img src="images/variability/signatures-blue-land6.png" width="700px" /><img src="images/variability/signatures-centered-blue-land6.png" width="700px" />
]

---

# Signature-level model

**2. accounting for location**

---

# Signature-level model

**3. removing data dependence**

.center[
<img src="images/variability/signatures-blue-land6.png" width="700px" /><img src="images/variability/signatures-centered-blue-land6.png" width="700px" />
]

---

# Signature-level model

**3. removing data dependence**

Autocorrelation functions (ACF) for signatures show extent of dependence:

.center[
<img src="images/variability/basic-acf-plot.png" width="400px" /><img src="images/variability/oneline-acf-plot.png" width="400px" />
]

---

# Signature-level model

**3. removing data dependence**

Autocorrelation functions (ACF) for signatures show extent of dependence:

---

# Signature-level model