FDA Validation

Digital Pathology Application

The Opportunity

Project team looking for FDA sign off on the use of their suite of products in clinical diagnosis.

The FDA has strict regulations on what needs to be submitted

Human Factors evaluation had been overlooked

Special Note

This study is a little different from most UX work in that it is a little less objective than I would like. Since we were looking for FDA approval, the purpose of this study is more to SHOW than to LEARN.


SO WHY INCLUDE IT?


I believe this is an excellent example of my thought process in preparing and conducting a study.  It shouldn’t be a stretch to see how the skills demonstrated in this case study can easily translate to any other generative or iterative usability.


The significance of this project demonstrates the level of work I have performed and am capable of. Anything done for the FDA requires meticulous attention to detail and a thorough understanding of the subject matter as nothing is off the table for questioning.


Thank you for hearing me out.

I hope you like what you see!


-Sam

Defining Scope

The focus was initially presented as one product.

The goal was to validate the usability of the “Primary Diagnosis” workflow using this specific product suite.

It soon became clear that it would not be that simple..

After some digging, I found that this “primary diagnosis” workflow would essentially mean the validation of everything needed to replace a physical slide and microscope.

The simplest solution was to plan and execute as two separate studies.

Over the course of a handful of interviews, I found that validating the primary diagnosis workflow would mean testing two interfaces (scanner and viewer) with two different user groups (lab imaging technicians and pathologists).

Lab Imaging Technicians:

Digitizing slides with scanner

Case creation in viewer

Pathologists:

Reviewing slides with Viewer

Now begins the split.

At this point, I began working on two separate studies. One for Lab Imaging Technicians and another for Pathologists.

Below, you will see examples of work I did for each. Though I will not cover ever step for each, most of the steps are replicated for both studies and it is safe to assume that the level of detail and method of execution for each step was consistent for both.

Imaging Tech

Pathologist

Discovery & Workflow Mapping

Defining Test Criteria

Protocol and Script Creation

Site Planning

Execution and Data Collection

What does it look like to scan slides?

I interviewed a lab imaging technician who was able to walk me through the slide scanner interface and how he uses it on a day-to-day basis.


Additionally, he shared photos and screenshots of the interface for further context. This proved to be invaluable as I had to plan the study without direct access to a scanner until sessions began.

Task analysis to define the workflow of a pathologist

I worked closely with a pathologist on the product team to understand how they would make a primary diagnosis in a clinical setting with a microscope.


I was then able to compare this with the digital pathology tool we were testing and break it down into tasks that would need to be successfully completed to reach the same diagnosis.

Drafted a protocol to test the critical tasks I defined

I took special care to ensure I was creating scenarios that would cover every task thoroughly.

Defined scenarios and drew up a script

I began by sorting critical tasks into user flows. This allowed me to then create scenarios that would require the user to complete functions that need to be tested but might typically be edge-cases.

Task analysis to identify the workflow

With the help of some specific probing questions, I was able to begin drawing up a fill picture of each user’s day-to-day workflow. From there, I broke down each step and mapped them out with sticky notes.

Simulating an “actual use” environment

One particular challenge with this study was the FDA’s requirement to simulate an actual use environment. This meant that in planning the sessions, I had to take special care to create an environment and session script that would feel “real” to the participants.

Simulated Use | Lab Technician

The sessions were conducted by delivering the scenarios as a series of cases that needed to be created for pathologist review

This took the techs through the entire workflow from scanning to case creation

The sessions were conducted in an actual anatomic pathology lab

Simulated Use | Pathologist

I carefully planned the layout and staging of the room to simulate the look and feel of a pathologist’s office while limiting the participant’s conscious awareness of the moderator and notetaker in the room

I worked closely with an internal pathologist to draft a script that used actual terminology and logic for scenarios that would feel familiar to the pathologists who participated in the study

An image from a pathologist session

An image from a lab imaging session

Intentional Structure

Each of the 30+ sessions conducted had a detailed structure to prevent priming and limit confusion that could affect the test results:

Host received participants into a waiting area

Delivered recording waivers

Provided a warm welcome

Prevented different participants crossing paths when sessions were back-to-back and could slightly overlap

Note-taker kept detailed record of the sessions

Sat in the peripheral of the participant

Limited distraction

Mitigated the sense of “being watched”

Interface was mirrored via video call to the note-taker’s laptop

Limited distraction

Allowed recording and detailed notes

Moderator (me) conducted the session

Script was warm and detailed so it could be read word-for-word

Consistent experience for each participant

Limited variation between sessions

Scenarios were offered verbally and in writing

Participants could process in the format most natural to them

Conducted risk analysis to see what I needed to test

By reviewing the previously defined workflow step-by-step, I was able to identify what the FDA calls “Critical Tasks”. In other words, I picked out key steps in the workflow that, if missed or completed incorrectly, could cause harm to the patient or user. These “Critical Tasks” are what I need to validate in order to obtain approval from the FDA.

Narrowed down to critical tasks

In a similar fashion as I did with the lab tech flow, I conducted a task analysis followed by a risk analysis to identify “critical tasks” that could cause harm if done incorrectly or left incomplete.

Meticulous record of data

In order to ensure the most comprehensive report possible, detailed notes were kept for each session documenting the success or failure of each task in a given scenario along with near misses or any other notable details. This was then supported by subjective input from participants by means of a post-session interview and a questionnaire.

Finally, the report...

At this point, the fun was over and the work began. It was time for everyone’s favorite pastime: Spreadsheets.

Data was compiled into a spreadsheet

All of our notes regarding task completion were entered into a Google sheet and sorted to show certain datapoints such as percentages of completion for particular tasks, etc.

Subjective data was laced into the report

The data we captured from post-session interviews and the questionnaire was aligned with the participants and tasks the were associated with. These notes were then included along with particular data points.

Real-life Example:

Over the course of the pathologist sessions, two participants had opened incorrect cases for the first scenario.


On the surface, this would be a cause for concern, but we had asked about these instances in the post-session interviews. In both cases, the pathologists in question admitted that they had not taken the level of care they would in real life. This feedback was noted alongside the data. Additionally, we observed that the users who made that error never repeated it after they realized their mistake.

Went to great lengths to ensure accuracy in our report

Working along with two other colleagues, we assembled the data and findings into a report. Through this process, I made it a point to ensure our data was accurate both technically and in meaning. If there was a statistic that was too general, I dug into the data to bring out the details that truly told the story.

So... Did it work?

Yes!


Though the submission as a whole is still under review, the FDA gave feedback that they approved of the usability portion of the submission!

That concludes this case study.

So, where do we go from here?

FDA Validation

Digital Pathology Application

The Opportunity

Project team looking for FDA sign off on the use of their suite of products in clinical diagnosis.

The FDA has strict regulations on what needs to be submitted

Human Factors evaluation had been overlooked

Special Note

This study is a little different from most UX work in that it is a little less objective than I would like. Since we were looking for FDA approval, the purpose of this study is more to SHOW than to LEARN.


SO WHY INCLUDE IT?


I believe this is an excellent example of my thought process in preparing and conducting a study.  It shouldn’t be a stretch to see how the skills demonstrated in this case study can easily translate to any other generative or iterative usability.


The significance of this project demonstrates the level of work I have performed and am capable of. Anything done for the FDA requires meticulous attention to detail and a thorough understanding of the subject matter as nothing is off the table for questioning.


Thank you for hearing me out.

I hope you like what you see!


-Sam

Defining Scope

The focus was initially presented as one product.

The goal was to validate the usability of the “Primary Diagnosis” workflow using this specific product suite.

It soon became clear that it would not be that simple..

After some digging, I found that this “primary diagnosis” workflow would essentially mean the validation of everything needed to replace a physical slide and microscope.

The simplest solution was to plan and execute as two separate studies.

Over the course of a handful of interviews, I found that validating the primary diagnosis workflow would mean testing two interfaces (scanner and viewer) with two different user groups (lab imaging technicians and pathologists).

Lab Imaging Technicians:

Digitizing slides with scanner

Case creation in viewer

Pathologists:

Reviewing slides with Viewer

Now begins the split.

At this point, I began working on two separate studies. One for Lab Imaging Technicians and another for Pathologists.

Below, you will see examples of work I did for each. Though I will not cover ever step for each, most of the steps are replicated for both studies and it is safe to assume that the level of detail and method of execution for each step was consistent for both.

Discovery & Workflow Mapping

What does it look like to scan slides?

I interviewed a lab imaging technician who was able to walk me through the slide scanner interface and how he uses it on a day-to-day basis.


Additionally, he shared photos and screenshots of the interface for further context. This proved to be invaluable as I had to plan the study without direct access to a scanner until sessions began.

Task analysis to identify the workflow

With the help of some specific probing questions, I was able to begin drawing up a fill picture of each user’s day-to-day workflow. From there, I broke down each step and mapped them out with sticky notes.

Task analysis to define the workflow of a pathologist

I worked closely with a pathologist on the product team to understand how they would make a primary diagnosis in a clinical setting with a microscope.


I was then able to compare this with the digital pathology tool we were testing and break it down into tasks that would need to be successfully completed to reach the same diagnosis.

Defining Test Criteria

Conducted risk analysis to see what I needed to test

By reviewing the previously defined workflow step-by-step, I was able to identify what the FDA calls “Critical Tasks”. In other words, I picked out key steps in the workflow that, if missed or completed incorrectly, could cause harm to the patient or user. These “Critical Tasks” are what I need to validate in order to obtain approval from the FDA.

Narrowed down to critical tasks

In a similar fashion as I did with the lab tech flow, I conducted a task analysis followed by a risk analysis to identify “critical tasks” that could cause harm if done incorrectly or left incomplete.

Protocol and Script Creation

Drafted a protocol to test the critical tasks I defined

I took special care to ensure I was creating scenarios that would cover every task thoroughly.

Defined scenarios and drew up a script

I began by sorting critical tasks into user flows. This allowed me to then create scenarios that would require the user to complete functions that need to be tested but might typically be edge-cases.

Site Planning

Simulating an “actual use” environment

One particular challenge with this study was the FDA’s requirement to simulate an actual use environment. This meant that in planning the sessions, I had to take special care to create an environment and session script that would feel “real” to the participants.

Simulated Use | Lab Technician

The sessions were conducted by delivering the scenarios as a series of cases that needed to be created for pathologist review

This took the techs through the entire workflow from scanning to case creation

The sessions were conducted in an actual anatomic pathology lab

Simulated Use | Pathologist

I carefully planned the layout and staging of the room to simulate the look and feel of a pathologist’s office while limiting the participant’s conscious awareness of the moderator and notetaker in the room

I worked closely with an internal pathologist to draft a script that used actual terminology and logic for scenarios that would feel familiar to the pathologists who participated in the study

Execution and Data Collection

An image from a pathologist session

An image from a lab imaging session

Intentional Structure

Each of the 30+ sessions conducted had a detailed structure to prevent priming and limit confusion that could affect the test results:

Host received participants into a waiting area

Delivered recording waivers

Provided a warm welcome

Prevented different participants crossing paths when sessions were back-to-back and could slightly overlap

Note-taker kept detailed record of the sessions

Sat in the peripheral of the participant

Limited distraction

Mitigated the sense of “being watched”

Interface was mirrored via video call to the note-taker’s laptop

Limited distraction

Allowed recording and detailed notes

Moderator (me) conducted the session

Script was warm and detailed so it could be read word-for-word

Consistent experience for each participant

Limited variation between sessions

Scenarios were offered verbally and in writing

Participants could process in the format most natural to them

Reporting

Finally, the report...

At this point, the fun was over and the work began. It was time for everyone’s favorite pastime: Spreadsheets.

Meticulous record of data

In order to ensure the most comprehensive report possible, detailed notes were kept for each session documenting the success or failure of each task in a given scenario along with near misses or any other notable details. This was then supported by subjective input from participants by means of a post-session interview and a questionnaire.

Data was compiled into a spreadsheet

All of our notes regarding task completion were entered into a Google sheet and sorted to show certain datapoints such as percentages of completion for particular tasks, etc.

Subjective data was laced into the report

The data we captured from post-session interviews and the questionnaire was aligned with the participants and tasks the were associated with. These notes were then included along with particular data points.

Real-life Example:

Over the course of the pathologist sessions, two participants had opened incorrect cases for the first scenario.


On the surface, this would be a cause for concern, but we had asked about these instances in the post-session interviews. In both cases, the pathologists in question admitted that they had not taken the level of care they would in real life. This feedback was noted alongside the data. Additionally, we observed that the users who made that error never repeated it after they realized their mistake.

Went to great lengths to ensure accuracy in our report

Working along with two other colleagues, we assembled the data and findings into a report. Through this process, I made it a point to ensure our data was accurate both technically and in meaning. If there was a statistic that was too general, I dug into the data to bring out the details that truly told the story.

So... Did it work?

Yes!


Though the submission as a whole is still under review, the FDA gave feedback that they approved of the usability portion of the submission!

That concludes this case study.
So, where do we go from here?