FDA Validation
Digital Pathology Application
The Opportunity
I was joined a project team looking for FDA sign off on the use of their suite of products in clinical diagnosis. The issue is the FDA has strict regulations on what needs to be submitted and the Human Factors evaluation had been overlooked
Special Note
This study is a little different from most UX work in that it is a little less objective than I would like. Since we were looking for FDA approval, the purpose of this study is more to SHOW than to LEARN.
SO WHY INCLUDE IT?
I believe this is an excellent example of my thought process in preparing and conducting a study. It shouldn’t be a stretch to see how the skills demonstrated in this case study can easily translate to any other generative or iterative usability.
The significance of this project demonstrates the level of work I have performed and am capable of. Anything done for the FDA requires meticulous attention to detail and a thorough understanding of the subject matter as nothing is off the table for questioning.
Thank you for hearing me out.
I hope you like what you see!
-Sam
Defining Scope
The focus was initially presented as one product.
The goal was to validate the usability of the “Primary Diagnosis” workflow using this specific product suite.
It soon became clear that it would not be that simple..
After some digging, I found that this “primary diagnosis” workflow would essentially mean the validation of everything needed to replace a physical slide and microscope.
The simplest solution was to plan and execute as two separate studies.
Over the course of a handful of interviews, I found that validating the primary diagnosis workflow would mean testing two interfaces (scanner and viewer) with two different user groups (lab imaging technicians and pathologists).
Lab Imaging Technicians:
Digitizing slides with scanner
Case creation in viewer
Pathologists:
Reviewing slides with Viewer
Now begins the split.
At this point, I began working on two separate studies. One for Lab Imaging Technicians and another for Pathologists.
Below, you will see examples of work I did for each. Though I will not cover ever step for each, most of the steps are replicated for both studies and it is safe to assume that the level of detail and method of execution for each step was consistent for both.
Imaging Tech
Pathologist
Discovery & Workflow Mapping
Defining Test Criteria
Protocol and Script Creation
Site Planning
Execution and Data Collection
What does it look like to scan slides?
I interviewed a lab imaging technician who was able to walk me through the slide scanner interface and how he uses it on a day-to-day basis.
Additionally, he shared photos and screenshots of the interface for further context. This proved to be invaluable as I had to plan the study without direct access to a scanner until sessions began.
Task analysis to define the workflow of a pathologist
I worked closely with a pathologist on the product team to understand how they would make a primary diagnosis in a clinical setting with a microscope.
I was then able to compare this with the digital pathology tool we were testing and break it down into tasks that would need to be successfully completed to reach the same diagnosis.
Drafted a protocol to test the critical tasks I defined
I took special care to ensure I was creating scenarios that would cover every task thoroughly.
Defined scenarios and drew up a script
I began by sorting critical tasks into user flows. This allowed me to then create scenarios that would require the user to complete functions that need to be tested but might typically be edge-cases.
Task analysis to identify the workflow
With the help of some specific probing questions, I was able to begin drawing up a fill picture of each user’s day-to-day workflow. From there, I broke down each step and mapped them out with sticky notes.
Simulating an “actual use” environment
One particular challenge with this study was the FDA’s requirement to simulate an actual use environment. This meant that in planning the sessions, I had to take special care to create an environment and session script that would feel “real” to the participants.
Simulated Use | Lab Technician
The sessions were conducted by delivering the scenarios as a series of cases that needed to be created for pathologist review
This took the techs through the entire workflow from scanning to case creation
The sessions were conducted in an actual anatomic pathology lab
Simulated Use | Pathologist
I carefully planned the layout and staging of the room to simulate the look and feel of a pathologist’s office while limiting the participant’s conscious awareness of the moderator and notetaker in the room
I worked closely with an internal pathologist to draft a script that used actual terminology and logic for scenarios that would feel familiar to the pathologists who participated in the study
An image from a pathologist session
An image from a lab imaging session
Intentional Structure
Each of the 30+ sessions conducted had a detailed structure to prevent priming and limit confusion that could affect the test results:
Host received participants into a waiting area
Delivered recording waivers and provided a warm welcome
Prevented different participants crossing paths when sessions were back-to-back and could slightly overlap
Note-taker kept detailed record of the sessions
Sat in the peripheral of the participant to limited distraction and mitigated the sense of “being watched”.
The computer screen was mirrored via video call to the note-taker’s laptop also limiting distraction and allowing recording and detailed notes
Moderator (me) conducted the session
Script was warm and detailed so it could be read word-for-word
Consistent experience for each participant
Limited variation between sessions
Scenarios were offered verbally and in writing
Participants could process in the format most natural to them
Conducted risk analysis to see what I needed to test
By reviewing the previously defined workflow step-by-step, I was able to identify what the FDA calls “Critical Tasks”. In other words, I picked out key steps in the workflow that, if missed or completed incorrectly, could cause harm to the patient or user. These “Critical Tasks” are what I need to validate in order to obtain approval from the FDA.
Narrowed down to critical tasks
In a similar fashion as I did with the lab tech flow, I conducted a task analysis followed by a risk analysis to identify “critical tasks” that could cause harm if done incorrectly or left incomplete.
Meticulous record of data
In order to ensure the most comprehensive report possible, detailed notes were kept for each session documenting the success or failure of each task in a given scenario along with near misses or any other notable details. This was then supported by subjective input from participants by means of a post-session interview and a questionnaire.
Finally, the report...
At this point, the fun was over and the work began. It was time for everyone’s favorite pastime: Spreadsheets.
Data was compiled into a spreadsheet
All of our notes regarding task completion were entered into a Google sheet and sorted to show certain datapoints such as percentages of completion for particular tasks, etc.
Subjective data was laced into the report
The data we captured from post-session interviews and the questionnaire was aligned with the participants and tasks the were associated with. These notes were then included along with particular data points.
Real-life Example:
Over the course of the pathologist sessions, two participants had opened incorrect cases for the first scenario.
On the surface, this would be a cause for concern, but we had asked about these instances in the post-session interviews. In both cases, the pathologists in question admitted that they had not taken the level of care they would in real life. This feedback was noted alongside the data. Additionally, we observed that the users who made that error never repeated it after they realized their mistake.
Went to great lengths to ensure accuracy in our report
Working along with two other colleagues, we assembled the data and findings into a report. Through this process, I made it a point to ensure our data was accurate both technically and in meaning. If there was a statistic that was too general, I dug into the data to bring out the details that truly told the story.
So... Did it work?
Thanks for checking out my work!
So, where do we go from here?
FDA Validation
Digital Pathology Application
The Opportunity
I was joined a project team looking for FDA sign off on the use of their suite of products in clinical diagnosis. The issue is the FDA has strict regulations on what needs to be submitted and the Human Factors evaluation had been overlooked
Special Note
This study is a little different from most UX work in that it is a little less objective than I would like. Since we were looking for FDA approval, the purpose of this study is more to SHOW than to LEARN.
SO WHY INCLUDE IT?
I believe this is an excellent example of my thought process in preparing and conducting a study. It shouldn’t be a stretch to see how the skills demonstrated in this case study can easily translate to any other generative or iterative usability.
The significance of this project demonstrates the level of work I have performed and am capable of. Anything done for the FDA requires meticulous attention to detail and a thorough understanding of the subject matter as nothing is off the table for questioning.
Thank you for hearing me out.
I hope you like what you see!
-Sam
Defining Scope
The focus was initially presented as one product.
The goal was to validate the usability of the “Primary Diagnosis” workflow using this specific product suite.
It soon became clear that it would not be that simple..
After some digging, I found that this “primary diagnosis” workflow would essentially mean the validation of everything needed to replace a physical slide and microscope.
The simplest solution was to plan and execute as two separate studies.
Over the course of a handful of interviews, I found that validating the primary diagnosis workflow would mean testing two interfaces (scanner and viewer) with two different user groups (lab imaging technicians and pathologists).
Lab Imaging Technicians:
Digitizing slides with scanner
Case creation in viewer
Pathologists:
Reviewing slides with Viewer
Now begins the split.
At this point, I began working on two separate studies. One for Lab Imaging Technicians and another for Pathologists.
Below, you will see examples of work I did for each. Though I will not cover ever step for each, most of the steps are replicated for both studies and it is safe to assume that the level of detail and method of execution for each step was consistent for both.
Discovery & Workflow Mapping
What does it look like to scan slides?
I interviewed a lab imaging technician who was able to walk me through the slide scanner interface and how he uses it on a day-to-day basis.
Additionally, he shared photos and screenshots of the interface for further context. This proved to be invaluable as I had to plan the study without direct access to a scanner until sessions began.
Task analysis to identify the workflow
With the help of some specific probing questions, I was able to begin drawing up a fill picture of each user’s day-to-day workflow. From there, I broke down each step and mapped them out with sticky notes.
Task analysis to define the workflow of a pathologist
I worked closely with a pathologist on the product team to understand how they would make a primary diagnosis in a clinical setting with a microscope.
I was then able to compare this with the digital pathology tool we were testing and break it down into tasks that would need to be successfully completed to reach the same diagnosis.
Defining Test Criteria
Conducted risk analysis to see what I needed to test
By reviewing the previously defined workflow step-by-step, I was able to identify what the FDA calls “Critical Tasks”. In other words, I picked out key steps in the workflow that, if missed or completed incorrectly, could cause harm to the patient or user. These “Critical Tasks” are what I need to validate in order to obtain approval from the FDA.
Narrowed down to critical tasks
In a similar fashion as I did with the lab tech flow, I conducted a task analysis followed by a risk analysis to identify “critical tasks” that could cause harm if done incorrectly or left incomplete.
Protocol and Script Creation
Drafted a protocol to test the critical tasks I defined
I took special care to ensure I was creating scenarios that would cover every task thoroughly.
Defined scenarios and drew up a script
I began by sorting critical tasks into user flows. This allowed me to then create scenarios that would require the user to complete functions that need to be tested but might typically be edge-cases.
Site Planning
Simulating an “actual use” environment
One particular challenge with this study was the FDA’s requirement to simulate an actual use environment. This meant that in planning the sessions, I had to take special care to create an environment and session script that would feel “real” to the participants.
Simulated Use | Lab Technician
The sessions were conducted by delivering the scenarios as a series of cases that needed to be created for pathologist review
This took the techs through the entire workflow from scanning to case creation
The sessions were conducted in an actual anatomic pathology lab
Simulated Use | Pathologist
I carefully planned the layout and staging of the room to simulate the look and feel of a pathologist’s office while limiting the participant’s conscious awareness of the moderator and notetaker in the room
I worked closely with an internal pathologist to draft a script that used actual terminology and logic for scenarios that would feel familiar to the pathologists who participated in the study
Execution and Data Collection
An image from a pathologist session
An image from a lab imaging session
Intentional Structure
Each of the 30+ sessions conducted had a detailed structure to prevent priming and limit confusion that could affect the test results:
Host received participants into a waiting area
Delivered recording waivers and provided a warm welcome
Prevented different participants crossing paths when sessions were back-to-back and could slightly overlap
Note-taker kept detailed record of the sessions
Sat in the peripheral of the participant to limited distraction and mitigated the sense of “being watched”.
The computer screen was mirrored via video call to the note-taker’s laptop also limiting distraction and allowing recording and detailed notes
Moderator (me) conducted the session
Script was warm and detailed so it could be read word-for-word
Consistent experience for each participant
Limited variation between sessions
Scenarios were offered verbally and in writing
Participants could process in the format most natural to them
Reporting
Finally, the report...
At this point, the fun was over and the work began. It was time for everyone’s favorite pastime: Spreadsheets.
Meticulous record of data
In order to ensure the most comprehensive report possible, detailed notes were kept for each session documenting the success or failure of each task in a given scenario along with near misses or any other notable details. This was then supported by subjective input from participants by means of a post-session interview and a questionnaire.
Data was compiled into a spreadsheet
All of our notes regarding task completion were entered into a Google sheet and sorted to show certain datapoints such as percentages of completion for particular tasks, etc.
Subjective data was laced into the report
The data we captured from post-session interviews and the questionnaire was aligned with the participants and tasks the were associated with. These notes were then included along with particular data points.
Real-life Example:
Over the course of the pathologist sessions, two participants had opened incorrect cases for the first scenario.
On the surface, this would be a cause for concern, but we had asked about these instances in the post-session interviews. In both cases, the pathologists in question admitted that they had not taken the level of care they would in real life. This feedback was noted alongside the data. Additionally, we observed that the users who made that error never repeated it after they realized their mistake.
Went to great lengths to ensure accuracy in our report
Working along with two other colleagues, we assembled the data and findings into a report. Through this process, I made it a point to ensure our data was accurate both technically and in meaning. If there was a statistic that was too general, I dug into the data to bring out the details that truly told the story.
So... Did it work?
Thanks for checking out my work!
So, where do we go from here?