Failures & Checklists

Catastrophic failures are often because practitioners believe, that they are doing no harm! Until the late 1800’s doctors didn’t believe that they should clean their hands before surgery. This led to countless deaths due to infections. Even when it was scientifically proved to be true; it took many decades for them to accept it as part of their standard operating procedures (SOP).

A data scientist (or any tech entrepreneur) doesn’t want to do any harm. We sincerely believe that the (data) products that we are building will improve the lives of our users. However, failures do occur.

So how can a data scientist avoid such a scenario? Using a checklist is one such approach.

A checklist ensures that data is used ethically. Review of the process (using a checklist) can happen 1) during conceptualization of the project; 2) During project execution and finally once 3) the project is completed.

A sample checklist is given below:

What kind of user consent is required?

2. Have we explained clearly what users are consenting to?

3. Have we tested for disparate error rates among different user groups?

4. Do we have a plan to protect & secure user data?

5. Have we tested our training data to ensure its fair & representative?

6. Does the team reflect diversity of opinions, backgrounds, and kind of thought?

7. Does the algorithm look at the correct artifacts/features before making a prediction?

This list isn’t exhaustive but evolving in nature…but it forces us to ask difficult questions while we plan our projects.

As an example to ensure that the algorithm is actually doing what it’s supposed to do — data science teams can use tools like SHAP & LIME. These tools help us identify features which are used by the machine learning algorithm to make a prediction.

This saves us from an embarrassing scenario where we use an incorrectly trained algorithm. An infamous example is of an algorithm which was using snow in the background of the image to predict the existence of a wolf in the image.