All about ‘Experiments’

The biggest complaint I have against the HR community is that we don’t experiment as often as we should. There aren’t enough questions asked and there just isn’t enough doubt. If you’ve been here long, you know how often I rant about the need to move away from the ‘there’s nothing new under the sun’ mindset.

The more I interact with people, the more I realize how fuzzy our idea of an experiment is. In a recent event, the organizer went around the room asking attendees to discuss their experiences with experiments. What emerged were examples of descriptive studies, pilots and optimization exercises. Rarely was the example cited truly an experiment. Since it is hard to design an experiment without understanding the basics, I’m going to spend some time today dedicated to this very topic.

What is an experiment?

Let’s start from the very beginning. What is an experiment? An experiment is essentially a type of research method in which the investigator manipulates one or more independent variables (IV) to determine the effect(s) on some behavior (the dependent variable) while controlling other relevant factors. There are of course many more ways an experiment can be defined but as long as there are a few key ingredients, we should be good. These ingredients are:

1. A problem statement
2. A hypothesis that you are trying to validate
3. Independent & dependent variables
4. Control & test group
5. Success measures

Problem statement & Hypothesis

Every experiment begins with a problem statement. For illustrative purposes, let’s assume that the problem statement is “Increase the readership of this blog.”

Now there are multiple factors that I could choose to tweak (independent variables) that would in turn influence the number of visitors per day (dependent variable). These could range from layout to marketing to number of posts per week.

For the purpose of this experiment, I will choose to go with number of posts per week. We now need a hypothesis.

A weak thesis would be – “Posting more often will impact the number of visitors per week.” (Extremely vague)

A better hypothesis is – “Posting twice a week would impact the number of visitors per week.” (Impact positively or negatively?)

A good hypothesis is – “Posting twice a week will increase the number of visitors per week.” (Let me know if you think this will hold true)

And then there’s the null hypothesis – “Posting twice a week will not change the number of visitors per week.”

Control & test groups

This creates the most confusion. A test group is one that is exposed to the modified independent variable and observed for changes in dependent variable. There can be multiple test groups with different independent variables being tested.

A control group is a group similar to the test group but is not exposed to the modified independent variable. All other environmental conditions for the two groups must be similar. Control groups are extremely useful where the experimental conditions are complex and difficult to isolate.

For our purpose, the test group is a group of readers who are exposed to a beta site that has two new posts per week whereas the control group is the one that has access to the current site. I can also include an extra test group that views three new posts a week.

Success Measures

You will never know if the experiment is successful without success measures. It is important in the design phase to know the measures that indicate success and how to measure. What are the key measurements you will track to determine if you are changing what you think you’re changing (and changing it in the direction you want)? How long will it take to see a change? What key measurements will you use to make sure you’re not unintentionally changing other important things?

Risks

Experiments are risky and rule 1 is always ‘do no harm’. There are other questions that you may need to answer and do contact your legal teams where necessary. Here are some:

1. Are you planning to experiment with or measure anything related to a protected employee class?
2. Are you planning to experiment with employee personal actions (e.g. hiring, promoting, compensating, terminating etc.)?
3. Are you planning to collect any information or change anything that might be subject to additional confidentiality, data-privacy, or other policies and guidelines specific to the country (ies) in which you plan to experiment?

These are just a few of the many components involved in designing an experiment. One also needs to keep in mind inclusion criteria and exclusion criteria, the sample size and validity checks (internal, external, construct and conclusion). It is OK to not follow all experimentation techniques to the ‘T’; however, I always believe one must know the rules to be able to break them.

Happy experimenting!