🔪 Sabotage Tactic #1: Skew your results before the survey begins

Messaging or imagery in your email invitation can introduce bias into your data that skews results before the survey even begins.

POOL OF COURSE 🏊

This invitation, from a condo developer, is curious whether future buyers would be more interested in a pool or additional parking in future developments.

Unfortunately, the email introduces bias into the results right from the get-go.

The subject line “How much would you love a pool?” can introduce voluntary selection bias into the study. Readers who love pools are more likely to open the email and complete the survey than those who would appreciate more parking. This bias will be reflected in the survey results.

The email also introduces bias by turning the invitation into a leading question. As humans, we’re influenced by emotionally-evocative imagery. This email has two glamour shots of pools, and a single, boring photo of a parking space. The options are described as “a resident-only outdoor swimming pool to enjoy,” or “an additional parking space.” The email invitation itself leads participants towards a single “correct” answer.

This email may prove to be an expensive mistake if they build a pool when condo-buyers would actually prefer additional parking.

How to fix it

Make sure you’re not accidentally creating leading questions through your email’s messaging . Be objective and use objective language.

Avoid putting questions from your survey in the subject line.

Avoid using imagery tied to your survey’s questions. Company or product logos are okay.

💥 Sabotage Tactic #2: Slash response rates and introduce bias with writing that readers don’t read

Users are busy humans who have things to do. There’s no better way to slash your response rates than to write a clunky, formal invitation email.

Not shown: personality.

The average American reads at a 7th or 8th grade level. This email invitation from Seagate scores in at grade 12.8.

The result? Less readers will read your email and click through to your survey. And those that do are subject to selection bias. There’s a limited subset of readers that will push through.

How to fix it

Use human language in your invitation. Keep it simple. Check it against reading level calculators.

Manage the four dimensions of tone in your messaging: funny v.s. serious, formal v.s. casual, respectful v.s. irreverent, and enthusiastic v.s. matter-of-fact.

💣 Sabotage Tactic #3: Overwhelm your users with information or instructions

Your invitation email has a single job to do: get your users to click through and begin taking the survey. Overload your email with info or instructions up-front and your readers will move on.

“Before you begin this survey, remember that this is what you should do when it’s done.”

This email asks readers to read a wall of text and make a deliberate decision up-front about whether they want enter a draw. Then, it provides specific instructions to remember and follow once they complete the survey.

How to fix it

Keep it simple.

Provide only the information that readers need to know to get going. Context is useful, need-to-know information, but know when to stop.

to get going. Context is useful, need-to-know information, but know when to stop. The only instructions should be how to access the survey, and how to contact you with questions. Your survey can provide the rest.

🔫 Sabotage Tactic #4: Show users how little you value their time

Studies show that personal incentives can boost response rates. But those same incentives can imply you don’t value your user’s time. This could result in lower response rates than if you’d offered nothing at all.

“Please dedicate your entire lunch break towards our survey, for the tiny chance to win $50.”

This email asks readers to dedicate 15–20 minutes of their valuable time in return for a small chance of winning a $50 gift card. If the incentive’s value is out of line with the effort asked of the reader, why would they take part?

The amount that you offer to users as an incentive depends on:

Your budget How you provide the incentive Your target audience

A good incentive balances these three factors. A neurosurgeon would pass on a chance to win a $25 gift card, but it may entice a student. If you really want to hear from a neurosurgeon, plan to fork over $200 in exchange for their response.

Science says that people respond to surveys for one of several reasons:

They want to be helpful They enjoy the topic of the survey A tangible benefit

With this in mind…

How to fix it

Offer an appropriate incentive for your specific audience. This may mean increasing the budget.

Craft your messaging to focus on how valuable their feedback is. Consider omitting the incentive altogether.

Tie the survey’s results to the user’s experience with your product or service. Let that be your incentive to them.

💀 Sabotage Tactic #5: Make the design “pop”

Your email survey invitation isn’t traditional email marketing. Your goal isn’t to drive sales: it’s to drive participation. Forget to design with purpose, and your engagement will plummet.

LOGO

This email from a sports governance organization is 90% logo and 5% survey invitation. The type is too small, with a low contrast ratio that makes it harder to read. And it’s stuffed with irrelevant links to other content.

This invitation is hard to read and makes it easy to get distracted. The result: lower response rates.

How to fix it