About This Project

Survey Methodology has shown when sending out a survey, the title of the survey alters not only response rate but what type of people take the survey. This project examines how online study titles on online platforms can alter treatment effects. We replicate online experiments with known treatment heterogeneity using titles designed to attract participants showing strong, weak, or neutral treatment effects. We aim to show online study titles can alter replicability.

Ask the Scientists

What is the context of this research?

Survey titles influence participants and demographics. With the rise of online studies on platforms like mTurk or Prolific, where study titles are rarely reported, this project investigates the unexplored impact of titles on treatment effects and replicability. Recognizing treatment heterogeneity in experiments, we aim to show how titles can significantly affect both participant response and study replicability through treatment heterogeneity. For treatment effects that are moderated by participant characteristics (e.g. sex, political orientation, personality), launching the same study with different titles could draw different participants, altering participation and replicability.

What is the significance of this project?

One possible reason for unsuccessful replication could be due to characteristics of the participants. As titles of online experiments may draw people with different characteristics, the size of some treatment effects may be stronger or weaker depending simply on the title of the study.

Knowing this will change the way studies are reported in scientific publications. It may become a requirement to publish the title of the study when reporting how a study was run, so future researchers can ensure a similar recruitment strategy.

What are the goals of the project?

We will run 4 studies that have previously been shown to have treatment heterogeneity and were run using best practices. Each study will be run with three different titles, one aiming to recruit participants who show a high treatment effect, one aimed to recruit those with a small treatment effect, and an inert title.

So, for example, If an experiment has been shown to have higher treatment effects for do not believe in global warming, we will run the study under three titles that would be similar to: 1) A study on Is Global Warming a Hoax? (high treatment effect); 2) A Study on the Need for Action to Stop Climate Change (low treatment effect); 3) A study on the Weather Near You (inert title).

We will pursue a Registered Report to ensure publication regardless of outcome.