What did the reminder pilot measure?
Lena is asked what a pilot can tell her about workshop attendance.
The question
Lena helped evaluate an extra reminder for a fictional neighborhood repair workshop. The team wants to know whether more invitees attended this workshop. Each invitee could check in once.
| Invited | Checked in | |
|---|---|---|
| Standard invitation plus extra reminder | 50 | 25 |
| Standard invitation alone | 50 | 15 |
Both counts refer to the same workshop.
“Explain what the group counts show and what you would need to learn next.”
Not saved by this site. Your browser may restore text when you navigate back.
A strong answer, step by step
Open each step in turn. This is one strong answer, not the only one.
Step 1: What outcome would you measure?
What would you do here? Open the move you’d make.
Divide check-ins by invitees within each group
The strong move. This measures attendance for each set of invitees.
Compare how many people opened each invitation
Tempting, but it misses something. Open counts would measure exposure, and the team asked about attendance.
Compare how many attendees came from each group
Tempting, but it misses something. That describes the mix of attendees. It leaves out invitees who did not check in.
See how a strong candidate says it
“I would measure the proportion of invitees in each group who checked in at this workshop.”
Why it works: The outcome tracks attendance in the stated window. The group denominators include invitees who did not attend.
Step 2: What absolute difference did you observe?
What would you do here? Open the move you’d make.
Report a 20 percentage-point difference
The strong move. The absolute difference between 50% and 30% is 20 percentage points.
Report about 67% and leave out the rates
Tempting, but it misses something. That is the relative difference using 30% as the reference. It leaves the absolute difference and its denominators unclear.
Use 40% attendance across all invitees
Tempting, but it misses something. The combined rate describes the whole workshop and hides the group difference.
See how a strong candidate says it
“In the pilot, 25 of 50 invitees checked in with the extra reminder and 15 of 50 checked in with the standard invitation. That is 50% versus 30%, a difference of 20 percentage points.”
Why it works: This compares the two observed rates on the requested absolute scale. It leaves the cause open.
Step 3: What can you still claim?
The extra reminders went to previous attendees. The comparison group consisted of first-time invitees.
What would you do here? Open the move you’d make.
Keep the descriptive difference and explain the confound
The strong move. The observed rates remain valid for the two groups. The cause is unresolved.
Adjust for prior attendance using the two group totals
Tempting, but it misses something. Each prior-attendance group received only one reminder condition. There is no within-group comparison to calculate.
Attribute the full difference to previous attendance
Tempting, but it misses something. The pilot cannot separate the contributions of previous attendance and the reminder.
See how a strong candidate says it
“The groups differ by 20 percentage points. Since previous attendees received the extra reminders, the group totals cannot tell me how much of that difference came from the reminder.”
Why it works: Prior attendance varies with reminder assignment and could affect attendance at this workshop.
Step 4: How would you test the reminder?
The project lead approves a follow-up among invitees who opt in. It may randomize the extra reminder. Everyone must receive the standard workshop information.
What would you do here? Open the move you’d make.
Use the approved random assignment and specify the outcome before the workshop
The strong move. Both groups get standard information, and the proposed outcome tests the extra reminder.
Give every opt-in invitee the extra reminder and compare with the old workshop
Tempting, but it misses something. The audience and workshop could change between events. That would add another source of uncertainty.
Randomly withhold all invitations from half the opt-in invitees
Tempting, but it misses something. That changes the base information and tests a different intervention.
See how a strong candidate says it
“I would randomly assign the extra reminder among the invitees who opt in. Everyone gets standard workshop information. I would define one check-in per invitee as the outcome and compare the two assigned groups.”
Why it works: The comparison tests the extra reminder while preserving standard information in both groups.
The answer, in short
- Decision
- Report the observed difference and propose the approved randomized comparison.
- Because
- Prior attendance and reminder assignment coincide in the pilot.
- What would change it
- A randomized follow-up can supply evidence about the extra reminder for the invitees in that study.
Common traps
- State which people are in each denominator.
- Keep descriptive comparisons separate from a causal estimate.
Check your answer
- I defined the outcome and its denominator.
- I reported the absolute difference.
- I explained why assignment matters.
- I proposed a comparison within the approval.
Last reviewed Nov 11, 2026