- Treatment Event: Nudge Clicked
- Outcome Event: Course Purchased
How does user behavior change after the first nudge click?Impact Analysis is designed to help teams measure behavioral changes, quantify observed changes in key metrics, and investigate whether a feature or action is associated with changes in subsequent behavior. Important: Impact Analysis identifies correlation, not causation. A change in behavior after a Treatment Event does not by itself prove that the Treatment Event caused the change.
What Impact Analysis helps you understand
Impact Analysis can be used to investigate questions such as:- Does performing a feature relate to increased engagement with another feature?
- Does user behavior change after users first interact with a particular feature?
- How does an outcome behave in the days or weeks before and after treatment?
- How large is the observed change?
- How confident is the analysis in the observed difference?
- Whether creating a dashboard is associated with more report exports.
- Whether clicking a nudge is associated with more purchases.
- Whether using a feature is associated with increased engagement with another feature.
- Whether behavior changes after a user first performs an onboarding action.
Important: Impact Analysis does not prove causation
Impact Analysis is a behavioral analysis tool. It compares behavior around a Treatment Event, but users are not randomly assigned to treatment and control groups. Therefore: A change after the Treatment Event does not necessarily mean the Treatment Event caused the change. Other actions or circumstances may also contribute to the observed change. Impact Analysis is therefore not:- A/B testing
- Causal inference
- Experiment randomization
- Multi-touch attribution
Core concepts
How Impact Analysis works
The analysis follows this basic flow:- Select a Treatment Event.
- Select one or more Outcome Events.
- Choose how to measure the Outcome Event.
- Apply filters if needed.
- Select the analysis date range.
- Select Daily or Weekly granularity.
- Review the relative-time chart.
- Review Lift and Confidence Score in the metric table.
- Export the table if needed.
Create an Impact Analysis
1. Start with the Treatment Event
The Treatment Events section defines the user action you believe may influence later behavior. The Treatment Event is required before the analysis can run.To add a Treatment Event
- Open the Treatment Events section.
- Click + Add Event.
- Select an event from the Event Taxonomy.
2. Add Outcome Events
The Outcome Events section defines the behavior you want to evaluate after the Treatment Event. At least one Outcome Event is required before the analysis can run.To add an Outcome Event
- Open the Outcome Events section.
- Click + Add Event.
- Select one or more events.
Dashboard Created
Outcome Event
Report Exported
The analysis then examines report-export behavior around the first time each user created a dashboard.
Treatment and Outcome Event rules
Impact Analysis supports multiple events, but there is an important relationship between the two selections. You can have:One Treatment Event → Multiple Outcome Events
For example: TreatmentDashboard Created
Outcomes
Report ExportedReport SharedReport Viewed
Multiple Treatment Events → One Outcome Event
For example: TreatmentsDashboard CreatedDashboard Shared
Report Exported
This prevents configurations where multiple Treatment Events are combined with multiple Outcome Events.
If the configuration violates this rule, the product displays an error indicating that only one side of the relationship can contain multiple events.
3. Choose how to measure impact
The Measured As option determines what the analysis measures for the Outcome Event. The default is: Average Available options are:- Average
- Active %
- Frequency
Average
Average measures the average number of Outcome Event occurrences. For example, if five users generated a total of four Outcome Events on a relative day: Average = 4 ÷ 5 = 0.8 This allows you to compare average Outcome Event activity before and after the Treatment Event.Average chart
The Y-axis displays average values. The tooltip includes:- Relative day
- Outcome Event
- Average value
- Event Count / Unique Users
Active %
Active % measures the percentage of users who performed the Outcome Event. For example, if three of five users performed the Outcome Event on Day +1: Active % = 3 ÷ 5 × 100 = 60%Active % chart
The Y-axis displays percentages. The tooltip includes:- Relative day
- Outcome Event
- Active % value
- Feature used count / Active users
Frequency
Frequency shows how many users performed the Outcome Event a particular number of times during a relative day or week. Users are grouped into frequency buckets. For example:
The chart plots these frequency buckets over relative time.
The Y-axis represents the number of unique users in each frequency bucket.
Frequency tooltip
The tooltip includes:- Relative day
- Outcome Event
- Frequency Count
- Unique Users
4. Apply filters
You can filter the Impact Analysis using the Filters section. When you select the Filter option, the properties popup provides:- All Properties
- User Properties
- Cohorts
5. Select the analysis time window
Impact Analysis provides predefined time windows:- Today
- Yesterday
- Last 7 days
- Last 30 days
- 3 months
- 12 months
6. Select the time granularity
Impact Analysis supports two granularities:- Daily
- Weekly
Daily
When Daily is selected, you can select dates individually using the calendar. The chart then uses relative days:- Day -7
- Day -6
- …
- Day -1
- Day 0
- Day +1
- …
- Day +7
Weekly
When Weekly is selected, the date selector aligns with complete weeks. You cannot select individual dates inside a week. The selected date range automatically aligns to the beginning and end of the corresponding weeks. The chart uses relative weeks such as:- Week -4
- Week -3
- …
- Week -1
- Week 0
- Week +1
- …
- Week +4
Understanding relative time
Impact Analysis does not simply plot calendar dates. Instead, users are aligned around their own first Treatment Event occurrence. For each user:- Find the user’s Treatment Event occurrences.
- Identify the first occurrence.
- Assign that occurrence to Day 0 or Week 0.
- Convert the user’s Outcome Event timestamps into relative time.
- Aggregate the results across users.
- Calculate the selected metric.
Day 0 and Week 0
Day/Week 0 represents the first occurrence of the Treatment Event for each user. Relative time is interpreted as:
For example:
Day -3
The Outcome Event behavior three days before the user’s first Treatment Event.
Day 0
The user’s first Treatment Event day.
Day +3
The Outcome Event behavior three days after the user’s first Treatment Event.
Impact Analysis visualization
Once the required events are configured, the visualization uses a Line chart. The chart shows Outcome Event behavior across relative time.X-axis
The X-axis represents relative time around the Treatment Event.Daily
Example:Day -7 → Day -1 → Day 0 → Day +1 → Day +7
Weekly
Example:Week -4 → Week -1 → Week 0 → Week +1 → Week +4
Y-axis
The Y-axis depends on the selected Measured As option.Treatment Event reference line
The chart contains a vertical reference line at Day/Week 0. The reference line marks the point at which users first performed the Treatment Event. The label identifies the first treatment-event point and separates:- Pre-treatment behavior
- Post-treatment behavior
Reading the chart
The chart can help you identify patterns such as:Increasing behavior after Day 0
If the Outcome Event metric increases after Day 0 compared with the preceding period, the analysis shows an increase in the observed behavior following the Treatment Event.Decreasing behavior after Day 0
If the metric decreases after Day 0, the observed Outcome Event behavior is lower following the Treatment Event.Little visible change
If the metric remains relatively similar before and after Day 0, the analysis does not show a substantial visible change in that metric. These observations describe behavioral patterns. They should not be interpreted as proof that the Treatment Event caused the change.Chart tooltips
The tooltip depends on the selected measurement.Average
Displays:- Relative day
- Outcome Event
- Average value
- Event Count / Unique Users
Money Transfer Completed
- Average: 2.5
- 3 times
- 2 users
Active %
Displays:- Relative day
- Outcome Event
- Active % value
- Feature used count / Active users
Money Transfer Completed
- Active %: 75%
- 3 out of 4 users
Frequency
Displays:- Relative day
- Outcome Event
- Frequency Count
- Unique Users
Money Transfer Completed
- 2 times
- 15 users
Metric Table
The metric table summarizes the impact results.When Measured As is Average or Active %
The default columns are:- Event Name
- Lift
- Confidence Score
- Day intervals
When Measured As is Frequency
The default columns are:- Event Name
- Lift
- Frequency
- Confidence Score
- Day intervals
Lift
Lift is provided in the metric table as an indication of the observed change associated with the analysis. It helps quantify how the Outcome Event behavior differs around the Treatment Event.Confidence Score
The metric table also provides a Confidence Score. The purpose of the score is to indicate the statistical confidence associated with the observed change. For Average, the analysis specifies a paired t-test, because the analysis compares:- The same users
- Before vs. after behavior
- Continuous numeric values
Interpreting Lift and Confidence Score
This provides four interpretation patterns:Why Day 0 matters for confidence calculations
Day 0 is the Treatment Event day. The confidence-score example excludes Day 0 because it is the treatment day itself. The example compares the Outcome Event values from relative days before and after treatment. For example:- Day -7
- Day -6
- …
- Day -1
- Day +1
- …
- Day +7
A complete analysis example
A typical Impact Analysis can be configured as:
The analysis then aligns each user’s first Stock Purchased event to Day 0 and plots Money Transfer Completed behavior around that point.
You can then examine:
- Behavior before treatment.
- Behavior after treatment.
- Lift.
- Confidence Score.
- Changes across relative days.
Empty state
When no Treatment Event and no Outcome Event have been selected, the visualization area displays an onboarding state. The header is:How does one action affect how users do other actions?The onboarding explains three steps.
Step 1 - Hypothesize Driver
Select an event that you think could drive different user behavior. This corresponds to selecting the Treatment Event.Step 2 - Test Its Effects
Choose events to see if their performance changes over time after the driver. This corresponds to selecting Outcome Events and the measurement.Step 3 - Investigate
Validate assumptions and investigate the observed impact. This represents the analysis and interpretation stage.Export results
The metric table can be downloaded using the Export CSV action. The export provides the table data so the results can be used outside the Impact Analysis view.Important limitations
Keep the following points in mind when interpreting an Impact Analysis:Correlation is not causation
A change after the Treatment Event does not prove that the Treatment Event caused the change.Treatment Event means first occurrence
The analysis aligns each user around their first Treatment Event occurrence within the analysis logic.Users are aligned individually
Different users can perform the Treatment Event on different calendar dates. Their timelines are converted into a common relative timeline.Day/Week 0 is special
Day/Week 0 represents the first Treatment Event occurrence and is treated differently from the surrounding pre- and post-treatment periods for confidence calculations.Multiple-event configurations are constrained
You cannot freely combine multiple Treatment Events with multiple Outcome Events. One side can contain multiple events while the other contains one.Granularity affects time selection
Daily analysis allows individual date selection, while Weekly analysis aligns the selected dates to complete weeks.Statistical results require context
Confidence Score should be considered alongside Lift, user counts, and the observed pattern rather than viewed in isolation.What you can do with Impact Analysis
With Impact Analysis, you can:- Select a Treatment Event.
- Select one or more Outcome Events.
- Measure outcomes using Average, Active %, or Frequency.
- Apply property and cohort filters.
- Analyze behavior using Daily or Weekly granularity.
- Choose predefined or custom date ranges.
- Align users around their first Treatment Event occurrence.
- Compare behavior before and after treatment.
- Review relative-time trends in a line chart.
- Inspect Lift and Confidence Score.
- Export the metric table as CSV.
- Investigate whether an observed behavioral change is associated with a specific user action.
In summary
Impact Analysis helps you investigate how user behavior changes around a user’s first interaction with a selected feature or action. The analysis works by: Treatment Event → First occurrence → User alignment → Relative time → Outcome measurement → Impact interpretation You define the action you want to investigate as the Treatment Event, select the behavior you want to evaluate as the Outcome Event, choose a measurement, and then examine the Outcome Event across relative time. The resulting chart shows behavior before and after the Treatment Event, while the metric table summarizes Lift and Confidence Score. The most important interpretation rule is:Impact Analysis identifies correlation, not causation.Use the analysis to identify meaningful behavioral patterns and hypotheses for further investigation, rather than treating an observed post-treatment change as proof that the Treatment Event caused it.