Personalized Paths in Experiment Paths
Personalized Paths is similar to Personalized Variant in campaigns and lets you personalize any point of a Canvas journey for individual users based on conversion likelihood.
When Personalized Paths is turned on in an Experiment Path step, a group of users is held back in a delay group. The remaining users pass into an initial test to train a look-alike model for a duration you configure. After the test, a model is created to learn which user behaviors were associated with a greater likelihood of converting on a given path. Finally, each user in the delay group is sent down the path most likely to result in conversion for them based on the behaviors they exhibit and what the look-alike model learned during the initial test.
Prerequisites
You can use Personalized Paths in your Experiment Step when your Canvas is set to send once. Personalized Paths is not available for recurring or triggered Canvases.
Using Personalized Paths
Step 1: Add an Experiment Path
Add an Experiment Path to your Canvas, then turn on Personalized Paths.
Step 2: Configure Personalized Paths settings
Specify the conversion event that should determine the winner. If there are no conversion events available, return to the first step of Canvas setup and assign conversion events. Note that if you determine the winner with opens and clicks, only the first message in the path that generates opens or clicks will contribute to determining the winner.
Then set the Experiment Window. The Experiment Window determines how long users will be sent down all paths before choosing the best path for each user in the delay group. The window begins when the first user enters the step.
Step 3: Determine fallback
By default, if the results of the test aren’t enough to determine a statistically significant winner, all future users will be sent down the single best-performing path.
Alternatively, you can select Continue sending all future users the mix of paths.
This option will send future users down the mix of paths according to the percentages specified in the experiment path distribution.
Step 4: Add your paths and launch the Canvas
A single Experiment Path component can contain up to four paths. However, when Personalized Paths is turned on, you can add up to three paths in your Experiment. The fourth path should be reserved for the Delay Group that Braze automatically adds to your experiment.
Finish setting up your Canvas as needed, then launch it. When the first user has entered the experiment, you can check the Canvas to see analytics as they come in and track your experiment’s performance.
When the experiment window passes and the experiment is complete, Braze will send users in the delay group to their respective paths with the highest personalized likelihood of conversion based on the recommendation of the look-alike model.
Analytics
If Personalized Paths was turned on, your analytics view is separated into two tabs: Initial Experiment and Personalized Paths.
The Initial Experiment tab shows the metrics for each path during the experiment window. You can see a summary of how all the paths performed for the specified conversion events.
By default, the test looks for associations between user’s custom events and their path preferences. This analysis detects whether custom events increase or decrease likelihood of responding to a particular path. These relationships are then used to determine which users gets assigned which path after the experiment window passes.
The relationships between custom events and message preferences are displayed in the table on the Initial Experiment tab.
If the test can’t find a meaningful relationship between custom events and path preferences, the test will fall back to a session-based analysis method.
Fallback analysis method
Session-based analysis method
If the fallback method is used to determine Personalized Paths, the Initial Experiment tab shows a breakdown of users’ preferred variants based on a combination of certain characteristics.
These characteristics are:
- Recency: When they last had a session
- Frequency: How often they have sessions
- Tenure: How long they have been a user
Think of recency as how recent their last interaction with you was, frequency as how often they engage, and tenure as the overall length of time they’ve been engaging with you. We group users into “buckets” based on these three things (as explained in the User Characteristics table) and then see which bucket likes which path more. It’s like sorting users into hundreds of different lists based on when they last shopped with you, how often they shop, and how long they’ve been customers.
When it comes to choosing a message for a user, Braze examines the buckets they fall into. Each bucket exerts a distinct influence on path selection for users. We quantify this influence using a statistical method called logistic regression, which is a way of predicting future behavior based on past actions. This method accounts for user interactions during the initial message send. This table only summarizes the results by displaying which path users in each bucket tended to engage with.
Ultimately, Braze combines all this data to select a tailored message path for each user, to make sure it’s as engaging and relevant as possible for them.
The time intervals for each bucket are determined based on Canvas-specific user data, which may vary between Canvases.
How Personalized Paths are selected
With this method, an individual user’s recommended message is the sum of the effects of their specific recency, frequency, and tenure. Recency, frequency, and tenure are split into buckets, as illustrated in the User Characteristics table. The time range of each bucket is determined by the data for users in each individual Canvas and will change from Canvas to Canvas.
Each bucket can have a different contribution or “push” toward each path. The strength of the push for each bucket is determined from user responses in the initial experiment using logistic regression. This table only summarizes the results by displaying which path users in each bucket tended to engage with. Any individual user’s actual Personalized Path depends on the sum of the effects of the three buckets they’re in—one for each characteristic.
The Personalized Paths tab shows the results of the final experiment, where the users in the Delay Group were sent down the best-performing path for them.
The three cards on this page show your projected lift, overall results, and the projected results if you sent just the Winning Path instead. Even if there’s no lift, which can sometimes happen, the result is the same as sending only the Winning Path (a traditional A/B test).
- Projected lift: The improvement in your selected conversion event due to using Personalized Paths instead of sending every user down the overall best-performing path.
- Overall results: The results of the second send based on your conversion event.
- Projected results: The projected results of the second send based on your chosen optimization metric if you had sent just the Winning Variant instead.
Using Personalized Paths with local time delivery
We don’t recommend using local time delivery in Canvases with Personalized Paths. This is because experiment windows begin when the first user passes through. Users who are in very early time zones may enter the step and trigger the start of the experiment window much earlier than you expect, which can result in the experiment concluding before the bulk of your users in more typical time zones have had enough time to enter the Canvas and convert.
Alternatively, if you wish to use local delivery, use an experiment window of 24-48 or more hours. That way, users in early time zones enter the Canvas and trigger the experiment to start, but plenty of time remains in the experiment window. Users in later time zones will still have enough time to enter the Canvas and the Experiment Step with Personalized Paths and possibly convert before the experiment window expires.
Edit this page on GitHub