Lookalike Model Summary

The Lookalike Model summary view consists of three tabs each containing valuable information about your model. The initial Summary tab displays key metrics about the model such as accuracy, reach, and its usage. The Configuration tab displays the options selected to build the model, and the Diagnostics tab displays the data science values of the model useful for debugging.

Model Summary

The model summary dashboard surfaces the key metrics of a model. Here you can see a model’s accuracy and reach, how the source and target audiences overlap, the important features that contribute to the model, what audiences are using the model, and any diagnostic and troubleshooting messages.

What determines model health?

At the top of the summary page, you will see a tag indicating whether your model is healthy or unhealthy. Lytics determines model health based on the model’s ability to make accurate predictions -- predictions from unhealthy models should not be trusted. Models are considered unhealthy if the accuracy is low (with an R² value less than 0.1), or if the model is overfit (with an R² value greater than 0.975).

Accuracy

Accuracy describes the precision of a Lookalike Model's predictions — that is, how closely a model's predictions match actual outcomes during model training. Models with higher accuracy can be used to narrowly target the strongest lookalike candidates, which are better applied to later stages of your funnel.

Reach

Reach describes the relative size of a Lookalike Model's addressable audience. Higher reach models can broadly target many users, which are better applied to earlier stages of your funnel.

Model predictions

The model prediction section displays the size of your source and target audiences and charts the predictions for those audiences. The X axis represents the prediction value and the Y axis represents relative density. The shape of the graph is most important, which indicates the amount of overlap between your source and target audiences. See Accuracy vs. Reach for examples of how different trade-offs between accuracy and reach will look in the model predictions graph.

Model feature importance

Model feature importance indicates features among users likely to convert from the source to the target audience. The chart displays the relative importance of these features according to the model calculation ordered from most to least important. Identifying these features may give you a better understanding of the similarities between your source and target audiences. You may even opt to include some of these features when building an audience with this model.

Model usage

Model usage displays a list of audiences built that are utilizing this model for targeting as well as a button to create a new Predictive Audience with this model. Campaigns run with Predictive Audiences from a healthy Lookalike Model perform very well, providing higher conversion rates. Clicking the Create Predictive Audience button will take you to the audience builder, pre-populated with a rule utilizing the prediction score user field for your model. More details on building these predictive audiences can be found here

Diagnostics & Troubleshooting

The Diagnostics and Troubleshooting section provides messages pertaining to the performance and status of your Lookalike Model. These messages are categorized as warnings, errors, or information. Warnings and error messages may indicate why your model is unhealthy or provide suggestions on how to improve its performance.

To resolve these messages, try changing the source or target audiences, or any of the other configurations as suggested. More help debugging these messages can be found here

Model Configuration

The Configuration tab displays the settings used when creating the model.

Model Diagnostics

The Diagnostics tab provides an advanced view into a Lookalike Model's performance and can help provide assurance that the model has a robust statistical foundation. This information is helpful for technical users looking to gain extra insight into the data science behind this Lookalike Model.

• R²: A measure of how successfully a model is able to predict its desired outcome. It can be interpreted as the percentage of variability in outcome for which the model accounts. A perfect model would have an R² value of 1, and an ineffective model would have a value of zero.
• Mean Square Error: A measure of how closely a model’s predictions matches actual outcomes. The scale of MSE has no inherent interpretability. When comparing two similar models, a lower MSE would indicate a better fit.
• False Positive Rate: The percentage of “false positives” in a model -- that is, percentage of users in the source audience that the model misclassified as users in the target audience. While the false positive rate shouldn’t be too high, a large false positive rate tends to yield a model with higher reach.
• False Negative Rate: The percentage of “false negatives” in a model -- that is, percentage of users in the target audience that the model misclassified as users in the source audience. A lower false positive rate means that a model succeeds in correctly identifying the latent characteristics of the target audience.
• Overall Error Rate: The total percentage of misclassifications that occurred during model training.
• Accuracy: A measure derived from a model’s R² value, rounded to the nearest decile.
• Reach: A measure derived from a model’s specificity value, typically ranging from 0 to 10. A larger value of reach indicates a larger proportion of the source audience that behave similarly to the target audience.
• AUC: A measure of the area under a receiver operating characteristic curve (or “Area Under the Curve”). ROC curves are used to determine the optimal decision threshold for a given model.
• Decision Threshold: The “optimal” model prediction value to use as a cutoff when constructing an audience definition. This threshold is calculated assuming that a false positive should have the same “penalty” as a false negative.