Biased Trend

[timhiebenthal]

Hey folks,

disclaimer: I don't code in R but Python, therefore I cannot dive too much into the code to understand

From my understanding Prophet is used to in the feature engineering part, right?
So the outputs (trend, seasonality, holidays, ...) then flow into the Regression-part. Isn't some kind of bias in there since the time-series is not "corrected" for media-effects, yet?

Imagine that every Sunday you have twice the marketing spend of a regular day. Wouldn't the output be a strong seasonality-coefficient for Sunday and ignore the fact that media is overproportionally spent on that day? Same for months: Imagine that a company is willing to spend overproportionally in low-season (to "generate" demand) and cuts spending in high-season. This would basically "smoothen" / "weaken" the seasonality captured by Prophet, wouldn't it?

So my point is: When first fitting the seasonality-features and then using these outputs to calculate the regression, don't the seasonality features have a bias AGAINST the media-features?

Where's the mistake in my thinking?

[CyprienRicque]

Hello,

I did not deep dive into the code either, so I'd love to have the opinion of a contributor. From my understanding your point is correct and this is indeed an important drawback of using Robyn.

This article confirm the problem https://getrecast.com/seasonality/ by stating

"""
To see why it’s wrong, we first need to understand how analysts typically include seasonality in their models. The typical approach is described perfectly in this post explaining how these variables are incorporated in Prophet, the forecasting library from Facebook, used in their open-source MMM tool Robyn. Each day of each cycle — for example, the day-of-week, or day-of-year — is given its own effect. This allows the analyst to draw a curve and say “this is the effect of seasonality on our business”, and then add that curve to the other components of their model.
While this makes a lot of intuitive sense, and is totally appropriate for pure forecasting (which is Prophet’s use-case), it is unfortunately completely incorrect when it comes to media mix modeling. Here’s why:

💡 Your marketing performs better when demand is high.

As obvious as this sounds, it’s not what the traditional approach to seasonality assumes. When you incorporate seasonality in your model, you’re attempting to estimate how marketing would perform independent of seasonal changes in demand. You get results that show how much of sales was due to seasonality, and separately how much was driven by marketing. The function looks like this:

Sales = Base Demand + Marketing + Seasonality.
The problem is marketing is never independent of seasonality: your marketing performs better when market demand is high, and falls flat when interest is low. You’re getting an answer to the wrong question: you don’t care how marketing might work in a world without the Super Bowl, you want to know how much to spend on game night!

It gets worse. Controlling for seasonality in your analysis will give all the credit to the season, which means you’ll under-credit the impact of marketing on your sales. Your model would tell you exactly the wrong thing: that you should spend less when people are most likely to buy!
"""

[gufengzhou]

Hello, interesting discussion. We're aware of the opinion from recast and others about trend and we share part of the critics. But we have different interpretation.

1. Is an additive model wrong? Philosophically, yes. But then so is multiplicative model, too. In the end, no one knows the true function of marketing effectiveness. An additive model is often an useful approximation esp. due to its convenient properties for result interpretation. Just as attribution has been (and in some cases still is) an useful tool.

2. One statement from recast is untrue that "all credit will go to season". The fact is that there exists marketing-independent seasonality. It's not about getting rid of seasonality as an independent variable, but rather how to also account for seasonal pattern within marketing. In an regularised regression, two perfectly collinear predictors will share the credit 50/50. This can be simply tested: just duplicate seasonality and call it "channel_1". You'll get the same beta coefficients for both. In other words, even though we don't explicitly account for seasonality within marketing, any simultaneous pattern will just "share credit". It's WRONG to say Robyn gives all credit to season. Actually we've even introduced a constraint to all prophet variables recently, thanks to an user feedback, to further restrict prophet contribution.

Finally, There're definitely more sophisticated solutions out there to explicitly account for and even quantify marketing seasonality. Robyn is not claiming any "moral high ground". We do evaluate new options and improve the package constantly. It's exactly our intention to inspire such discussion in the industry.

[CyprienRicque]

Thank you for your reply! It is more clear.

I think I understand the statement "all credit will go to season". In fact it is neither “50/50” nor “all credits” but “the unusual additional part of marketing efficiency” (maybe it is what was meant by “all credit”)

What leads me to this is an attempt to detail how Robyn do in the different cases. I am writing this to better understand. Therefore I would really appreciate if you let me know where I am not correct or inaccurate.

In short

The more you invest in the given marketing efficient event, the more credits goes to seasonality, because then prophet would take the repetitive good performance for seasonality. And then beta would adjust based on the whole time series (say 0.9) but really not enough compared to how important it should for this specific event (say 0.3). Problem is beta is not time sensitive.

The problem is also there in the other direction. If you invest, say only once in the event, then the spike will be neither seasonality nor marketing. because marketing also has his fixed beta, limiting the marketing performance for the event. The spike would not be modelled. -> It means that depending on the situation, good marketing performance can be assigned to seasonality or not modelled.

Note: It seems that I am saying that making beta time sensitive would solve the problem but I don’t know and think while beta is part of the problem, it is also part of the reason why everything works out and making it time sensitive would introduce new challenges.

Full explanation

Note: We consider that the dependant variable is only seasonalities (smooth and spikes but seasonal) and marketing to make the explanation easier. But adding other variables should not change the conclusions.

Case 1: Constant marketing spending (we know it will give bad results and it cannot be otherwise)

• Prophet: The seasonal variable captures all the dependant variable
• Robyn
  • With NRMSE (model fit) Only
    • The marketing spending do not explain the seasonality so NRMSE would not give credit to the marketing because it would lower the performance compared to seasonality only
    • i.e. beta is 0 for the marketing and 1 for the seasonality.
  • With NRMSE (model fit) + DECOMP.RSSD (business fit)

    • Same as the above because this loss function takes care of the rate between spend share and a channel’s coefficient decomposition share. But therefore it is cross channel and not between the seasonality and the channels.

      It means that as long as NRMSE did not “decide” to give some weight to the marketing channels, the DECOMP.RSSD does not interact.

As expected, being constant does not help. https://facebookexperimental.github.io/Robyn/docs/analysts-guide-to-MMM/#how-does-an-mmm-actually-work

Variation: As regression modelling looks at the correlation between the dependent and independent variables, there needs to be some variation in the data. For example, if there is a weekly variation in sales but TV activity has remained constant for the whole time period, the model can have difficulty determining how TV has impacted sales.

Case 2: Non-seasonal marketing spending (simplest scenario)

• Prophet: The seasonal variable captures the seasonality and maybe, a small part of marketing
• Robyn
  • Adjust beta seasonality to 0.95 for the parts where prophet took them. Attributes the non captured part of the dependant variable to channels. and make sure the relation between spending and share of the dependant variable is not off in between channels. Everything is fine! Example: 0.95 * seasonality + 0.5 * channel_1 + 0.5 * channel_2

Case 3: Seasonal marketing spending

• Prophet: The seasonal variable captures all the dependant variable

• Robyn: The marketing spending explains only part (time-axis) of the dependant variable so NRMSE does not give credit to the marketing because it would lower the performance compared to “seasonality only”

  → Same as Constant marketing spending

• Note (case 3b): There exists an ideal case in which the spendings perfectly follow the seasonality and in which the weight (beta) would be equally shared between seasonality and channels

Case 4a: Mix of the above (common use case)

• Prophet: The seasonal variable captures part of the dependant variable. Sometimes taking too much for seasonality sometimes the right amount.

  Note: Cannot be considered taking “not enough” see: Constrain betas of prophet variables #666

  You invested in 1/2 super bowl, therefore the seasonality for it is in between the min and max sales.

• Robyn

  • Tries to adjust beta to get a good tradeoff between the attributions. It sets seasonality beta to 0.8 while for super bowl event is should have been 0.6. Seasonality takes too much credit.

    The stadium channel (say it is the one used for super bowl) has a beta of 0.2 while it should have been 0.7 for this specific event. Part of the super bowl spike is not modelled.

Case 4.b You invest often is super bowl

Seasonality concerns a bigger part of the sales spike. is not reduced by beta because it would impact too much of the other event. Sales spike is explained by seasonality. Seasonality takes too much credit.

Case 4.c You invest rarely in super bowl

Seasonality concerns a smaller part of the sales spike. Marketing beta is limited by most of the timeline. Part of the super bowl spike is not modelled

Generalization

Problem is, when prophet takes too much for seasonality somewhere in the timeline, it cannot be taken back by Robyn without impacting the whole timeline.
→ Limiting the power of beta in that case because it is not going to sacrifice the whole performance just for one event)

Phrased differently

The contribution attributions (beta s) are constant overtime for the seasonality and any given channel. While

• In some periods prophet has the correct seasonality and in some others it does not. When it does not, we expect beta to make up for it but beta is not time varying so if it changes it would impact the whole timeline.
• Marketing contributions efficiency vary overtime

[gufengzhou]

Thanks for the comprehensive write-up. Two things upfront:

1. the beta coefs are not constrained between 0-1, but I get what you mean.
2. Prophet is used in a first step (with raw media as regressor) to get trend/season decomposition out of the dep_var, then the decomposed vectors are used as predictors in the core model (glmnet for ridge regression). Two separate steps. The commit Constrain betas of prophet variables #666 is to further constrain the beta coef of the trend/season variables from the ridge regression, making sure trend/season don't get more credit in ridge than when estimated in prophet.

• 1: Right, no variation no effect. beta = 0
• 2: Yes, possible
• 3. Not true, it's not the same as 1. Just as you recognised in the special case, when season & channel1 are identical, they will get the same credit. When they differ a bit, they will share credit on a diff ratio, but still sharing credit.
• 4a. I'm not sure if I understand you. How do you assume "season gets .8 credit, but truth is .6, so season gets too much"? You can't assume something and make conclusion out of that, right? Or probably I got you wrong?
• 4b. When season and marketing fluctuate similarly, they just share credit somehow. There's no way to tell who gets too much and who too few, because there's no subjective truth anywhere. Isn't it?
• 4c. When data is sparse, e.g. 99 days has 0 and one day has 10k as a spike in spend, this is of course also not ideal. The beta coef from ridge reflect the average effect from the 100 days as any linear model does. But in this case, because 99 rows are 0, so you can just "assign" this effect to that one day. It's again difficult to say if it's too much/too few, because there's no ground truth.

In all, I won't say Robyn is doing everything perfectly. We design the package to our best knowledge and improve over time. The only way I see to really check over-/underestimation is using a set a carefully simulated dataset, and try to recover those parameters. We haven't done such exercise yet. I remember one user has done similar things with siMMMulator and had difficulties to recover the parameters, but there're many nuances in the simulation too, and we do have quite limited resources at the moment, so we would really appreciate if some users will be interested enough to run these tests:)