Model Results Identical Whether Modeled on Spend-Only or Mixed with Impressions

[GrizzledLotus]

A quick background, I'm trying to convince my company to use Robyn for our MMM needs instead of our current methodology. To demonstrate this, I ran 12 "experiments". We have 2 market buckets we run models separately for. I showed 3 different ways to use Prophet (trend, trend+season, trend+season+holiday) with hopes of eliminating the dummy variables we currently use. Finally, I ran those 6 with all spend then again with all impressions.

When I check the model results, model results are exactly the same. The same number of models, the same model numbers and the same model metrics. I replicated this with the demo.R file, changing the paid_media_vars to all spend instead of the default mix of impressions and spend. The spend results spit out the same 4 models as the default with the exception of adding a 5th model in the pareto results.

1. Is this correct? If so, why don't the models differ based on the inputs?
2. If this is not correct, what am I doing wrong?

Thanks for your help in this.

I use the all-in-one InputCollect() step since I did it 12 times Example of one code block below:

InputCollect_sr_v3a2 <- robyn_inputs(
  dt_input = data_sr # change depending on model
  ,dt_holidays = dt_prophet_holidays ##
  ,date_var = "MediaWeek" ##
  ,dep_var = "TotalSales" ##
  ,dep_var_type = "revenue" ##
  ,prophet_vars = prophet_vars_vXa2 # change depending on Model
  ,prophet_country = "US" ##
  ,context_vars = context_vars_sr_vXa2 # change depedning on model
  ,paid_media_spends = paid_media_spends # change depedning on model
  ,paid_media_vars = paid_media_vars_v3 # change depending on model
  ,window_start = "2018-02-05" ##
  ,window_end = "2022-05-30" ##
  ,adstock = "geometric" ##
  ,hyperparameters = hyperparameters_im # change depending on model
)

Here's the change to paid_media_vars I made to the demo file to use all spend:

InputCollect_d2 <- robyn_inputs(
  dt_input = dt_simulated_weekly
  ,dt_holidays = dt_prophet_holidays
  ,date_var = "DATE" # date format must be "2020-01-01"
  ,dep_var = "revenue" # there should be only one dependent variable
  ,dep_var_type = "revenue" # "revenue" (ROI) or "conversion" (CPA)
  ,prophet_vars = c("trend", "season", "holiday") # "trend","season", "weekday" & "holiday"
  ,prophet_country = "DE"# input one country. dt_prophet_holidays includes 59 countries by default
  ,context_vars = c("competitor_sales_B", "events") # e.g. competitors, discount, unemployment etc
  ,paid_media_spends = c("tv_S","ooh_S",	"print_S"	,"facebook_S", "search_S") # mandatory input
  ,paid_media_vars = c("tv_S", "ooh_S"	,	"print_S"	,"facebook_S" ,"search_S") # mandatory.
  # paid_media_vars must have same order as paid_media_spends. Use media exposure metrics like
  # impressions, GRP etc. If not applicable, use spend instead.
  ,organic_vars = c("newsletter") # marketing activity without media spend
  ,factor_vars = c("events") # specify which variables in context_vars or organic_vars are factorial
  ,window_start = "2016-11-23"
  ,window_end = "2018-08-22"
  ,adstock = "geometric" # geometric, weibull_cdf or weibull_pdf.
)

[gedkitchener-CT]

• R version = 4.2.1
• Robyn version = 3.6.3

@GrizzledLotus - I am experiencing the same phenomenon, i.e., I get exactly the same model names and performance, regardless of whether I define paid_media_vars as "spend" or "impressions" (ceteris paribus):

InputCollect <- robyn_inputs(
  window_start      = "2020-08-03",
  window_end        = "2022-08-31",
  ...
  paid_media_vars   = x,              <- whether `x` is "spends" or "imps", the
                                         output from Robyn remains unchanged
  ...
)

I think that as of Robyn 3.6.0, the hyperparameters argument accepts "spend" variable names as default, so I wonder whether this is something to do with it (see release):

• More budget allocation stability by defaulting fitting media variables from paid_media_vars to paid_media_spends. Spend exposure fitting with Michaelis Menten function will only serve robyn_response() function output and plotting. robyn_allocator() now only relies on direct spend - response transformation. (see release)

Did you get an answer from the community/developers, or did you manage to figure it out yourself?

[GrizzledLotus]

I got an answer in my other post here: (#466). The impressions are no longer used in modeling as the transformations required for the budget recommendations make them unreliable (my understanding).

[gedkitchener-CT]

Nice one. Thanks for confirming. Back to using "spend" then...