Issue with Calibration Input in Robyn (Python)

[KAICJ]

Hello everyone,

I’m testing out Robyn with Python using the simulated data provided in the GitHub repo. I need guidance on incorporating calibration input into the model run.

I referred to the R demo (link, see section 2a-5: Effect size calibration) but could not find similar instructions for Python. Based on my understanding, I implemented the calibration process as follows:

Calibration Code Example

Create calibration data

fb_calibration_data = ChannelCalibrationData(
lift_start_date="2018-05-01",
lift_end_date="2018-06-10",
lift_abs=400000,
spend=421000,
confidence=0.85,
metric="revenue",
calibration_scope="immediate"
)

calibrationinput = CalibrationInput(channel_data={"facebook_S": fb_calibration_data})
hyperparameters.channel_hyperparameters = hyperparameters.hyperparameters

calibrator = MediaEffectCalibrator(
mmm_data=mmm_data,
hyperparameters=hyperparameters,
calibration_input=calibrationinput
)

Temporary workaround for error in source code

MediaTransformation._apply_saturation = MediaTransformation.apply_saturation

Compute MAPE (using immediate effect as example)

fb_mape = calibrator.calculate_immediate_effect_score(
predictions=fb_predictions,
lift_value=fb_calibration_data.lift_abs,
spend=fb_calibration_data.spend,
channel="facebook_S",
data=fb_calibration_data
)

Create calibration dataframe

combined_channel_data = {
('facebook_S',): {
'calibration_data': fb_calibration_data,
'mape': calibrate_result.channel_scores.get(('facebook_S',))
}
}

data_rows = [{
"channel": channel[0] if isinstance(channel, tuple) else channel,
"lift_start_date": info['calibration_data'].lift_start_date,
"lift_end_date": info['calibration_data'].lift_end_date,
"lift_abs": info['calibration_data'].lift_abs,
"spend": info['calibration_data'].spend,
"confidence": info['calibration_data'].confidence,
"metric": info['calibration_data'].metric,
"calibration_scope": info['calibration_data'].calibration_scope,
"mape": info['mape']
} for channel, info in combined_channel_data.items()]

calibration_df = pd.DataFrame(data_rows)

Issue & Warning

When running the model, Robyn returns the following warning:

WARNING:robyn.modeling.convergence.convergence:'mape' column not found or all zeros. Assuming model is not calibrated.

Here’s my model execution setup:

model_executor = ModelExecutor(
mmmdata=mmm_data,
holidays_data=holidays_data,
hyperparameters=hyperparameters,
calibration_input=calibration_df, # Passing calibration dataframe
featurized_mmm_data=featurized_mmm_data
)

trials_config = TrialsConfig(iterations=5000, trials=10)

output_models = model_executor.model_run(
ts_validation=True,
add_penalty_factor=False,
trials_config=trials_config,
rssd_zero_penalty=True,
nevergrad_algo=NevergradAlgorithm.TWO_POINTS_DE,
intercept=True,
intercept_sign="non_negative",
model_name=Models.RIDGE
)

What I’ve Tried

• Increased trials: Some discussions suggest increasing the number of trials for better calibration, but I believe 5000 iterations and 10 trials should be sufficient.
• Checked dataframe structure: The calibration dataframe (calibration_df) seems to be structured correctly.

Questions

• Why does Robyn think my model is not calibrated?
• Am I correctly passing the calibration data to model_executor?
• Any additional steps required to properly incorporate calibration in Python?

Any help would be greatly appreciated!

Thanks in advance! 🚀