ExternalPythonOperator and XCom pull\push

[exelents]

I need to exchange some data/metadata between airflow operators in DAG.
On usual PythonOperator I get context["task_instance"] or context["ti"] and call xcom_pull/xcom_push to exchange data.
When I use ExternalPythonOperator there are no task_instance in context. So I don't know where to call xcom.
Can you help with it? Maybe there are another ways to exchange operator execution results to other operators in the DAG?

[potiuk]

In order to get more variables in the context you should make sure you use the same Python Version as your airflow and install Airflow in the external env. Then your context will contain more of the variables (not task_instance) but at least dag_run that should give you all that you need to run xcom_pull/push:

    AIRFLOW_SERIALIZABLE_CONTEXT_KEYS = {
        "macros",
        "conf",
        "dag",
        "dag_run",
        "task",
        "params",
        "triggering_dataset_events",
    }

This is explained in the Docs of the EPO

[JHLi22]

a follow up question. what is the difference between conf and dag_run["conf"]

[potiuk]

I am not sure- but maybe you can experiment and see (and then contribute it back in the docs?). I think I would have to do the same to give a good answer, so maybe you can just do it out of curiosity.

[exelents]

@potiuk Thank you, dag_run variable from context really helped!

[potiuk]

Glad it helped.

[exelents]

For somebody, who will meet the same problem.
Get task_instance:
ti = context["dag_run"].get_task_instance("<your task_id here>")

Then use ti for xcom pull\push.

[ayush7800ishere]

can you give a example dag using externalpythonoperator since i am still not able to use xcom inside external python operators . It is showing error as
in _write_args
file.write_bytes(self.pickling_library.dumps({"args": self.op_args, "kwargs": self.op_kwargs}))
TypeError: cannot pickle 'module' object