Add FlinkRunner support in the Beam Example

[jen6]

Before starting native Flink support, I tested Beam’s FlinkRunner and it worked well.
It cannot be run with exec:java due to Flink dependency conflicts. However, it runs correctly on a Flink cluster.

Test Result

[RamSaw]

Thank you very much for such an awesome contribution!
It's already a lot of cool information and improvements, happy to merge it.

Do you mind if you have capacity also updating our CI to run these examples bazel and maven. That way we will know that it always works and we don't break it.

[RamSaw]

I wonder whether we really need this <build> section for shade plugin. Other runners don't have it.

[jen6]

I keep the Shade plugin here because Flink runs user code on TaskManagers and needs the full dependency closure packaged.

For Flink we have three practical options:

1. Uber-jar - This is the most repeatable path for submitting to an existing cluster. (recommended by flink official docs)
2. Local exec (no shade) — for quick local checks we can run Beam’s FlinkRunner with exec:exec (as below). This works but doesn’t solve cluster submission.
3. Attach external JARs at submit time — flink run -C file:///path/dep.jar … can inject deps without shading, but it is brittle and operationally heavier than a single uber-jar.

Given that we also want production-like testing against an existing Flink cluster, I suggest we keep the shade step in this module.

[RamSaw]

I see. Yes, it makes sense. Thank you for explanation!

We had the same issue for Spark on Dataproc (GCP) and we solved it in a slightly different way: command, pom.xml.

Will it work for Flink? That way we will have one way of building Uber jars for any backend.

[jen6]

I tried to use with assembly, but it didn't work 🤔
Shade plugin do not only packaging jar files but also excluding META-INF.
Also it recommended by the Flink official docs, so I would like to keep it

[RamSaw]

I give approval in case your answers to my comments are negative, if not and you agree with making the proposed changes, then please send for another round of review.

[RamSaw]

I wonder whether we can just merge it with previous java_binary target and add just another runtime_dep. Less code and since it is just example, should be fine to have unnecessary deps.

[jen6]

fixed in ddf9636

[RamSaw]

I see. Yes, it makes sense. Thank you for explanation!

We had the same issue for Spark on Dataproc (GCP) and we solved it in a slightly different way: command, pom.xml.

Will it work for Flink? That way we will have one way of building Uber jars for any backend.