This is a Docker image appropriate for running Flink. You can run it locally with docker-compose in which case you get three containers by default:
flink-jobmanager- runs a Flink JobManager in cluster mode and exposes a port for Flink and a port for the WebUI.flink-taskmanager- runs a Flink TaskManager and connects to the Flink Job Manager via static DNS namejobmanager.flink-history-server- runs a Flink History Server
You only need to build the Docker image if you have changed Dockerfile or the startup shell script, otherwise skip to the next step and start using directly.
To build, get the code from Github, change as desired and build an image by running docker build .
Get the docker-compose.yml from Github and then use the following snippets
Start JobManager and TaskManager
docker-compose up -d will start in background a JobManager with a single TaskManager and the History Server.
Scale TaskManagers
docker-compose scale taskmanager=5 will scale to 5 TaskManagers.
Deploy and Run a Job
- Copy the Flink job JAR to the Job Manager
docker cp /path/to/job.jar $(docker ps --filter name=jobmanager --format={{.ID}}):/job.jar to
- Copy the data to each Flink node if necessary
for i in $(docker ps --filter name=flink --format={{.ID}}); do
docker cp /path/to/data.csv $i:/data.csv
done- Run the job
docker exec -it $(docker ps --filter name=jobmanager --format={{.ID}}) flink run -c <your_job_class> /job.jar [optional params]
where optional params could for example point to the dataset copied at the previous step.
Accessing Flink Web Dashboard
Navigate to http://localhost:8081
Stop Flink Cluster
docker-compose down shuts down the cluster.
Apache®, Apache Flink™, Flink™, and the Apache feather logo are trademarks of The Apache Software Foundation.