quick-start-guide.md

Quick Start Guide

If you are looking to develop and test the operator in your local machine, refer to Local development guide.

Follow the steps below if you have a Kubernetes cluster up and running.

Setup kubectl

Follow the instructions here to install and setup kubectl

Operator installation

• Let's first create the custom resource definition, namespace, and roles for running the flink operator.

$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/crd.yaml
$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/namespace.yaml
$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role.yaml
$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/role-binding.yaml

• Before creating the flink operator deployment, edit/update the operator config:

$ curl https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/config.yaml

Replace the {ingress_suffix} to indicate your cluster's ingress url.

data:
  config: |-
    operator:
      ingressUrlFormat: "{{$jobCluster}}.{ingress_suffix}"
    logger:
      level: 4

Note: If ingressUrlFormat is not set, then no ingress is created for the application.

Then create the ConfigMap on the cluster:

$ kubectl create -f config.yaml

Finally, create the operator Deployment:

$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/deploy/flinkk8soperator.yaml

• Ensure that the flink operator pod is RUNNING, and check operator logs if needed.

$ kubectl get pods -n flink-operator
$ kubectl logs {pod-name} -n flink-operator

Running the example

You can find a sample application to run with the flink operator here. Make sure to edit the value of sha with the most recently pushed tag found here

  image: docker.io/lyft/wordcount-operator-example:{sha}

To run a flink application, run the following command:

$ kubectl create -f https://raw.githubusercontent.com/lyft/flinkk8soperator/v0.5.0/examples/wordcount/flink-operator-custom-resource.yaml

The above command will create the flink application custom resource in kubernetes. The operator will observe the custom resource, and will create a flink cluster in kubernetes.

Command below should show deployments created for the application

$ kubectl get deployments -n flink-operator

Check the phase and other status attributes in the custom resource

$ kubectl get flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example -o yaml

The output should be something like this

apiVersion: v1beta1
kind: FlinkApplication
metadata:
  clusterName: ""
  creationTimestamp: 2019-07-30T07:35:42Z
  finalizers:
  - job.finalizers.flink.k8s.io
  generation: 1
  labels:
    environment: development
  name: wordcount-operator-example
  namespace: flink-operator
  resourceVersion: "1025774"
  selfLink: v1beta1
  uid: a2855178-b29c-11e9-9a3b-025000000001
spec:
  entryClass: org.apache.flink.WordCount
  flinkConfig:
    state.backend.fs.checkpointdir: file:///checkpoints/flink/checkpoints
    state.checkpoints.dir: file:///checkpoints/flink/externalized-checkpoints
    state.savepoints.dir: file:///checkpoints/flink/savepoints
    taskmanager.heap.size: 200
  flinkVersion: "1.8"
  image: docker.io/lyft/wordcount-operator-example:3b0347b2cdc1bda817e72b3099dac1c1b1363311
  jarName: wordcount-operator-example-1.0.0-SNAPSHOT.jar
  jobManagerConfig:
    envConfig: {}
    replicas: 1
    resources:
      requests:
        cpu: 200m
        memory: 200Mi
  parallelism: 3
  restartNonce: ""
  savepointInfo: {}
  taskManagerConfig:
    envConfig: {}
    resources:
      requests:
        cpu: 200m
        memory: 200Mi
    taskSlots: 2
status:
  clusterStatus:
    availableTaskSlots: 4
    health: Green
    healthyTaskManagers: 2
    numberOfTaskManagers: 2
    numberOfTaskSlots: 4
  deployHash: d9f8a6a8
  failedDeployHash: ""
  jobStatus:
    completedCheckpointCount: 0
    entryClass: org.apache.flink.WordCount
    failedCheckpointCount: 0
    health: Green
    jarName: wordcount-operator-example-1.0.0-SNAPSHOT.jar
    jobID: acd232a002dd5204669d1041736b8fa0
    jobRestartCount: 0
    lastCheckpointTime: null
    lastFailingTime: null
    parallelism: 3
    restorePath: ""
    restoreTime: null
    startTime: 2019-07-30T07:35:59Z
    state: FINISHED
  lastSeenError: null
  lastUpdatedAt: 2019-07-30T07:36:09Z
  phase: Running
  retryCount: 0

To check events for the FlinkApplication object, run the following command:

$ kubectl describe flinkapplication.flink.k8s.io -n flink-operator wordcount-operator-example

This will show events similarly to the following:

Events:
  Type    Reason           Age   From              Message
  ----    ------           ----  ----              -------
  Normal  CreatingCluster  4m    flinkK8sOperator  Creating Flink cluster for deploy d9f8a6a8
  Normal  JobSubmitted     3m    flinkK8sOperator  Flink job submitted to cluster with id acd232a002dd5204669d1041736b8fa0