Monitor the internal state of Java programs with Kubernetes

Introduction

This article is a reminder of how Kubernetes checks the status of a particular Java class at regular intervals and kills the pod if there is a problem. For more information on Kubernetes Liveness Probe, please see here (https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-probes/#define-a-tcp-liveness-probe). ..

Java implementation

Monitored class

For testing purposes, implement ʻisAlive () == false` 10 seconds after the instance is created.

SomeResource.java

public class SomeResource {
  final long createdTime;

  public SomeResource() {
    this.createdTime = System.currentTimeMillis();
  }
  public boolean isAlive() {
    return System.currentTimeMillis() - createdTime < 10000;
  }
}

Monitoring endpoint

Implement to return the response code 200 when SomeResource # isAlive () == true and 500 when false.

HeartBeat.java

import com.sun.net.httpserver.HttpServer;
import java.io.IOException;
import java.net.InetSocketAddress;

public class HeartBeat {
  private final SomeResource target;
  private HttpServer httpServer;
  private final int port;

  public HeartBeat(SomeResource target, int port) {
    this.target = target;
    this.port = port;
  }

  void start() throws IOException {
    httpServer =
        HttpServer.create(new InetSocketAddress(port), 0);
    httpServer.createContext("/heartbeat", httpExchange -> {
      int responseCode;
      if (target.isAlive()) {
        responseCode = 200;
      } else {
        responseCode = 500;
      }
      byte[] responseBody = String.valueOf(target.isAlive()).getBytes(StandardCharsets.UTF_8);
      httpExchange.getResponseHeaders().add("Content-Type", "text/plain; charset=UTF-8");
      httpExchange.sendResponseHeaders(responseCode, responseBody.length);
      httpExchange.getResponseBody().write(responseBody);
      httpExchange.close();
    });
    httpServer.start();
  }

  void stop() {
    httpServer.stop(0);
  }
}

Main class

Main.java

import java.util.concurrent.CountDownLatch;

public class Main {

  public static void main(String[] args) {
    final CountDownLatch latch = new CountDownLatch(1);
    SomeResource resource = new SomeResource();
    HeartBeat heartbeat = new HeartBeat(resource, 1234);//Use port 1234

    Runtime.getRuntime().addShutdownHook(new Thread() {
      @Override
      public void run() {
        heartbeat.stop();
        latch.countDown();
      }
    });

    try {
      heartbeat.start();
      latch.await();
    } catch (Exception e) {
      e.printStackTrace();
      System.exit(-1);
    }
  }
}

Test endpoints with curl

Immediately after creating a SomeClass instance

console

$ curl --dump-header - http://localhost:1234/heartbeat
HTTP/1.1 200 OK
Date: Tue, 02 Jul 2019 07:24:20 GMT
Content-type: text/plain; charset=UTF-8
Content-length: 4

true

10 seconds later

console

$ curl --dump-header - http://localhost:1234/heartbeat
HTTP/1.1 500 Internal Server Error
Date: Tue, 02 Jul 2019 07:24:30 GMT
Content-type: text/plain; charset=UTF-8
Content-length: 5

false

Deploy to Kubernetes

The steps to create a container and push to the registry are omitted.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: java-resource-watch-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: java-resource-watch-test
  template:
    metadata:
      labels:
        app: java-resource-watch-test
    spec:
      containers:
      - name: java-resource-watch-test
        image: some_registry/test:latest
        livenessProbe:
          httpGet:
            path: /heartbeat
            port: 8080
            httpHeaders:
            - name: Custom-Header
              value: HealthCheck
          initialDelaySeconds: 1
          periodSeconds: 1

kubectl apply -f ./deployment.yaml

You can see that the pod ends and restarts as soon as the SomeResource expires.

Summary

I was able to restart the pod depending on the status of the class and thread. There are various other methods such as a sidecar model using JMX, but I think the method introduced in this section is easy and flexible.