Devbox 1.3.17 on k8s 1.21.1 / containerd rpi3/4

This is my experience on running devbox 1.3.17 arm32v8 on k8s 1.21.1 with containerd runtime 1.3.7/1.4.4 on rpi3 / rpi4 armv7l / aarch64 RaspberryPI OS 32 buster.

The cluster consists of 3 masters rpi4 4GB aarch64, 1 rpi4 8GB aarch64, 2 rpi4 4GB aarch64, 1 rpi3 aarch64, 5 rpi3 armv7l.
The rpis have alls RaspberryPI OS 32 buster. The rpis with aarch64 have /boot/config.txt arm_64bit=1, with this configuration the rpi can run 64 bits arm code, the OS can remain 32 bits, obviouvsly the program to run must be statically linked or a container image.

Steps:

  1. There are multi arch nodes and so I have labeled they with arch=armv7l or arch=aarch64 depending on node type
  2. [optional] create a development namespace, I assume this in the next configurations
  3. create a configmap with a waterstream.license
  4. create a statefulset for the devbox
  5. publish the ports so can I play with devbox from every home computer :smiley:

Currently the devbox pod are in a rpi4 4GB and it run seamless.

The only problem encountered with this hardware / OS is that the JVM OpenJDK16 crashes without any feedback when running in k8s. No problem if the image run on containerd without k8s.
The problem IMHO is in the handling of cgroupsv1. The crash is in initializing jvm code when is enabled the container javavm support that is enabled by default on recently jvm. It can be disabled with -XX:-UseContainerSupport jvm flag. I’ll investigate.

--- yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: waterstream-devbox
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  serviceName: waterstream-devbox
  replicas: 1
  selector:
    matchLabels:
      app: waterstream-devbox
  template:
    metadata:
      labels:
        app: waterstream-devbox
    spec:
      nodeSelector:
        arch: aarch64
      containers:
      - image: docker.io/simplematter/waterstream-kafka-devbox-arm64v8:1.3.17
        imagePullPolicy: IfNotPresent
        name: waterstream-devbox
        env:
        - name: MQTT_WS_PORT
          value: "9001"
        - name: KAFKA_OPTS
          value: "-XX:-UseContainerSupport"
        - name: WATERSTREAM_JAVA_OPTS
          value: "-XX:-UseContainerSupport"
        securityContext:
          readOnlyRootFilesystem: false
        ports:
        - containerPort: 9001
          name: mqtt-ws-port
        - containerPort: 8080
          name: http
        - containerPort: 1884
          name: monitoring-port
        - containerPort: 2181
          name: zk-port
        - containerPort: 9092
          name: kafka-port
        - containerPort: 1883
          name: mqtt-port
        volumeMounts:
        - mountPath: /etc/waterstream.license
          name: waterstream-license
          subPath: waterstream.license
        #- mountPath: /tmp/kafka-logs
        #  name: kafka-logs
        #  readOnly: false
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
      volumes:
      - name: waterstream-license
        configMap:
          name: waterstream-license
      #- name: kafka-logs
      #  emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: waterstream-devbox
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  selector:
    app: waterstream-devbox
  ports:
  - name: mqtt-ws-port
    port: 9001
  - name: http
    port: 8080
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: waterstream-devbox-np
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  selector:
    app: waterstream-devbox
  ports:
  - name: mqtt-ws-port
    port: 9001
    nodePort: 31001
    protocol: TCP
  - name: http
    port: 8080
    nodePort: 31080
    protocol: TCP
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: waterstream-license
  namespace: development
data:
  waterstream.license: |+ <embed license here>

Found the problem, my mistake! You can safely drop the two env properties KAFKA_OPTS and WATERSTREAM_JAVA_OPTS.

--- yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: waterstream-devbox
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  serviceName: waterstream-devbox
  replicas: 1
  selector:
    matchLabels:
      app: waterstream-devbox
  template:
    metadata:
      labels:
        app: waterstream-devbox
    spec:
      nodeSelector:
        arch: aarch64
      containers:
      - image: docker.io/simplematter/waterstream-kafka-devbox-arm64v8:1.3.17
        imagePullPolicy: IfNotPresent
        name: waterstream-devbox
        env:
        - name: MQTT_WS_PORT
          value: "9001"
        securityContext:
          readOnlyRootFilesystem: false
        ports:
        - containerPort: 9001
          name: mqtt-ws-port
        - containerPort: 8080
          name: http
        - containerPort: 1884
          name: monitoring-port
        - containerPort: 2181
          name: zk-port
        - containerPort: 9092
          name: kafka-port
        - containerPort: 1883
          name: mqtt-port
        volumeMounts:
        - mountPath: /etc/waterstream.license
          name: waterstream-license
          subPath: waterstream.license
        #- mountPath: /tmp/kafka-logs
        #  name: kafka-logs
        #  readOnly: false
        resources:
          limits:
            cpu: 2
            memory: 2Gi
          requests:
            cpu: 500m
            memory: 1Gi
      volumes:
      - name: waterstream-license
        configMap:
          name: waterstream-license
      #- name: kafka-logs
      #  emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: waterstream-devbox
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  selector:
    app: waterstream-devbox
  ports:
  - name: mqtt-ws-port
    port: 9001
  - name: http
    port: 8080
  clusterIP: None
---
apiVersion: v1
kind: Service
metadata:
  name: waterstream-devbox-np
  namespace: development
  labels:
    app: waterstream-devbox
spec:
  selector:
    app: waterstream-devbox
  ports:
  - name: mqtt-ws-port
    port: 9001
    nodePort: 31001
    protocol: TCP
  - name: http
    port: 8080
    nodePort: 31080
    protocol: TCP
  type: NodePort
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: waterstream-license
  namespace: development
data:
  waterstream.license: |+ <embed license here>
1 Like

Thanks for this contribution @vescoc!