SSL/TLS configuration problems

Hello!

We’re having issues enabling SSL/TLS on waterstream-kafka-devbox.
Container is created via Docker Hub, SSL/TLS configuration added based on Waterstream Docker Configuration — Waterstream documentation 1.3.14 documentation

docker run $INTERACTIVITY $CLEANUP \
     -e MQTT_WS_PORT=$MQTT_WS_PORT \
     -v $LICENSE_FILE:/etc/waterstream.license:ro \
     -p $HTTP_PORT:80 \
     -p $MONITORING_PORT:1884 \
     -p $MQTT_PORT:1883 \
     -p $MQTT_WS_PORT:$MQTT_WS_PORT \
     -p $KAFKA_PORT:9092 \
     -p $ZK_PORT:2181 \
     -v $KEY_FILE:/etc/waterstream.key:ro \
     -v $CRT_FILE:/etc/waterstream.crt:ro \
     -e SSL_ENABLED="true" \
     -e SSL_KEY_PATH=/etc/waterstream.key \
     -e SSL_CERT_PATH=/etc/waterstream.crt \
     --name $CONTAINER_NAME $IMAGE_NAME

The added parts are:

 -v $KEY_FILE:/etc/waterstream.key:ro \
 -v $CRT_FILE:/etc/waterstream.crt:ro \
 -e SSL_ENABLED="true" \
 -e SSL_KEY_PATH=/etc/waterstream.key \
 -e SSL_CERT_PATH=/etc/waterstream.crt \

Setup seems to fail with SSL_ENABLED=‘true’ added, so I would assume the problem is in unsuitable cert files (these are dummy certs generated for testing purposes, private key and the public cert), but having tried various paths and different dummy certs, I have not been able to get it running successfully.

As I could not find any meaningful errors from the the container creation process and there does not seem to be much documentation regarding any potential constraints/requirements, we’re a bit stuck at this point.

Any ideas what could be wrong with the script, or where one would look for more verbose logging?

Hello @suislab. Could you please tell us what problem exactly do you encounter with SSL and how did you try to connect?
If you connect via WebSockets using a built-in MQTT Board then you need to add the certificate of your Certificate Authority to the list of the trusted certificates in your browser. In my experience, this is problematic in Chrome. It’s far easier in Firefox.

If you have a Mosquitto MQTT client installed on your machine you can send or receive a test message with the CLI:

mosquitto_pub -h localhost -p 1883 -t "sample_topic" -i client_1 -q 0 -m "Hello, world!" \
    --cafile <your CA certificate>

mosquitto_sub -h localhost -p 1883 -t "#" -i client_2 -q 0 -v \
    --cafile <your CA certificate> 

As for the requirements for the key - it can be:

  • a non-encrypted private key in PKCS8 format wrapped in a PEM block
  • or non-encrypted private key in PKCS1 format wrapped in a PEM block

A certificate file must contain an X.509 certificate wrapped in a PEM block.
See PemKeyCertOptions (Vert.x Stack - Docs 4.0.3 API) for the details.

Hi @paul.lysak, thanks for the reply.

Problem is that “docker run” / initial setup itself fails, it does not get as far as being able to connect.

The private key and public cert were generated with “/etc/pki/tls/certs/make-dummy-cert”, which creates a .pem that splits into the key/cert. Key/cert match and everything seems to be good according to the requirements you provided.

The initialization logs don’t provide much meaningful information (screenshot below).

Docker container initialization logs show the following: (screenshot below)
2021-05-19T15:00:16.903027800+03:00 container die 29b4692777293f6b39d11af762beec13b92dd66e503bb758c93319a3f9782d09 (desktop.docker.io/binds/0/Source=C:\Users\Ergo\waterstream.crt, desktop.docker.io/binds/0/Sou
rceKind=hostFile, desktop.docker.io/binds/0/Target=/etc/waterstream.crt, desktop.docker.io/binds/1/Source=C:\Users\Ergo\waterstream.license, desktop.docker.io/binds/1/SourceKind=hostFile, desktop.docker.io/bin
ds/1/Target=/etc/waterstream.license, desktop.docker.io/binds/2/Source=C:\Users\Ergo\waterstream.key, desktop.docker.io/binds/2/SourceKind=hostFile, desktop.docker.io/binds/2/Target=/etc/waterstream.key, exitC
ode=1, image=simplematter/waterstream-kafka-devbox:latest, name=waterstream-kafka-devbox)
2021-05-19T15:00:17.205934500+03:00 network disconnect 5a5f7549b83d3377d88010abb370e3634c5184d9f83a758830245af83673c83d (container=29b4692777293f6b39d11af762beec13b92dd66e503bb758c93319a3f9782d09, name=bridge,
type=bridge)

I’m wondering if the problem could be that it runs some authority validation on the certs before allowing them to be used?

Other than using dummy certs, everything else seems to be in order from what I can tell.

Complete log of the Waterstream Devbox container would be helpful, if you can get me one…
Here are the things I can suggest:

  • Which of the following messages do you see in the container logs: “Starting Kafka”, “Creating Kafka topics”, “Starting MQTT Board”, “Starting Waterstream”, “Devbox script complete”?
  • Try to increase the memory available for the container
  • In most recent version 1.3.16 we have improved error logging somewhat, try to run it - simplematter/waterstream-kafka-devbox:1.3.16 and tell me if you see any new information in the logs
  • To cross out the certificate issues - try to use test key and certificate that I’ve used today: Waterstream test certificates - Google Drive (broker_test.key and broker_test.crt)
  • This could also be Windows-specific. We’ll try to run this on Windows, meanwhile you can try to run on Linux or MacOS machine

Hi again.

Digging a bit deeper into container logs as you suggested, I found very odd mount paths from what I assume is the Waterstream jar:

Exception in thread "main" io.vertx.core.VertxException: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /opt/waterstream-kafka/C:/Program Files/Git/etc/waterstream.key
        at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:303)
        at io.vertx.core.net.impl.SSLHelper.getContext(SSLHelper.java:478)
        at io.vertx.core.net.impl.SSLHelper.validate(SSLHelper.java:503)
        at io.vertx.core.net.impl.NetServerImpl.listen(NetServerImpl.java:186)
        at io.vertx.core.net.impl.NetServerImpl.listen(NetServerImpl.java:330)
        at io.vertx.core.net.impl.NetServerImpl.listen(NetServerImpl.java:346)
        at io.vertx.a.a.e.a(Unknown Source)
        at io.waterstream.d.a(Unknown Source)
        at io.waterstream.q.invokeSuspend(Unknown Source)
        at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith(ContinuationImpl.kt:33)
        at kotlinx.coroutines.DispatchedTask.run(DispatchedTask.kt:56)
        at kotlinx.coroutines.EventLoopImplBase.processNextEvent(EventLoop.common.kt:274)
        at kotlinx.coroutines.BlockingCoroutine.joinBlocking(Builders.kt:84)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking(Builders.kt:59)
        at kotlinx.coroutines.BuildersKt.runBlocking(Unknown Source)
        at kotlinx.coroutines.BuildersKt__BuildersKt.runBlocking$default(Builders.kt:38)
        at kotlinx.coroutines.BuildersKt.runBlocking$default(Unknown Source)
        at io.waterstream.AppKt.main(Unknown Source)
Caused by: io.vertx.core.file.FileSystemException: java.nio.file.NoSuchFileException: /opt/waterstream-kafka/C:/Program Files/Git/etc/waterstream.key
        at io.vertx.core.file.impl.FileSystemImpl$16.perform(FileSystemImpl.java:865)
        at io.vertx.core.file.impl.FileSystemImpl$16.perform(FileSystemImpl.java:857)
        at io.vertx.core.file.impl.FileSystemImpl.readFileBlocking(FileSystemImpl.java:297)
        at io.vertx.core.net.impl.KeyStoreHelper.create(KeyStoreHelper.java:100)
        at io.vertx.core.net.KeyCertOptions.getKeyManagerFactory(KeyCertOptions.java:53)
        at io.vertx.core.net.impl.SSLHelper.getKeyMgrFactory(SSLHelper.java:308)
        at io.vertx.core.net.impl.SSLHelper.createContext(SSLHelper.java:261)
        ... 17 more
Caused by: java.nio.file.NoSuchFileException: /opt/waterstream-kafka/C:/Program Files/Git/etc/waterstream.key
        at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:106)
        at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
        at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:375)
        at java.base/java.nio.file.Files.newByteChannel(Files.java:426)
        at java.base/java.nio.file.Files.readAllBytes(Files.java:3272)
        at io.vertx.core.file.impl.FileSystemImpl$16.perform(FileSystemImpl.java:861)
        ... 23 more

I tested this both on Windows (with wsl2 Docker and Git bash to run the script) and on an EC2 linux instance.

Solution is to accept these constraints and mount the certs to the predicted “/opt/waterstream-kafka/” directory and set the path as just the filename.

Eg, change following:

 -v $KEY_FILE:/etc/waterstream.key:ro \
 -v $CRT_FILE:/etc/waterstream.crt:ro \
 -e SSL_ENABLED="true" \
 -e SSL_KEY_PATH=/etc/waterstream.key \
 -e SSL_CERT_PATH=/etc/waterstream.crt \

into

 -e SSL_ENABLED="true" \
 -e SSL_KEY_PATH="waterstream.key" \
 -e SSL_CERT_PATH="waterstream.crt" \
 -v $KEY_FILE:/opt/waterstream-kafka/waterstream.key:ro \
 -v $CRT_FILE:/opt/waterstream-kafka/waterstream.crt:ro \

The container comes up now and seems to work without problems. Looks to me like these constraints exist regardless if the system is Windows or Linux, so would be good if you could confirm this and if possible either allow the constraints to be overriden by the env variable or have this added to the documentation.

java.nio.file.NoSuchFileException: /opt/waterstream-kafka/C:/Program Files/Git/etc/waterstream.key can happen if you have SSL_KEY_PATH environment variable set to C:/Program Files/Git/etc/waterstream.key. It doesn’t start from / and is, therefore, considered a relative path. Can you double-check what are the actual environment variables in the container you run? Something like this:

docker run -ti ...  --name waterstream-devbox simplematter/waterstream-kafka-devbox:1.3.16 env

As for the volume mounting - a colon in Windows-style path may interfere with Docker volume mounting syntax. Try wrapping its components in quotes or using --mount (Use bind mounts | Docker Documentation)