A while ago, in a project, I found that it is necessary to encrypt messages to be exchanged between producers and consumers kafka, because, even if you set up an https channel, that does not protect you completely. On that site they used a proprietary library, for my taste, very complicated, many classes, very obscure. Obviously they weren’t using something open source that had gone through the review process. When I left, I decided that something like that can be achieved with open source software.
Adding an extra layer of security, such as ensuring that a consumer can read such information, is a must in environments where scrupulous compliance with the law is required. It is achieved by creating public/private key pairs between sender and receiver, so that if someone needs to read what you are going to write, you will ask them to share their public key with you.
Then, you would need to use something like this, adapt it into classes that implement the interfaces I describe below and use them in the producer and consumer configuration . In the example I’m using AVRO, although you could use Protobuf or any other binary format.
In a real environment, the keys would be stored on a secure server, they would never be in the library’s resource folder, nor would they be stored in some temporary folder indefinitely. It is brought in, loaded into the process, deleted, and periodically new pairs of passwords are generated and would be delivered method some authorized secure procedure.
You can see that the library that does the hard work is JPGPJ, an adaptation of Bouncy Castle.
Then, to add this functionality to be used with something like Kafka, we would have to start reading something like this. We see that we will have to create a class in charge of producing our messages and doing the encryption before sending the message to the topic. The same for the consumer.
Perhaps using this in a kafka environment or another type of consumer producer technology is unnecessary because from a security point of view the traffic is encrypted using TLS, but nothing prevents a pirate consumer from pointing to that topic and starting to consume information. If someone were to puncture the joint where the information passes, they would see encrypted traffic. To do this, in theory, you only have to run a kafka consumer that is on the same network where the cluster can attend to requests. Something like this would only be done to prevent this situation, so that there are consumers and producers authorized to talk to each other.
The truth is that I find it strange that Confluent does not have such functionality implemented. I ask them, but they don’t respond. I guess they will have more important things to do on the Road Map.
Anyways, happy weekend and stay safe, stay home.
This is a translation of this thread