SSL

[tigran-a]

Hello,

Any chance of implementing SSL support?
https://cwiki.apache.org/confluence/display/KAFKA/Deploying+SSL+for+Kafka

Thanks for the software!

[dspeterson]

SSL support would be a welcome addition to Bruce, but I imagine it will be a while before I get a chance to implement it. In the meantime, if you or someone else wants to implement the feature, it would be a nice contribution.

[matsbror]

I'd like to start looking at this. First a couple of questions:

• I could use either raw openssl API or use some class library, e.g. Boost asio. Do you have any preference of which to choose? If I would have done it myself, I would have used Boost asio, but you use native system calls for the current socket communication so I though I should ask. I could replace the current INET sockets implementation to StreamSockets while I'm at it.
• It's only the communication with Kafka that needs to be protected. As far as I have been able to tell, the communication is confined to the following functions and files:
  • NewCompatSocket in cursor.cc (socket)
  • ConnectToHost in connect_to_host.cc (connect)
  • SendOneProduceRequest in sender.cc (send)
  • WaitForShutDownAck in sender.cc (poll)
  • WaitForSendReady in sender.cc (poll)
  • DoRun in sender.cc (poll)
  • WaitForShutDownAck in receive.cc (poll)
  • DoSockRead in receive.cc (recv)
  • DoRun in receive.cc (poll)

Would this cover it all (when it comes to streaming Internet sockets, that is, local socket communication is not affected)?

I will surely come back with questions when it comes to designing tests, but that's a later story.

[dspeterson]

Yes that would be great if you're interested in adding SSL support. Thanks very much! I'll get back to you soon (hopefully by some time tomorrow) with answers to your questions.

[matsbror]

Great! Also, in order to run the mock kafka server, what do you need to do? I was thinking of testing the SSL support by using that and run stud (https://github.com/bumptech/stud) in front of it.

[dspeterson]

For now, I prefer using the raw openssl API. Eventually it may be good to look into using a class library like Boost asio, but I think it would involve more refactoring than I'm prepared for at the moment. The communication with Kafka is pretty self-contained, all occurring in sender.cc, receiver.cc and utility code called from those two source files. It looks like your list of places in the code covers everything.

[dspeterson]

I'll get back to you shortly with instructions for running the mock kafka server. Using stud in front of it sounds like a good way to implement SSL-related unit tests. The mock kafka server is an ugly duckling. I wrote it under extreme time pressure, and it's kind of a mess. Someday maybe I'll rewrite it, if I can find the time. Reimplementing the mock kafka server using Boost asio or something similar might be a good trial run for making similar changes in Bruce, but that's a whole different project. Thanks again for offering to help!

[matsbror]

Thanks! Then I go forward from that. I'll probably use the OpenSSL BIO abstraction layer which makes it even closer to normal socket operation.

[dspeterson]

Also please be sure to fill out a contributor license agreement as described here and send it to if(we). They need it before I can merge your code. I apologize for the inconvenience - my former employer insists on it. If it were up to me, I wouldn't bother people with legal paperwork.

[dspeterson]

The mock Kafka server simulates a Kafka broker cluster by creating a number of threads, each thread simulating a broker. It can run as either a standalone executable, or part of a unit test. For an example of running it from a unit test, take a look at src/bruce/bruce.test.cc.

When running in standalone mode, the mock Kafka server requires a config file. When running from a unit test, you can specify the config file contents directly as a vector of strings, each string representing a line of file content, as done by CreateKafkaConfig() in the above-mentioned unit test. The file format is described in the big comment in src/bruce/mock_kafka_server/setup.h, with further explanation provided in a comment in src/bruce/mock_kafka_server/port_map.h.

As illustrated in the unit test, you can send commands to the mock Kafka server that cause it to simulate various error conditions, or add artificial delays before reading produce requests or sending produce responses. When running it in standalone mode, there is another executable you can run that will send it error injection commands.

Although you will probably be more interested in running the mock Kafka server from a unit test, if you want to try running it in standalone mode, you can run it with a shell command such as:

out/debug/bruce/mock_kafka_server/mock_kafka_server --output_dir \
    ~/mk_out --setup_file ~/mock_kafka_setup --log_echo

The above assumes that you have built the mock Kafka server executable, and are running it from the root of the source tree. It also assumes that you have created an empty directory ~/mk_out where it will write logfiles as it runs. The config file, ~/mock_kafka_setup, is in the above-mentioned format. Building the standalone mock Kafka server and error injector executables (see src/bruce/mock_kafka_server/mock_kafka_server.cc and src/bruce/mock_kafka_server/inject_error/inject_error.cc) can be done in the same manner as building Bruce, which is documented here.

Let me know if you have more questions, or run into problems getting things working.

Thanks,
Dave

[mgimelfarb]

Just wanted to check back to see if there was any progress on the SSL support.

[dspeterson]

matsbror has an SSL patch that I believe he's been using for a while. I'll ping him and see if he's ready to merge it.

[mgimelfarb]

Thank you, @dspeterson. Be glad to give it a try.

[dspeterson]

@mgimelfarb I haven't yet heard from @matsbror but I believe you can find his code changes in https://github.com/matsbror/bruce in a branch named addssl.

[mgimelfarb]

Will do and will provide comments. Thank you, @dspeterson and @matsbror.

[matsbror]

Hi, I haven't been able to work on it for a while. I got sidetracked at work. As far as I remember, Dave found a bug in Kafka 0.9 necessitating the explicit setting of client_id when starting Bruce. My code should be ready for testing with Kafka (it passes the mock kafka tests) but be warned, there are some debug printouts left and the code probably needs some cleanup.
By Mid March, I should be able to work on it again.

[dspeterson]

Cool, thanks for the update!

[mgimelfarb]

Thank you, @matsbror. I noticed that your repo was 43 commits behind @dspeterson 's master, so I wanted to be sure before I start putting the code through its paces on whether there is a chance for a catchup or at least a PR into master looming on the horizon.