Kafka-node-avro

A wrapper that combine node-rdkafka and avsc packages together in an easy to reason about API.

Installation

npm install @kaluza/kafka-node-avro

As this ultimately uses rdkafka, a C library, your system needs to be set up to build that library.

Mac OS

For the library to work on mac os you will need to add 2 flags to your ~/.bash_profile see installation instructions from node-rdkafka:

export CPPFLAGS=-I/usr/local/opt/openssl/include
export LDFLAGS=-L/usr/local/opt/openssl/lib

Make sure Python point to a version 2.. and openssl is installed correcly on the machine.

Linux

It should just work, but if not make sure the following are installed:

• Python
• Make
• gcc
• g++

Windows

For Windows, run npm install -g windows-build-tools first, then install kafka-node-avro.

If you have multiple python versions installed, you need to make sure that python in your PATH resolves to a Python 2 version, otherwise the build of rdkafka will fail

Examples

• Consumer:

const { consumer } = await createClients({
  host: kafka.host,
  schemaRegistry: kafka.schemaRegistry,
  ca,
  cert,
  key
});

await consumer.subscribe(
  ['OUTPUT_TEST_TOPIC'],
  { groupId: 'kafka-node-example-consuming-group' },
  (message) => {
    console.log(message);
  },
  (err) => {
    console.error(err);
  }
);

// Will stop listening to topic
consumer.unsubscribe();

• Producer:

const { producer } = await createClients({
  host: kafka.host,
  schemaRegistry: kafka.schemaRegistry,
  ca,
  cert,
  key
});

await producer.registerSchema('TEST_OUTPUT_TOPIC', [
  {
    default: null,
    name: 'name',
    type: ['null', 'string']
  }
]);

const response = await producer.sendMessage({
  topic: 'TEST_OUTPUT_TOPIC',
  messages: {
    name: 'Bender Bending Rodriguez'
  }
});

console.log(response);

API Documentation

Maintainers

Install dependencies:

yarn

Run build:

yarn build

Run tests:

yarn test

Run examples:

• create a certificates folder in example/certificates with the following files ca.pem, service.key, service.cert
• create a config.json file with the following content:

{
  "kafka": {
    "host": "HOST_NAME_HERE",
    "schemaRegistry": "SCHEMA_REGISTRY_HERE",
    "queues": [
      {
        "topic": "TOPIC_NAME",
        "partition": 0,
        "offset": 0
      }
    ],
    "queuesGroup": ["TOPIC_NAME"]
  }
}

• Run script cd example && node produce-message-to-topic.js for example