Node-RED Kafka nodes: Send, Receive, and Schema validation with modern KafkaJS
npm install @oriolrius/node-red-contrib-kafkaKafka Consumer and Producer
This Node-RED Kafka client is based on the original node-red-contrib-kafka implementations but has been completely modernized and enhanced. We have migrated from the legacy kafka-node library to the modern kafkajs library for better performance, reliability, and active maintenance support.
- Modern Library: Uses kafkajs instead of the deprecated kafka-node
- Enhanced SASL Support: Better SASL authentication with support for PLAIN, SCRAM-SHA-256, and SCRAM-SHA-512 mechanisms
- Improved SSL/TLS: Enhanced certificate handling and SSL configuration options
- Better Error Handling: More detailed error reporting and debugging capabilities
- Active Maintenance: Built on actively maintained libraries for long-term reliability
This node can be used to produce and consume messages to/from Kafka. It consists of five nodes:
- Kafka Broker (oriolrius-kafka-broker) - Connection configuration
- Kafka Send (oriolrius-kafka-producer) - Send messages to Kafka topics
- Kafka Receive (oriolrius-kafka-consumer) - Receive messages from Kafka topics
- Kafka Schema Send (oriolrius-kafka-producer) - Send messages with Avro schema validation
- Kafka History Reader (oriolrius-kafka-history-reader) - Retrieve historical messages by type
NEW: The oriolrius-kafka-producer node adds Avro schema validation support using Confluent Schema Registry. This node:
- Validates message payloads against registered Avro schemas
- Supports both latest and specific schema versions for production stability
- Supports automatic schema registration if schema doesn't exist
- Provides schema-only validation mode for testing
- Integrates with Confluent Schema Registry authentication
- Ensures data consistency and compatibility across your Kafka ecosystem
- Version-aware caching: Automatically fetches new schemas when version changes
NEW: The oriolrius-kafka-history-reader node allows you to retrieve historical messages of specific types from Kafka topics. This is particularly useful for:
- Getting the last known values of specific message types before starting real-time consumption
- Initializing your application state with historical data
- Implementing catch-up mechanisms for offline periods
- Debugging and analyzing message patterns
Key features:
- Type-specific filtering: Search for messages by type (checks type, messageType, eventType, kind, msgType fields)
- Configurable limits: Set maximum number of messages to retrieve per type
- Offset control: Start reading from earliest or latest available messages
- Schema Registry support: Works with both plain JSON and Avro-encoded messages
- Non-destructive: Uses temporary consumer groups to avoid affecting your main consumers
This library provides comprehensive SASL authentication support based on the kafkajs project. It now supports:
- PLAIN - Basic username/password authentication
- SCRAM-SHA-256 - Secure password authentication with SHA-256
- SCRAM-SHA-512 - Secure password authentication with SHA-512
These enhanced authentication mechanisms provide better security compared to the original kafka-node implementations.
npm install @oriolrius/node-red-contrib-kafka
Documentation
This project includes comprehensive documentation to help you get started and understand all features:
$3
- Schema Guide - Complete guide to using Avro schema validation and version management with the Schema Producer node
- Kafka History Reader Guide - Complete guide to using the Kafka History Reader for retrieving historical messages
- Migration Guide - Guide for migrating from legacy kafka-node based implementations
- IoT Integration Guide - Complete guide to IoT cloud configuration features
$3
- Implementation Summary - Technical overview of the project's architecture and components
- Debug Guide - Troubleshooting and debugging information for common issues
$3
- Schema Producer Status Improvements - Details about status reporting enhancements in the Schema Producer
- Node Names Improvement - Information about node naming conventions and improvements
License
This project is licensed under the MIT License - the same license as the original Node-RED Kafka projects that served as its foundation.
Acknowledgments
This project is based on the excellent work by emrebekar and the original node-red-contrib-kafka-client project. We extend our sincere gratitude to emrebekar for creating the foundational Node-RED Kafka integration that served as the basis for this modernized implementation.
The original project provided the core architecture and functionality for Kafka producer and consumer nodes in Node-RED, which we have enhanced and migrated from the legacy
library to the modern kafkajs` library for improved performance and reliability.