Overview
Kafka Connection Manager is SSIS Connection Manager for establishing connections with Apache Kafka (version 0.8 or and newer). It can be used with Message Queue Task Plus.
Quick Start
In this section we will show you how to set up a Kafka Connection Manager to access an Apache Kafka broker.
Add Kafka Connection Manager
- Right-click on the Connection Manager canvas.
- Select New Connection.
- When the list of connection managers appears, expand the Type column, if necessary, so you can see the complete names of the connection managers.
- Scroll down to find KAFKA and select it.
Configure Basic connectivity settings
- Once the connection manager dialog opens in General tab enter Broker Server(s).
The value would look like:pkc-419q3.us-east4.gcp.confluent.cloud:9092 - Enter Authentication.
- Select SASL SSL for Security Protocol .
- Select Plain for SASL Mechanism.
- Enter Username.
- Enter Password.
Configure Advanced connectivity settings
- In tab Advanced tab enter Additional Options. The syntax of Additional Options is
name=value1[;name2=value2]...[;nameN=valueN]. The value would look like:reconnect.backoff.ms=1000;retry.backoff.ms=50.You can see available options in Available Options text field.
Verify connectivity
- Click on Test Connection.
- When you see Test connection succeeded, click OK.
- Click OK to close the connection manager dialog.
Congratulations! You have successfully configured a Kafka Connection Manager.
Parameters
General
Use the General page of the Kafka Connection Manager dialog to specify properties for connecting to the Kafka broker
The broker server name(s) and an optional port (default is 9092) using the syntax
host[:port](e.g.localhost:9092). When connecting to bootstrap servers set, its members should be enumerated like this:host1[:port],host2[:port]...,hostN[:port].The security protocol:
Value Description Plaintext Don't use any authentication or encryption method. SSL Authenticate via SSL certificate SASL Plaintext Authenticate by unencrypted username and password. SASL SSL Authenticate with a username and password that are encrypted with an SSL certificate. The SASL mechanism:
Value Description GSSAPI Use Generic Security Service Application Program Interface. Plain Use unencrypted authentication mechanism SCRAM-SHA-256 Use Salted Challenge Response Authentication Mechanism with SHA-256 encryption. SCRAM-SHA-512 Use Salted Challenge Response Authentication Mechanism with SHA-512 encryption. OAuth Bearer Use OAuth 2 Authorization Framework. The username to authenticate with (SASL Plaintext, SASL SSL security protocol when using Plain, SCRAM-SHA-256, SCRAM-SHA-512 SASL mechanism).
The password to authenticate with (SASL Plaintext, SASL SSL security protocol when using Plain, SCRAM-SHA-256, SCRAM-SHA-512 SASL mechanism).
Тhe timeout to attempt a connection before timing out.
SSL
Path to client's private key (PEM) used for authentication.
The passphrase for the client certificate private key
The location of the client certificate PEM file.
The passphrase for the client certificate key store
Path to client's PKCS#12 keystore used for authentication (usually a .pfx file that is generated from a .jks file).
File or directory path to CA certificate(s) for verifying the broker's key. If empty, the default Windows Certificate Store is used
Specify whether to check for certificate revocation
Path to CRL for verifying broker's certificate validity.
Logging
Specify a log file
Specify the scope for the logging. Either use
allto log everything or specify a comma-separated list of debug contexts:generic,broker,topic,metadata,feature,queue,msg,protocol,cgrp,security,fetch,interceptor,plugin,consumer,admin,eos,mock,assignor,conf,telemetry
Advanced
Use the Advanced page of the Kafka Connection Manager dialog to specify advanced options
Specify additional connection options, when necessary, via the syntax:
name=value1[&name2=value2]...[&nameN=valueN]. Refer to Kafka documentation for detailed explanation of each option.
Knowledge Base
- Where can I find the documentation for the Kafka Connection?
- Does the Kafka connector work with Kafka that is backended by Azure Event Hub?
What's New
- Fixed: Message not delivered if package execution completed quickly (Thank you, Dotti).
- New: Additional SSL parameters.
- New: Logging options.
- New: Introduced connection.
Related documentation
COZYROC SSIS+ Components Suite is free for testing in your development environment.
A licensed version can be deployed on-premises, on Azure-SSIS IR and on COZYROC Cloud.