1. Sources
  2. Kafka

Basic config

connector_type: kafka
bootstrap_server: "localhost:9092"
topic: topic_name
consumer_group: unique_consumer_group
api_version: "0.10.2"
batch_size: 100

SSL authentication

security_protocol: "SSL"
  cafile: "CARoot.pem"
  certfile: "certificate.pem"
  keyfile: "key.pem"
  password: password
  check_hostname: true

SASL authentication

security_protocol: "SASL_SSL"
  mechanism: "PLAIN"
  username: username
  password: password

Deserialize message with protobuf schema

  serialization_method: PROTOBUF
  schema_classpath: "path.to.schema.SchemaClass"
  • Specify the serialization_method to PROTOBUF.
  • Set the schema_classpath to the path to the Python schema class. Test whether you have access the the schema with the code in a scratchpad.
    from path.to.schema import SchemaClass

Pass raw message to transformer

  serialization_method: RAW_VALUE