>= 0.9.61
offset
config allows you to reset offset when starting the streaming pipeline. If using the offset
config, the partitions
config is required.
offset
config has 4 optional values:
beginning
and end
set the consumer to consume data from the beginning and end of the queue, respectively.
Also, they do not require offset_value
int
config set the consumer to consume data from the given offset value inside the queue.
This value correspond to the numeric position inside each partition.
timestamp
config set the consumer to consume data from the given offset timestamp value inside the queue.
This value correspond to the timestamp of the message (Unit should be milliseconds since beginning of the epoch)
include_metadata
is false, the kafka data loader returns data from value field, e.g.
include_metadata
is set to true, the kafka data loader returns these elements as messages with data = {key: value}
and metadata
with (topic, partition, offset and time), e.g.
serialization_method
to PROTOBUF
.schema_classpath
to the path to the Python schema class. Test whether you have access the the schema with the code
in a scratchpad.