QBoard » Big Data » Big Data - Data Ingestion Tools : Sqoop, Flume, Kafka, Nifi.. » Purge Kafka Topic
User Dashboard

Purge Kafka Topic

Back To Topics
Tags : apache_kafka purge
  • 461
    Is there a way to purge the topic in kafka?

    I pushed a message that was too big into a kafka message topic on my local machine, now I'm getting an error:
     
    kafka.common.InvalidMessageSizeException: invalid message size

    Increasing the fetch.size is not ideal here, because I don't actually want to accept messages that big.

      December 24, 2020 12:04 PM IST
    0
  • 53
    Temporarily update the retention time on the topic to one second:
     
    kafka-topics.sh --zookeeper <zkhost>:2181 --alter --topic <topic name> --config retention.ms=1000

    And in newer Kafka releases, you can also do it with

    kafka-configs --entity-type topics
    kafka-configs.sh --zookeeper <zkhost>:2181 --entity-type topics --alter --entity-name <topic name> --add-config retention.ms=1000

    then wait for the purge to take effect (about one minute). Once purged, restore the previous retention.ms value.

      December 25, 2020 12:08 PM IST
    0