Apache Flume - incorrect timestamp

  • When I use Apache Flume, I get a millisecond timestamp rahter then a second timestamp. This is my flume conf file:

    # Name the components on this agent
    a1.sources = r1
    a1.sinks = k1
    a1.channels = c1

    # Describe/configure the source
    a1.sources.r1.type = org.apache.flume.source.http.HTTPSource
    a1.sources.r1.port = 44444

    # Describe the sink
    a1.sinks.k1.type = hdfs
    a1.sinks.k1.hdfs.path = flume/ads/%y-%m-%d/%H
    a1.sinks.k1.hdfs.fileType = DataStream

    # Use a channel which buffers events in memory
    a1.channels.c1.type = memory
    a1.channels.c1.capacity = 10000

    # Bind the source and sink to the channel
    a1.sources.r1.channels = c1
    a1.sinks.k1.channel = c1

    Flume creates folder flume/ads/70-01-17/02. The folder contains files "FlumeData.timestamp" and this timestamp has twelve digits.

    I get an incorrect folder's name.

    What can I do?
    hadoop flume
      June 12, 2019 11:47 AM IST
    0
  • I found it.

    In section Sink should be:

    a1.sinks.k1.hdfs.useLocalTimeStamp = True
      June 12, 2019 11:49 AM IST
    0