how to restore the old logs sent by promtail

[Naveen Badam]

Hi All,

How can we preserve the actual logs sent by promtail to loki?

Issue: 

we have mutliple promtail services sending logs to loki. We have recreated the ec2s and re-deployed the promtail on the new nodes recently. Now, we can only see the latest logs, but not the previous logs which were showing up in the loki. 

below is the loki config, am storing the index and chunks on EBS:

auth_enabled: false

server:
  http_listen_port: 3100
  grpc_listen_port: 9096



common:
  instance_addr: 127.0.0.1
  path_prefix: /tmp/loki
  storage:
    filesystem:
      chunks_directory: /tmp/loki/chunks
      rules_directory: /tmp/loki/rules
  replication_factor: 1
  ring:
    instance_addr: 127.0.0.1
    kvstore:
      store: inmemory

query_range:
  parallelise_shardable_queries: true
  results_cache:
    cache:
      embedded_cache:
        enabled: true
        max_size_mb: 100

query_scheduler:
  max_outstanding_requests_per_tenant: 4096

frontend:
  max_outstanding_per_tenant: 4096


schema_config:
  configs:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  boltdb:
    directory: /loki/index

compactor:
  working_directory: /loki/compactor
  shared_store: filesystem
  retention_enabled: true

chunk_store_config:
  max_look_back_period: 672h

limits_config:
  retention_period: 30d
  split_queries_by_interval: 15m
  max_query_parallelism: 32
  max_query_series: 100000

table_manager:
  retention_deletes_enabled: true
  retention_period: 30d

ingester_client:
  remote_timeout: 20s

we have EBS mount at /data and we have the volume mapping on the container as below

/opt/n6cloudagent/shared/loki/:/etc/loki/
/data/loki/index:/data/loki/index
/data/loki/chunks:/data/loki/chunks
/data/loki:/tmp/loki

How can we store the logs shipped by promtail services into local EBS, so that we can still have access the old logs on promtail service restarts.

Looking forward to hearing from you.

Regards,
Naveen.