Need help with Loki configuration logs retention

[Lhullier David]

Hello,

i'm a french developper, that I'm trying to use the open-source stack of grafana. I have grafana, mimir, tempo and loki.
Grafana and Tempo are deployed on one server. Mimir on a other server. Loki on a third server.
Each stack had their own configuration, and all run almost perfectly.
But with Loki, I had issues. I have 20Go on the server to store logs, and I tried to configure to have 1h of retention logs.
But in 8 days, it's runing out stockage.

I have an idea of the process. If I have 1h of logs retention, and I get the logs of all my backends in 1h, either I have enough storage during this hour to collect and delete logs (and that's good).
Either, I do not have enough storage to collect and delete the logs, and in one hour, my 20 Go are full ?

Tell me if I'm wrong ?

But in my case, i have the figure : (in x-axis this days of the month)

I start my loki with docker compose :
services:

    loki:

        image: grafana/loki:latest #3.1.2 # https://hub.docker.com/r/grafana/loki/tags and https://github.com/grafana/loki/releases

        network_mode: bridge

        command: ['-config.file=/etc/loki/local-config.yaml']

        volumes:

          - ./loki.yml:/etc/loki.yml:ro

        ports:

            - "3100:3100"


This is my configuration :
schema_config:

  configs:

    - from: 2020-10-24

      store: boltdb-shipper

      object_store: filesystem

      schema: v11

      index:

        prefix: index_

        period: 1h

storage_config:

  boltdb_shipper:

    active_index_directory: /loki/index

    cache_location: /loki/index_cache

    shared_store: filesystem

  filesystem:

    directory: /loki/chunks

 # Configuration cruciale pour la rétention avec boltdb-shipper et filesystem

  table_manager:

    retention_deletes_enabled: true

    retention_period: 1h # Exemple pour 7 jours

    # working_directory: /loki/table-manager # Chemin pour les opérations du table_manager

      #limits_config:

      #retention_periods:

      #chunk: 1h

      #index: 1h

max_outstanding_per_tenant: 4096

compactor:

  retention_enabled: true

  working_directory: /loki/compactor # Ou un autre chemin pour les opérations du compacteur

  shared_store: filesystem

  compaction_interval: 10m # Exemple

  retention_enabled: true

  retention_delete_delay: 1h # Délai avant la suppression effective

  retention_delete_worker_count: 150 # Nombre de workers

  apply_retention_period: 1h # Ex: 7d, 168h

Tell me if my configuration is wrong and how to improve it ?

Thank you for your help,
David Lhullier