Migrating from Dynamodb Index store to boltdb-shipper

[Mahesh Bhat]

hi,

 

I am looking to migrate to Loki 2.0 and move onto to the single store index. Currently, dynamodb is being used as the Index store. I have added the schema_config for boltdb-shipper. But, when queries come in, the Ingesters simply crash with the following error :

-------------------

panic: runtime error: invalid memory address or nil pointer dereference

[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x14cf316]

goroutine 10588 [running]:

github.com/grafana/loki/pkg/logproto.(*GetChunkIDsResponse).MarshalTo(0x0, 0x3963bc0, 0x0, 0x0, 0x40a973, 0x1e6f3e0, 0x20c6780)

/src/loki/pkg/logproto/logproto.pb.go:4043 +0x26

github.com/grafana/loki/pkg/logproto.(*GetChunkIDsResponse).Marshal(0x0, 0x20c6780, 0x0, 0x7f076df7d078, 0x0, 0xbffc99cbd84db601)

/src/loki/pkg/logproto/logproto.pb.go:4031 +0x7a

google.golang.org/grpc/encoding/proto.codec.Marshal(0x20c6780, 0x0, 0x0, 0x273d9c0, 0x4094fb, 0xc00003a000, 0x1e88da0)

/src/loki/vendor/google.golang.org/grpc/encoding/proto/proto.go:70 +0x193

google.golang.org/grpc.encode(0x7f076e353658, 0x3963bc0, 0x20c6780, 0x0, 0x3963bc0, 0xb048b5, 0x273d9c0, 0x40c736, 0x20de2e0)

/src/loki/vendor/google.golang.org/grpc/rpc_util.go:545 +0x52

google.golang.org/grpc.(*Server).sendResponse(0xc0001936c0, 0x2761c40, 0xc001771380, 0xc0008e7200, 0x20c6780, 0x0, 0x0, 0x0, 0xc000b92660, 0x0, ...)

/src/loki/vendor/google.golang.org/grpc/server.go:869 +0x89

google.golang.org/grpc.(*Server).processUnaryRPC(0xc0001936c0, 0x2761c40, 0xc001771380, 0xc0008e7200, 0xc00080a450, 0x3915f08, 0x0, 0x0, 0x0)

/src/loki/vendor/google.golang.org/grpc/server.go:1117 +0x5fb

google.golang.org/grpc.(*Server).handleStream(0xc0001936c0, 0x2761c40, 0xc001771380, 0xc0008e7200, 0x0)

/src/loki/vendor/google.golang.org/grpc/server.go:1405 +0xccd

google.golang.org/grpc.(*Server).serveStreams.func1.1(0xc001e8b2c0, 0xc0001936c0, 0x2761c40, 0xc001771380, 0xc0008e7200)

/src/loki/vendor/google.golang.org/grpc/server.go:746 +0xa1

created by google.golang.org/grpc.(*Server).serveStreams.func1

/src/loki/vendor/google.golang.org/grpc/server.go:744 +0xa1

------------------

My schema_config and storage_config:

------------

schema_config:

  configs:

  - from: 2020-04-16

    store: aws-dynamo

    object_store: s3

    schema: v11

    index:

      prefix: loki_int_logs_index_

      period: 168h

      tags:

        Environment: loki-int

  - from: 2021-01-29

    store: boltdb-shipper

    object_store: s3

    schema: v11

    index:

      prefix: loki_int_logs_index_

      period: 24h

      tags:

        Environment: loki-int

storage_config:

    boltdb_shipper:

         active_index_directory: /var/loki/index

 cache_location: /var/loki/index_cache

 resync_interval: 5s

 shared_store: s3

---------------------

How do we migrate onto the single store Index ?

Thanks !

[raul...@zartis.com]

The following worked for me.


schema_config:
  configs:
    - from: 2020-01-01
      store: aws

      object_store: s3
      schema: v11
      index:

        prefix: xx-xx
        tags:
          x: "value"
          y: "value"
    - from: 2021-01-10
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h

storage_config:
  aws:
    s3: s3://xx-west-1/bucketname
    dynamodb:
      dynamodb_url: dynamodb://xx-west-1
  boltdb_shipper:
    active_index_directory: /tmp/loki/boltdb-shipper-active
    cache_location: /tmp/loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
    shared_store: filesystem
  filesystem:
    directory: /tmp/loki/chunks

compactor:
  working_directory: /tmp/loki/boltdb-shipper-compactor
  shared_store: filesystem


hope it helps

[Mahesh Bhat]

This bug is fixed in version 2.1.0 - https://github.com/grafana/loki/pull/2830