Loki High Availability install on OCI issues

[Sandeep Bangera]

Greetings,

I am trying to install Loki in High Availability mode on OCI using Helm. I was able to get it to work out of the box with some minor changes.  But now I was trying to customize the config to use https://grafana.com/docs/loki/latest/configuration/examples/#almost-zero-dependencies-setup.

OCI does have AWS compatible API:
https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm

I have not been able to get the chunks/indexes uploaded to OCI Object Storage :(. Also the gossip member list does not seem to work,  See the below error messages in the loki logs:


caller=memberlist_logger.go:74 level=warn msg="Failed to resolve loki-headless.prod-sbangera.svc.cluster.local:7946: lookup loki-headless.prod-sbangera.svc.cluster.local on 10.96.5.5:53: no such host"

caller=flush.go:199 org_id=fake msg="failed to flush user" err="InvalidParameter: 1 validation error(s) found.\n- minimum field size of 1, PutObjectInput.Bucket.\n"

Running this within one of the util pods:

host loki-headless.prod-sbangera.svc.cluster.local
loki-headless.prod-sbangera.svc.cluster.local has address 10.244.3.162
loki-headless.prod-sbangera.svc.cluster.local has address 10.244.3.43
loki-headless.prod-sbangera.svc.cluster.local has address 10.244.4.33

nc -z -v loki-headless.prod-sbangera.svc.cluster.local 7946
Connection to loki-headless.prod-sbangera.svc.cluster.local 7946 port [tcp/*] succeeded!

I have attached my loki values.yaml. Would appreciate any help here.

Cheers,
Sandeep

 

[Sandeep Bangera]

I was able to get the Oracle OCI S3 ObjectStorage integration with Loki working using the S3 compatible API https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm. Below is the relevant loki config snippet. I had to use the full "https://" URL instead of "s3://" in the s3 parameter, Also had to quote the secret_access_key parameter since it had some special characters. I believe that was causing the invalid parameter

   aws:
        s3: https://<mynamespace>.compat.objectstorage.<my-region-1>.oraclecloud.com/loki-poc-bucket
        access_key_id: my_key_id
        secret_access_key: "my_access_key"
        s3forcepathstyle: true
        region: my-region-1
         insecure: false