Bad Gateway on particular Loki search failures only

[Kendall Chenoweth]

Hello,

I have a Loki instance running in Kubernetes.  I'm running using the Loki 2.1.0 image.  My storage is on NFS and I'm using a single Loki container instance.  (I don't believe that NFS supports more than 1 container instance.)  

When I run the following query everything works

{job="akamai-myapp"} |="/supportfiles/"  !="access_check.txt" 

When I enhance the query, it fails

{job="akamai-myapp"} |="/supportfiles/"  !="access_check.txt" !="package_registry.xml"

However, if I change the third filter to something like !="Java" it doesn't fail.

This makes me think it's related to data size and transport settings.  I'm using boltdb storage and have 64GB in chunks and 41M in index.  I have a 14 day retention.

Do these sizes seem unreasonable?

I believe it's a problem with data size or persistent storage configuration.  Here is my pv.

apiVersion: v1

kind: PersistentVolume

metadata:

  name: loki-storage

  namespace: loki-stack

spec:

  capacity:

    storage: 200Gi

  volumeMode: Filesystem

  accessModes:

    - ReadWriteMany

  persistentVolumeReclaimPolicy: Retain

  storageClassName: nfs-www-observability

  mountOptions:

    - hard

  nfs:

    path: <none_of_your_business>/loki-storage

    server: some-random-server.com

I've experimented with several Loki and Grafana server time out options without success.

The Loki server is logging these errors

level=warn ts=2021-01-08T02:24:22.301859554Z caller=logging.go:60 traceID=4776d833a710019f msg="GET /loki/api/v1/query_range?direction=BACKWARD&limit=4000&query=%7Bjob%3D%22akamai-myapp%22%7D%20%7C%3D%22%2Fsupportfiles%2F%22%20%7C%3D%22R2020b%22&start=1609899825000000000&end=1610072626000000000&step=16 36.849862796s, error: write tcp 192.168.249.39:3100->192.168.242.0:34870: i/o timeout ws: false; Accept: application/json, text/plain, */*; Accept-Encoding: gzip, deflate; Accept-Language: en-US,en;q=0.5; Dnt: 1; User-Agent: Grafana/7.0.5; X-Forwarded-For: 172.31.75.17, 172.31.75.17; X-Grafana-Org-Id: 1; "

 I | http: superfluous response.WriteHeader call from github.com/opentracing-contrib/go-stdlib/nethttp.(*statusCodeTracker).WriteHeader (status-code-tracker.go:17)

Has anybody seen this before or have any ideas on how to either debug or resolve this issue?

Thanks in advance!

-Kendall Chenoweth