problem in concurrent writes

[Mona Hanna]

Hello

I am new to rqlite, but based on what I've read there should be 1 writer at a time.

So if more than one write request to the database, the other requests should be blocked, right?

I wrote a small command line that will select records from a table, then update them one by one. There is no problem one 1 instance of the process executes.

But when I run 2 separate instances of the same command line, the rqlited exits with the following error, and won't start unless I restore the database to a valid file (only one rqlited is running):

[http] 2021/03/23 18:27:47 service listening on [::]:4001
[rqlited] 2021/03/23 18:27:47 node is ready

panic: failed to unmarshal execute subcommand: flate: corrupt input before offset 4

goroutine 7 [running]:
github.com/rqlite/rqlite/store.(*Store).Apply(0xc00011fba0, 0xc00043bba8, 0x0, 0x0)
        /home/jinny/rqlite/src/github.com/rqlite/rqlite/store/store.go:1042 +0xc18
github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc000205560)
        /home/jinny/rqlite/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:90 +0x2c9
github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc00019ee00, 0x1, 0x40)
        /home/jinny/rqlite/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:113 +0x78
github.com/hashicorp/raft.(*Raft).runFSM(0xc000290000)
        /home/jinny/rqlite/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:219 +0x3a4
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc000290000, 0xc000204320)
        /home/jinny/rqlite/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:146 +0x55
created by github.com/hashicorp/raft.(*raftState).goFunc
        /home/jinny/rqlite/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:144 +0x66

I would appreciate any feedback

[Philip O'Toole]

Thank you for your report -- this looks important. I need a bit more information.

• what version are you running?
• can you reproduce this issue with a brand new (not upgrading from other versions) single node?
  
• can you supply the program you use to reproduce this? I need the exact steps.

Philip

--
You received this message because you are subscribed to the Google Groups "rqlite" group.
To unsubscribe from this group and stop receiving emails from it, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/63843f37-8779-45bb-bcbd-21e81a3a0bc3n%40googlegroups.com.

[Mona Hanna]

./rqlited --version
rqlited v5.8.0 linux amd64 go1.14 (commit b41e5c19c5875f559bb523397f2e00d567df21a4, branch master)

to start the server:

 ./rqlited -raft-log-level "DEBUG"  -http-addr 0.0.0.0:4001 -node-id 1 ~/node.1 &

The database contains a testing table with some data (attached testing.sql)

I am also attaching 2 scripts to update the table

test_update_ok.sh with multiple curl command like:

curl -XPOST 'localhost:4001/db/execute?pretty&timings' -H "Content-Type: application/json" -d "[
[\"UPDATE testing SET msisdn = '962790006391' , messageid = 0 , date = '2021-03-24 12:28:02' , retry_scheme_id = 0  WHERE msisdn = '962790006391'\"]
]"

test_update_fail.sh with the same update but an extra field to update:

curl -XPOST 'localhost:4001/db/execute?pretty&timings' -H "Content-Type: application/json" -d "[
[\"UPDATE testing SET msisdn = '962790006391' , messageid = 0 , date = '2021-03-24 12:28:02' , retry_scheme_id = 0 , retry_nbr = 0  WHERE msisdn = '962790006391'\"]
]"

No problem when running simultanously test_update_ok.sh, from 2 terminals, but with test_update_fail.sh, the server exits with this error and doesn't start again:

[rqlited] 2021/03/24 12:48:48 http: panic serving [::1]:40636: runtime error: index out of range [43] with length 30
goroutine 843 [running]:
net/http.(*conn).serve.func1(0xc0006a0a00)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:1772 +0x139
panic(0xc21c00, 0xc000022520)
        /home/philip/.gvm/gos/go1.14/src/runtime/panic.go:973 +0x396
compress/flate.(*huffmanBitWriter).indexTokens(0xc0002861e0, 0xc0002ee000, 0x2, 0x4001, 0xc0003c57e0, 0x669b9e)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/huffman_bit_writer.go:551 +0x2df
compress/flate.(*huffmanBitWriter).writeBlock(0xc0002861e0, 0xc0002ee000, 0x2, 0x4001, 0x0, 0xc0002bc000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/huffman_bit_writer.go:440 +0xa1
compress/flate.(*compressor).writeBlock(0xc0003e0000, 0xc0002ee000, 0x1, 0x4001, 0xa9, 0xa9, 0x0)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/deflate.go:170 +0xc5
compress/flate.(*compressor).deflate(0xc0003e0000)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/deflate.go:415 +0x8be
compress/flate.(*compressor).close(0xc0003e0000, 0xc0004e42c0, 0xa9)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.14/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc00012a210, 0xc0004e42c0, 0xa9)
        /home/philip/.gvm/gos/go1.14/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000112aa0, 0xdbf7e0, 0xc00016f3c0, 0xc0003c5ab8, 0xa5037f, 0x203000, 0x203000, 0x203000, 0xc001208218)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x227
github.com/rqlite/rqlite/store.(*Store).execute(0xc0001246c0, 0xc00016f3c0, 0x41b568, 0x50, 0xc1c800, 0xc0007c3701, 0xc000f68500)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/store/store.go:498 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc0001246c0, 0xc00016f3c0, 0x200, 0xc001208218, 0x1, 0x1, 0x0)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/store/store.go:472 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc00016a000, 0xdbd6a0, 0xc0004e07e0, 0xc00086c800)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/http/service.go:637 +0x42c
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc00016a000, 0xdbd6a0, 0xc0004e07e0, 0xc00086c800)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/http/service.go:245 +0x5a4
net/http.serverHandler.ServeHTTP(0xc00016a0e0, 0xdbd6a0, 0xc0004e07e0, 0xc00086c800)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0006a0a00, 0xdbf4a0, 0xc00016f240)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:2933 +0x35c
[rqlited] 2021/03/24 12:48:48 http: panic serving [::1]:40634: interface conversion: interface {} is nil, not *store.fsmExecuteResponse
goroutine 842 [running]:
net/http.(*conn).serve.func1(0xc0006a0960)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:1772 +0x139
panic(0xbb59e0, 0xc000166cc0)
        /home/philip/.gvm/gos/go1.14/src/runtime/panic.go:973 +0x396
github.com/rqlite/rqlite/store.(*Store).execute(0xc0001246c0, 0xc000238f40, 0x41b568, 0x50, 0xc1c800, 0xc000113b01, 0xc000607860)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/store/store.go:527 +0x498
github.com/rqlite/rqlite/store.(*Store).Execute(0xc0001246c0, 0xc000238f40, 0x200, 0xc0010ba1c0, 0x1, 0x1, 0x0)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/store/store.go:472 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc00016a000, 0xdbd6a0, 0xc00016a7e0, 0xc0002a0900)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/http/service.go:637 +0x42c
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc00016a000, 0xdbd6a0, 0xc00016a7e0, 0xc0002a0900)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/http/service.go:245 +0x5a4
net/http.serverHandler.ServeHTTP(0xc00016a0e0, 0xdbd6a0, 0xc00016a7e0, 0xc0002a0900)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:2807 +0xa3
net/http.(*conn).serve(0xc0006a0960, 0xdbf4a0, 0xc000238dc0)
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:1895 +0x86c
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.14/src/net/http/server.go:2933 +0x35c
panic: failed to unmarshal execute subcommand: unexpected EOF

goroutine 10 [running]:
github.com/rqlite/rqlite/store.(*Store).Apply(0xc0001246c0, 0xc00037f568, 0x1486020, 0xc00ee42438d5ff2f)
        /tmp/tmp.iLtO0nvm0M/src/github.com/rqlite/rqlite/store/store.go:873 +0x838
github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc00052e630)
        /tmp/tmp.iLtO0nvm0M/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:90 +0x2c1
github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc000644800, 0x1, 0x40)
        /tmp/tmp.iLtO0nvm0M/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:113 +0x75
github.com/hashicorp/raft.(*Raft).runFSM(0xc000298000)
        /tmp/tmp.iLtO0nvm0M/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:219 +0x42f
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc000298000, 0xc0002043a0)
        /tmp/tmp.iLtO0nvm0M/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:146 +0x55
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.iLtO0nvm0M/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:144 +0x66

[Philip O'Toole]

Can you try reproducing the issue with the latest version, 5.10.2? It's worth upgrading anyway. There have been a fair number of changes to the code since 5.8.0, and I'd rather debug this issue using the latest code base.

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/5bd1b0d8-ed84-4cd1-b753-1984dad7443fn%40googlegroups.com.

[Mona Hanna]

 ./rqlited  --version
rqlited v5.10.2 linux amd64 go1.15 (commit 125ae547879fc5a5b2cbb672b8f5011c171e5907, branch master)

I changed the timings to transaction in the requests, but still the same result

[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59358: index > windowEnd
goroutine 782 [running]:
net/http.(*conn).serve.func1(0xc0003d0aa0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xbbca80, 0xd90a60)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:397 +0xa25
compress/flate.(*compressor).close(0xc000448000, 0xc000366160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc000366160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0000b0cc0, 0x66, 0x40, 0x203000, 0x203000, 0x203000, 0x7fa860e96830)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0000b0cc0, 0x41c878, 0x50, 0xc57ee0, 0xc000600d01, 0xc0003873b0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0000b0cc0, 0x200, 0xc000252220, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc00035a380, 0xc000346700)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc00035a380, 0xc000346700)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc00035a380, 0xc000346700)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0003d0aa0, 0xdb3860, 0xc0000b0b40)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59360: index > windowEnd
goroutine 783 [running]:
net/http.(*conn).serve.func1(0xc0003d0b40)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xbbca80, 0xd90a60)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:397 +0xa25
compress/flate.(*compressor).write(0xc000448000, 0xc00064a4d0, 0xa9, 0xa9, 0xa9, 0xabe5295b, 0x0)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:554 +0x85
compress/flate.(*Writer).Write(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:712
compress/gzip.(*Writer).Write(0xc000152580, 0xc00064a4d0, 0xa9, 0xa9, 0xc0005ada40, 0x0, 0xc00064a4d0)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:196 +0xc5
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0005ada40, 0xc000647ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000354350)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:102 +0x20a
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0005ada40, 0x41c878, 0x50, 0xc57ee0, 0xc000359a01, 0xc0000bd400)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0005ada40, 0x200, 0xc000354350, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc00063cb60, 0xc000236f00)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc00063cb60, 0xc000236f00)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc00063cb60, 0xc000236f00)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0003d0b40, 0xdb3860, 0xc0005ad8c0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59362: runtime error: index out of range [43] with length 30
goroutine 733 [running]:
net/http.(*conn).serve.func1(0xc0005bc0a0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc000022120)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).writeTokens(0xc00012e3c0, 0xc000500000, 0x7f, 0x4001, 0xc00024a480, 0x11e, 0x11e, 0xc0004ea080, 0x1e, 0x1e)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:598 +0x2f3
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x7f, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:495 +0x39c
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x7e, 0x4001, 0xa9, 0x18, 0x74)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc000366000, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc000366000, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0000b0380, 0xc0005c5ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000252010)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0000b0380, 0x41c878, 0x50, 0xc57ee0, 0xc000600001, 0xc00041c140)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0000b0380, 0x200, 0xc000252010, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc00035a000, 0xc000346000)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc00035a000, 0xc000346000)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc00035a000, 0xc000346000)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0005bc0a0, 0xdb3860, 0xc0000b0000)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59364: runtime error: index out of range [43] with length 30
goroutine 764 [running]:
net/http.(*conn).serve.func1(0xc0002ce6e0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc0002da000)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).indexTokens(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0xc0006457e8, 0x674cde)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:551 +0x2e5
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:440 +0xa8
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x1, 0x4001, 0xa9, 0xa9, 0x0)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc00002a0b0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc00002a0b0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc00020f440, 0xc000645ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000354008)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc00020f440, 0x41c878, 0x50, 0xc57ee0, 0xc000128001, 0xc0000bcb40)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc00020f440, 0x200, 0xc000354008, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0005cc1c0, 0xc000198000)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0005cc1c0, 0xc000198000)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0005cc1c0, 0xc000198000)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0002ce6e0, 0xdb3860, 0xc00020e080)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59380: runtime error: index out of range [43] with length 30
goroutine 745 [running]:
net/http.(*conn).serve.func1(0xc0003495e0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc0002da120)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).indexTokens(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0xc0006437e8, 0x674cde)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:551 +0x2e5
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:440 +0xa8
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x1, 0x4001, 0xa9, 0xa9, 0x0)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc00002a210, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc00002a210, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc00020fa40, 0xc000643ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc0003540b0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc00020fa40, 0x41c878, 0x50, 0xc57ee0, 0xc000128d01, 0xc0000bd7c0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc00020fa40, 0x200, 0xc0003540b0, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0005cc620, 0xc000198300)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0005cc620, 0xc000198300)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0005cc620, 0xc000198300)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0003495e0, 0xdb3860, 0xc00020f8c0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59378: runtime error: index out of range [43] with length 30
goroutine 744 [running]:
net/http.(*conn).serve.func1(0xc000349400)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc0006a0120)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).writeTokens(0xc00012e3c0, 0xc000500000, 0x80, 0x4001, 0xc000150000, 0x11e, 0x11e, 0xc000132080, 0x1e, 0x1e)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:598 +0x2f3
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x80, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:495 +0x39c
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x7f, 0x4001, 0xa9, 0x18, 0x74)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc00064a160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc00064a160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0005aca80, 0xc000649ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000138220)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0005aca80, 0x41c878, 0x50, 0xc57ee0, 0xc000358901, 0xc000280fa0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0005aca80, 0x200, 0xc000138220, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc00063c380, 0xc000236500)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc00063c380, 0xc000236500)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc00063c380, 0xc000236500)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000349400, 0xdb3860, 0xc0005ac900)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59410: runtime error: index out of range [43] with length 30
goroutine 818 [running]:
net/http.(*conn).serve.func1(0xc000422be0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc0006a01c0)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).writeTokens(0xc00012e3c0, 0xc000500000, 0x7c, 0x4001, 0xc00024a480, 0x11e, 0x11e, 0xc0004ea080, 0x1e, 0x1e)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:598 +0x2f3
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x7c, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:495 +0x39c
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x7b, 0x4001, 0xa9, 0x18, 0x74)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc0002f2160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc0002f2160, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0005acd00, 0xc0005c5ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000138538)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0005acd00, 0x41c878, 0x50, 0xc57ee0, 0xc000358f01, 0xc000281680)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0005acd00, 0x200, 0xc000138538, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0002ec540, 0xc0002f6700)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0002ec540, 0xc0002f6700)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0002ec540, 0xc0002f6700)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc000422be0, 0xdb3860, 0xc0005acb80)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59412: runtime error: index out of range [43] with length 30
goroutine 834 [running]:
net/http.(*conn).serve.func1(0xc0003d0c80)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc00020a160)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).indexTokens(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0xc0006437e8, 0x674cde)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:551 +0x2e5
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x2, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:440 +0xa8
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x1, 0x4001, 0xa9, 0xa9, 0x0)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc0003602c0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc0003602c0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0005dacc0, 0xc000643ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc000252428)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0005dacc0, 0x41c878, 0x50, 0xc57ee0, 0xc000601301, 0xc00041d630)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0005dacc0, 0x200, 0xc000252428, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0002e0540, 0xc0002b2800)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0002e0540, 0xc0002b2800)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0002e0540, 0xc0002b2800)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0003d0c80, 0xdb3860, 0xc0005dab40)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59504: runtime error: index out of range [43] with length 30
goroutine 598 [running]:
net/http.(*conn).serve.func1(0xc0002cf5e0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xc5d200, 0xc00020a120)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
compress/flate.(*huffmanBitWriter).indexTokens(0xc00012e3c0, 0xc000500000, 0x3, 0x4001, 0xc0006437e8, 0x674cde)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:551 +0x2e5
compress/flate.(*huffmanBitWriter).writeBlock(0xc00012e3c0, 0xc000500000, 0x3, 0x4001, 0x0, 0xc0004ee000, 0xa9, 0x10000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/huffman_bit_writer.go:440 +0xa8
compress/flate.(*compressor).writeBlock(0xc000448000, 0xc000500000, 0x2, 0x4001, 0xa9, 0xa8, 0x0)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:170 +0xd1
compress/flate.(*compressor).deflate(0xc000448000)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:415 +0x906
compress/flate.(*compressor).close(0xc000448000, 0xc00024e0b0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:646 +0x50
compress/flate.(*Writer).Close(...)
        /home/philip/.gvm/gos/go1.15/src/compress/flate/deflate.go:732
compress/gzip.(*Writer).Close(0xc000152580, 0xc00024e0b0, 0xa9)
        /home/philip/.gvm/gos/go1.15/src/compress/gzip/gzip.go:242 +0x6a
github.com/rqlite/rqlite/command.(*RequestMarshaler).Marshal(0xc000128a60, 0xdb3b60, 0xc0002e8600, 0xc000643ac0, 0xa9475d, 0x203000, 0x203000, 0x203000, 0xc0000101e8)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/command/marshal.go:105 +0x232
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0002e8600, 0x41c878, 0x50, 0xc57ee0, 0xc000128c01, 0xc0001432c0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:562 +0x5d
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0002e8600, 0x200, 0xc0000101e8, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0002201c0, 0xc000198100)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0002201c0, 0xc000198100)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0002201c0, 0xc000198100)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0002cf5e0, 0xdb3860, 0xc0002e8480)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
[rqlited] 2021/03/24 17:31:27 http: panic serving [::1]:59502: interface conversion: interface {} is nil, not *store.fsmExecuteResponse
goroutine 926 [running]:
net/http.(*conn).serve.func1(0xc0003d1180)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xbf3f80, 0xc00011b140)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0000b0a80, 0x41c878, 0x50, 0xc57ee0, 0xc000358701, 0xc00041d3b0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:591 +0x4a9
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0000b0a80, 0x200, 0xc0006fe300, 0x1, 0x1, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:536 +0x78
github.com/rqlite/rqlite/http.(*Service).handleExecute(0xc0005cc000, 0xdb1ba0, 0xc0002a42a0, 0xc000570300)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:642 +0x437
github.com/rqlite/rqlite/http.(*Service).ServeHTTP(0xc0005cc000, 0xdb1ba0, 0xc0002a42a0, 0xc000570300)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/http/service.go:250 +0x5df
net/http.serverHandler.ServeHTTP(0xc0005cc0e0, 0xdb1ba0, 0xc0002a42a0, 0xc000570300)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2843 +0xa3
net/http.(*conn).serve(0xc0003d1180, 0xdb3860, 0xc0000b08c0)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1925 +0x8ad
created by net/http.(*Server).Serve
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:2969 +0x36c
panic: failed to unmarshal execute subcommand: flate: corrupt input before offset 5

goroutine 30 [running]:
github.com/rqlite/rqlite/store.(*Store).Apply(0xc00011f520, 0xc0003d12e8, 0x0, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:1042 +0xc0b
github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc00058a320)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:90 +0x2c2
github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc0002bc200, 0x1, 0x40)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:113 +0x75
github.com/hashicorp/raft.(*Raft).runFSM(0xc0001f6000)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:219 +0x3c4
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc0001f6000, 0xc000119240)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:146 +0x55
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:144 +0x66

[Philip O'Toole]

Thanks for the report. Just so I am clear, you just upgraded to 5.10.2, using the existing database that was underneath the system right?

Does the problem happen if you start with a *brand new 5.10.2 node*, load that with the data, and then run your test? 

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/3488737a-6e09-42ea-bc2c-16883af69a5en%40googlegroups.com.

[Philip O'Toole]

I just loaded your attached database into a brand new rqlite 5.10.2  node, and then ran both scripts in parallel. I didn't experience into any issues, and ran the test twice.

Some of the older versions of rqlite used a different method for compressing and storing the data. The later versions (such as 5.10.2) attempt to handle those different formats automatically. Perhaps there is a bug in that conversion code, which you don't hit if you start with a brand new 5.10.2 node. That's what I'm wondering.

Did you ever run versions of rqlite *older* than 5.8.0, and 5.8.0 was also an upgrade from an earlier version?

I am definitely interested in learning if you can reproduce the issue with a brand new 5.10.2 node.

Philip

[Mona Hanna]

it is strange that you couldn't reproduce the problem.

The machine has CentOS Linux release 7.6

Do you think that anything in the envirornment/libraries might affect the server?

I am not upgrading from an earlier version, in fact I just started working with rqlite

I tried using a new node:

[rqlited] 2021/03/25 16:36:08 rqlited starting, version v5.10.2, commit 125ae547879fc5a5b2cbb672b8f5011c171e5907, branch master
[rqlited] 2021/03/25 16:36:08 go1.15, target architecture is amd64, operating system target is linux
[rqlited] 2021/03/25 16:36:08 launch command: ./rqlited -raft-log-level DEBUG -http-addr 0.0.0.0:4001 -node-id 1 /home/jinny/node.test.1
[rqlited] 2021/03/25 16:36:08 no preexisting node state detected in /home/jinny/node.test.1, node may be bootstrapping

it failed with:

 panic: failed to unmarshal execute subcommand: flate: corrupt input before offset 7

goroutine 38 [running]:
github.com/rqlite/rqlite/store.(*Store).Apply(0xc0001301a0, 0xc000219388, 0x0, 0x0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:1042 +0xc0b
github.com/hashicorp/raft.(*Raft).runFSM.func1(0xc0001146f0)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:90 +0x2c2
github.com/hashicorp/raft.(*Raft).runFSM.func2(0xc00029e400, 0x1, 0x40)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:113 +0x75
github.com/hashicorp/raft.(*Raft).runFSM(0xc0002a0000)
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/fsm.go:219 +0x3c4
github.com/hashicorp/raft.(*raftState).goFunc.func1(0xc0002a0000, 0xc000114900)

        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:146 +0x55
created by github.com/hashicorp/raft.(*raftState).goFunc
        /tmp/tmp.rOi1r2qzkt/pkg/mod/github.com/hashicorp/ra...@v1.2.0/state.go:144 +0x66

sometimes I get a different error in addition to the unmarshal error:

interface conversion: interface {} is nil, not *store.fsmExecuteResponse

goroutine 931 [running]:
net/http.(*conn).serve.func1(0xc000215f40)
        /home/philip/.gvm/gos/go1.15/src/net/http/server.go:1801 +0x147
panic(0xbf3f80, 0xc0003e7020)
        /home/philip/.gvm/gos/go1.15/src/runtime/panic.go:975 +0x3e9
github.com/rqlite/rqlite/store.(*Store).execute(0xc00011f520, 0xc0000b0940, 0x41c878, 0x50, 0xc57ee0, 0xc000128e01, 0xc0000bd0e0)
        /tmp/tmp.rOi1r2qzkt/src/github.com/rqlite/rqlite/store/store.go:591 +0x4a9
github.com/rqlite/rqlite/store.(*Store).Execute(0xc00011f520, 0xc0000b0940, 0x200, 0xc0004ca188, 0x1, 0x1, 0x0)

...

[Philip O'Toole]

This is very strange -- if you're seeing issues like that just starting a brand new node, with no existing state, then it could be something specific to your environment. Maybe some incompatibility between the binary I built on Ubuntu 18 (assuming you downloaded the release from GitHub) and what's on your machine. Something libc related, perhaps.

Perhaps you should try building rqlite yourself, and see if that helps? It is not difficult. All you really need is git, go, and gcc. 

https://github.com/rqlite/rqlite/blob/master/CONTRIBUTING.md#building-rqlite

Alternatively, perhaps try the docker image? docker pull rqlite/rqlite

To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/a398c32f-ffbd-41eb-ae20-1fde27c8ff5en%40googlegroups.com.

[Mona Hanna]

it worked when we started the server with :  -compression-size 1000

Can you please explain how the compression works?

is there any documentation on the available options that can be used when starting the server?

[Philip O'Toole]

-compression-size: the length at which a single SQL command will trigger compression of that command

-compression-batch: the number of SQL commands in a bulk request, which will trigger compression of that entire request, regardless of the size of any SQL command in that request.

Compression is gzip, nothing special. Gzip the data before it's written to the Raft log. Decompress when it's read back.

By setting it that high, you're most likely avoiding running the compression code at all, which is what is crashing in the traces you sent. If you effectively disable compression that's OK, but you'll use more disk space. However the system always compresses the SQLite database when snapshotting the log. You can't totally avoid running the compression code.

Run "rqlited -h" for help on the server start-up options. If you need more information, let me know. I suggest you build your own version, and see if you can repro the issue. I cannot repro what you are seeing, so I am wondering if it's something about your environment that could be solved by building in your environment.

You received this message because you are subscribed to a topic in the Google Groups "rqlite" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/rqlite/N_gxV_sTZr4/unsubscribe.
To unsubscribe from this group and all its topics, send an email to rqlite+un...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/rqlite/60581914-cbef-42a1-bf88-6b22c05c73a7n%40googlegroups.com.

[Philip O'Toole]

There are counts of the compression activity available via the expvar endpoint.

https://github.com/rqlite/rqlite/blob/master/DOC/DIAGNOSTICS.md#expvar-support

[Philip O'Toole]

I believe I found the root cause of this issue, and fixed it in 5.11.1.

https://github.com/rqlite/rqlite/issues/781'

https://github.com/rqlite/rqlite/releases/tag/v5.11.1