Handling prefixes in gloss codecs

[Andrew Cholakian]

I've got a dead simple codec:

(defcodec wire-protocol (finite-block (prefix :int32)))

That's giving me issues under load. Engulf uses that codec plus lamina TCP to communicate internally (code here: https://github.com/andrewvc/engulf/blob/master/src/engulf/comm/netchan.clj)

It works fine 95% of the time. However, when I really push engulf (say hammering localhost w/ 100 simultaneous clients), it seems that reading off the channel gives me the right data, but missing the prefix. Normally I get back the frames, take all the data from byte 5 to the end, and process it. Under heavy load however sometimes the 4 prefix bytes are missing.

I hacked around it with the try catch block here: https://github.com/andrewvc/engulf/blob/master/src/engulf/comm/netchan.clj#L27 , but it's pretty terrible. Any idea why the prefix is missing sometimes?

Am I misusing something in lamina/gloss/aleph? Is there a better way to decode that doesn't require me to remove the prefix myself? For the record, I need bytes because I gzip compress all the data.

[Andrew Cholakian]

Looking a bit further this seems related to message size. Small messages come with a prefix, large ones don't. If you look at the test suite here:

https://github.com/andrewvc/engulf/blob/master/test/engulf/test/comm/netchan_test.clj

The second half (the large message tests) will show the error in the console (since they are caught by the hack)