Re: [jackson-user] SMILE and vocabulary
Tatu Saloranta
> I've got some json that has many recurring instances of a relatively
> small number of long strings. Unfortunately, it's in the form of many
> distinct documents.
>
> What I wish is that, along the lines of the XML FIS, there was a way
> to 'preload' Smile with these guys so that it would write them out
> using smaller strings.
This might be best handled at a higher level, using some kind of
replacements -- Smile format is designed not to use dictionaries.
Although it is not out of the question that perhaps a new revision of
format could add support: there are unused bit/byte combinations,
reserved for future use, which could be used.
Btw, I will cc my response to Smile format mailing list, even more
appropriate there. :)
-+ Tatu +-
Tatu Saloranta
> Could you build up the dictionary on the fly as you see new keys?
Indeed this is how Smile back references work, both for keys (of any
length) and short String values (64 bytes or less) -- at most 1024
most recently seen entries are in (sort of) sliding window dictionary,
and can be encoded using just a reference id. This is partly the
reason for not supporting explicit/external dictionaries.
Benson's use case does not benefit from this, however, since he
mentioned long Strings.
-+ Tatu +-