Encode unstruct_event as LZO in Redshift

[Eric Pantera]

Hello there, 

the generalization of unstruct_event and shredding is really great !

We have more and move shredded tables here.

And more and more atomic.events rows with a significant unstruct_field json content (More than 10 Millions unstruct_event per day)

What do you thing about changing the encoding of the atomic.events.unstruct_event column ?

- Today encoding is RAW

- It could be a good idea to move to encoding LZO

Do you see any warning with that ?

Thanks to the shredding, atomic.events.unstruct_event is more and more useless, so the performance overhead won't be an issue.

thanks !

[Alex Dean]

Hey Eric!

This sounds like a promising idea. For anyone interested, here is the relevant Redshift documentation: http://docs.aws.amazon.com/redshift/latest/dg/lzo-encoding.html

We are working on a "spring refresh" of Snowplow Core including the atomic.events table in this milestone, so this would be a good time to add this in.

Does anybody else have feedback on this idea - pros/cons? The ticket is here: https://github.com/snowplow/snowplow/issues/1325

Thanks,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean

[Alex Dean]

Whoops ticket is here: https://github.com/snowplow/snowplow/issues/1350