Delta Lake for unstructure and semi-structured data

[Pavel Kazlou]

Can I use Delta Lake to store and process unstructured and semi-structured data(json, xml, images, videos, text, etc.)? Or I should use another solution?

[Chris Hoshino-Fish]

Delta lake is a great option for storing unstructured data due to the data types it can support. Delta stores all your data in the Parquet file format currently, so your data will be stored in a columnar file format. But Parquet supports unstructured/semi-structured types like: 

• StructType
• MapType
• ArrayType
• BinaryType

This means you can do things like parse some of your JSON into top-level columns, but leave other parts in a Struct with the same key-value structure. Or you can store videos/images as Binary types and do your own decoding.

On Thu, Jan 21, 2021 at 9:39 AM Pavel Kazlou <pavel....@fortegrp.com> wrote:

Can I use Delta Lake to store and process unstructured and semi-structured data(json, xml, images, videos, text, etc.)? Or I should use another solution?

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/74af1dde-6cd1-47ca-9f89-7263d3bf2a57n%40googlegroups.com.

--

Chris Hoshino-Fish

Sr. Solutions Architect

Databricks Inc.

fi...@databricks.com

(415) 610-8520

[a...@databricks.com]

Also, have a look at this blog which uses Delta to store images in Delta to make ML ETL and inference much faster (processing millions of tiny images on S3 can be very slow and costly, much more efficient in Delta).

https://docs.databricks.com/applications/machine-learning/reference-solutions/images-etl-inference.html

[Denny Lee]

Quick note, there was a fun Data + AI Online Meetup on this topic on Tuesday: 

https://www.meetup.com/data-ai-online/events/275872626/. 

You can watch the video at: 

https://youtu.be/79PkqWkSxYU 

HTH!

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/17e33531-8036-4bff-8e8c-0bb063a2dfefn%40googlegroups.com.

[Chidananda Unchi]

Hi Team,

I want to tokenization data after reading data . can you please suggest any UDF or pattern to tokenization data.

Regards,

Chidananda

On Thu, 21 Jan 2021, 23:09 Pavel Kazlou, <pavel....@fortegrp.com> wrote:

Can I use Delta Lake to store and process unstructured and semi-structured data(json, xml, images, videos, text, etc.)? Or I should use another solution?

--

[Shixiong(Ryan) Zhu]

Hey Chidananda,

What kind of tokenization are you looking for? Spark has a SQL expression split which can be used to do simple tokenization. For complex cases, I usually write my own Scala UDF to manipulate strings directly.

Best Regards,

Ryan

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAOFUOrUDbSCiMk%2BTSbLiorD3b%2BpgYnCnaCnRi3_1zGqMuuuNfQ%40mail.gmail.com.