Mock Delta table standalone for testing purpose using local file system, is it possible ?

[Rajesh Datla]

Hi All,

How to Mock Delta table standalone for testing purpose using local file system, is it possible ? without using S3 or blob storage ?

I checked below link but unable to find right solution to Mock Delta tables

https://docs.delta.io/latest/quick-start.html#read-data

Thank You

Raj

[Matthew Powers]

We're working on a Delta Acceptance Testing project (DAT) that contains Delta Lake reference tables.  You can clone the repo and use these tables for testing.

Here's where the first reference table is located.

I'd be cool to open source some code snippets that show how to read the DAT reference tables with Delta Standalone.  A blog post on this would also be useful.  Ping me anytime if you'd ever like to collab on these ;)

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/e5033144-30c7-45bc-9b25-909aeef9c1den%40googlegroups.com.

[Tathagata Das]

Hey Rajesh, 

What kind of testing do you want to do? Delta Standalone currently does not natively support writing the parquet files in a Delta table, it can only read already written Delta tables. So the current best approach is to create Delta tables with Delta-on-spark or delta-rust, and then read them with Delta Standalone. 

What Matthew mentioned, the DAT project, is a community attempt to bridge the gap across all the individual delta projets and make sure they are all cross-compatible in what tables they can read and writes (that is, they all follow the delta protocol correctly). Its very new, so you can play with it, but if you want to really created tables, then best to use what i said.

Hope this helps.

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CANS69_VfoWwKm-dpZ%2Bg8Hs%2BGHpKfH0L58dHLOg89CYOy3s_6kQ%40mail.gmail.com.

[Aurangzeb Khan]

I think, it's just a matter of installing delta locally.

pip install delta 

use PySpark to create / read a table using a local path ./folder/