Delta table partition on quarter and year

71 views
Skip to first unread message

Guru V

unread,
May 27, 2021, 3:54:17 AM5/27/21
to Delta Lake Users and Developers
Hi,
Is there option to have partition by quarter and year.
so that all data for 
2021, JAN, FEB, March -> Partition 1
2021, APril May, June-> Partition 2

similarly for next year
2022, APril May, June-> Partition 10


Thanks

Jacek Laskowski

unread,
May 27, 2021, 4:13:31 AM5/27/21
to Guru V, Delta Lake Users and Developers
Hi,

I think you could have it with a custom partition composite key (not sure if that's a proper name).

Add an artificial column for the group of rows that you'd like to be included in partition 1 and so on.

Just an idea. Never done that before.

--
You received this message because you are subscribed to the Google Groups "Delta Lake Users and Developers" group.
To unsubscribe from this group and stop receiving emails from it, send an email to delta-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/d684f859-0c4b-48c6-a5a3-50ec7633ebb2n%40googlegroups.com.

Jacek Laskowski

unread,
May 27, 2021, 4:28:50 AM5/27/21
to Guru V, Delta Lake Users and Developers
Hi,

BTW Could Experimental support for Generated Columns [1] in 1.0.0 be of some help here? Looks like a viable option.

Chris Hoshino-Fish

unread,
May 27, 2021, 7:03:29 PM5/27/21
to Jacek Laskowski, Guru V, Delta Lake Users and Developers
Yes I would recommend the Generated Columns here - you can partition by Date, and then Generate Columns for Quarter & Year, and Delta will know how those relate to the underlying Date partitions.

To view this discussion on the web visit https://groups.google.com/d/msgid/delta-users/CAB_G8ZtJ7zWcAp6548fpGxdXJoOCvhgBQc4nW-Ha-HOhMdyFag%40mail.gmail.com.


--

Chris Hoshino-Fish

Sr. Solutions Architect

Databricks Inc.

fi...@databricks.com

(415) 610-8520
Pronouns: He/Him/His

Reply all
Reply to author
Forward
0 new messages