GSoC Proposal: Add Cross-DB JSONField, ArrayField, and HStoreField

[Sage M.A.]

Hello, everyone! My name is Sage. I'm a 19-year-old computer science student from Indonesia. I'm planning to join the Google Summer of Code (GSoC) this year, and I want to contribute to Django. I have written a draft for my proposal in this gist. I have submitted two small patches for Django, and I hope to contribute more in the future. Feedbacks are much appreciated, thanks!

[Michal Petrucha]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

On Tue, Apr 02, 2019 at 04:41:36AM -0700, Sage M.A. wrote:
> Hello, everyone! My name is Sage. I'm a 19-year-old computer science
> student from Indonesia. I'm planning to join the Google Summer of Code
> (GSoC) this year, and I want to contribute to Django. I have written a
> draft for my proposal in this gist

> <https://gist.github.com/laymonage/b53a1acbbab36b77776cd526b48fd2a5>. I

> have submitted two small patches for Django, and I hope to contribute more
> in the future. Feedbacks are much appreciated, thanks!

Hi Sage,

I'm curious about your intention to implement ArrayField and
HStoreField for other databases – you don't really go into any detail
about those. Are those types even supported by other SQL databases
than Postgres? I can't find any hstore or array type for mysql, for
instance, let alone sqlite, and it's been my understanding that these
types are pretty specific to Postgres.

Could you explain in more detail why and how you want to port those
field types?

Michal
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCgAGBQJco09GAAoJEHA7T/IPM/klB6oP/2I1TaEFG/SzRh5ArjTEzH60
Nd46at1133iJEL/QEz1SzFFkoXT7uRe+71aJroW8NE4KUEXN4giOO/jjhAYB74cv
Rzxay45r03g7Io+labwe0ZBIMhfhKRbk+8W4QVyHY8yfZklazwiswRy/3zJENoA4
GqML3yQ/0oq72pQtHHbhALp5FCcE5kILadCoYMqJ3BrqQEt5aWi3oQWoIYBCqkXF
zV3aM8cH4xsLVHG28646Eu10o5vsf06poYxOpQpF7OwfMy0k2mAjQyfVCeZ6qJzX
2aQpimgjiICBD21Asme4kBUEBmxdGiAiz15zQlnZBW+JeXzQsCFqUf+/T6PqTeyU
0uiX4O8/LhW+4PRG/aXLszcgqsDGOwFk3OstcrI3d+BlJ39ZCM0Kv7Y6L8H5DYKm
g9IfMeqD2c+/QECG3tEMhtJwhrvMsxod3Upq+6DhtK3G7xxemlRHjEvOIr/EmV8X
I9HRAU5ZmbE1piwZx45oTOD9ZCI2HsQtHuJ6jVsF2mwgdL3FdRGrLqCb2TPV/61x
0P1ehjukYU5fBcdua/2sgD1nRjzQBw54yhlwaTiUoL82JmLoW8tzKCG6mycMfbtY
TfXG9wP7Ig+mWYcRdINoM83NSaVQG+Fi98KxJoW/3BFpk3mz4QGrMBLFbs6P3ANc
9+IFrRA/wvJW2+jjBtqo
=8GaE
-----END PGP SIGNATURE-----

[sagar ninave]

I appreciate Sage

[Sage M.A.]

Hi Michal,

From what I know,

SQLite and MySQL have the json_array() function which can be used to store arrays.

https://www.sqlite.org/json1.html#jarray

https://dev.mysql.com/doc/refman/8.0/en/json-creation-functions.html#function_json-array
Meanwhile, Oracle Database stores JSON as a VARCHAR with IS JSON constraint, and arrays are a valid JSON value.

https://docs.oracle.com/en/database/oracle/oracle-database/18/adjsn/overview-of-storage-and-management-of-JSON-data.html

About HStore, yes, it's not widely used but it's basically just a flat (not nested) JSON.

As to why I want to port those field types, to be honest, I don't really think it's necessary, but they're nice to have.

However, the idea listed on Django's GSoC 2019 page says that those fields would be desirable.

Those two fields would not be my main priority, and I guess maybe their implementations could be optional.

[Sage M.A.]

Thank you.

On Tuesday, 2 April 2019 19:26:29 UTC+7, sagar ninave wrote:

I appreciate Sage

[sagar ninave]

by the way how much you know django

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b02ac6c4-7db3-4d3a-ae08-d1fa848b4b68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Sage M.A.]

Hi Sagar, 

I started using Django last year. I've been studying the codebase lately, and I'm still trying to understand the mixins and stuff for the database backends. I have submitted patches for two tickets (#30294 and #30295), and I'm looking to contribute more in the future.

On Wednesday, 3 April 2019 04:12:52 UTC+7, sagar ninave wrote:

by the way how much you know django

On Tue, Apr 2, 2019 at 8:52 PM Sage M.A. <laym...@gmail.com> wrote:

Thank you.

On Tuesday, 2 April 2019 19:26:29 UTC+7, sagar ninave wrote:

I appreciate Sage

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.

[sagar ninave]

same bro i am too, actually i have learned python in last semester and using djnago build a small project and trying to get more about django. if you dont mind can i ask question when i will stuck somewhere about django and you may ask to me if you will not getting we will try to understand what matter will be

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.

To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/3737f569-263d-4d09-acb2-2fdc1e2bebf8%40googlegroups.com.

[Sage M.A.]

Hi Sagar,

That's cool! However, I am not quite sure what you meant. If you meant to ask questions on how to use Django,

I think it's best to ask them in the django-users forum or the #django channel on IRC. I'm sure the community

will be able to help you. I also join those rooms when I can. If you actually meant to ask questions about Django

itself, then I guess you could ask your questions by posting a new topic in this django-developers forum instead

of this topic. This topic is meant to be a place for me to gather feedbacks for my GSoC proposal.

Thank you for your interest :)

On Wednesday, 3 April 2019 18:32:03 UTC+7, sagar ninave wrote:

same bro i am too, actually i have learned python in last semester and using djnago build a small project and trying to get more about django. if you dont mind can i ask question when i will stuck somewhere about django and you may ask to me if you will not getting we will try to understand what matter will be

On Wed, Apr 3, 2019 at 8:16 AM Sage M.A. <laym...@gmail.com> wrote:

Hi Sagar, 

I started using Django last year. I've been studying the codebase lately, and I'm still trying to understand the mixins and stuff for the database backends. I have submitted patches for two tickets (#30294 and #30295), and I'm looking to contribute more in the future.

On Wednesday, 3 April 2019 04:12:52 UTC+7, sagar ninave wrote:

by the way how much you know django

On Tue, Apr 2, 2019 at 8:52 PM Sage M.A. <laym...@gmail.com> wrote:

Thank you.

On Tuesday, 2 April 2019 19:26:29 UTC+7, sagar ninave wrote:

I appreciate Sage

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/b02ac6c4-7db3-4d3a-ae08-d1fa848b4b68%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-d...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

[Carlton Gibson]

Hi Sage. 

Thanks for the proposal. It's looking OK. Couple of points: 

• There IS an Oracle implementation. See the ticket here: https://code.djangoproject.com/ticket/29821
  
• Something that looks like an ArrayField, yes. HStore... not so sure it's worth mimicking. 
• On the timeline: I think you've spread the coding bit too thin and not allocated enough for Documenting.
• I think you if you want full-guns at the SQLite PoC in week 1 and 2 you'd have something in place quite quickly. 
• The base field should be simple enough™
• The SQLite only lookups shouldn't be too complicated. 
• As I commented on the other thread on this topic, we'll need to advise on getting people set up with SQLite with the JSON extension. 
  
• There's more in this that you think I'd guess. No harm in putting time in the schedule for it. 
• Allow for the possibility you complete early and have time to work on other things...
• This stuff is difficult to get right. It's more balance that exact times: 
• The actual timeline won't match the plan ever. 
• you don't need to worry about two days off for holiday 🙂

Your contributions so far have been super. Thank you. 

Kind Regards,

Carlton

[Sage M.A.]

Hi, Carlton.

Thanks a lot for the feedback!

Ah, yes, looks like I missed that Oracle implementation. I've updated my proposal accordingly.

To summarize:

• Added info about Oracle implementation (see 1.1)
• Replaced 1 week of research with 1 week of writing tests and docs for POC SQLite JSONField (see 3.1.1 and 3.1.2)
• Replaced the idea of implementing HStoreField with writing documentation for ArrayField, migration path, and SQLite+JSON1 (see 3.3.2)
• Took half a week from the merging process into writing the aforementioned docs.
• Wrote about the possibility of merging the first and second milestones so the final milestone would not be so big to merge. (see 3.1.2 and 3.2.2)
• Wrote about the possibility of finishing the project early (see 3.4)

About #29548, it seems interesting to me but I'm not quite sure what's left to be done.

Regards,

Sage

[Asif Saif Uddin]

Sage, regarding MariaDB, I have a little suggestion with you. If you could setup a latest version of mariadb locally and try to run Django test suits with that, you might have some ideas of remaining issues analyzing the test failures initially.

[Sage M.A.]

Hi Asif,

That's a good idea! I've installed the latest stable release of MariaDB (10.3.14) on my machine and ran Django test suits with it.

The results (I've truncated them to only contain the failures and errors) show 19 failures and 15 errors, all of which seem to be

related to datetime functions. I suppose that'll be enough for me to work on if my GSoC project finishes early. I'll also need to

write docs about the new support for MariaDB.

On a side note, setting up and destroying the database for the test took hours on my machine. Is that normal? I had to use

verbosity=2 to find out that it was not freezing, just really slow.

[Sage M.A.]

Oh, I forgot, one of them is not related to datetime functions: test_unsupported_backend.

[Asif Saif Uddin]

You can open a work in progress pr to fix them one by one gradually and trying to fix some of them will give you a much better understanding of django orm internals as it is considered the most complex part of the framework. This will also improve your chance to represent yourself as a much stronger candidate and you will be able to interact with the community in a very meaningful way. Just my 2 cents. 

[Tim Graham]

MariaDB is supported (see https://code.djangoproject.com/ticket/29548). The test failures are (as the error message says) because you haven't installed the timezone definitions.

[Sage M.A.]

Thanks for the information Asif, but it turns out Django already supports MariaDB.

[Sage M.A.]

I see, thanks for the info! Why is the ticket still open, though? Also, I still found one failing window test case..

[Raphael Michel]

Hi Sage, Hi everyone,

I lacked the time to read this mailing list in the past months, and
someone at DjangoCon Europe just pointed me to this thread, so just for
reference, I want to add some prior work that I did to the list:

At DjangoCon US and thereafter, I developed a third-party
implementation that does a little more than the ones that are more
well-known and mentioned in the proposal:

https://github.com/raphaelm/django-jsonfallback

django-jsonfallback supports native JSON fields on PostgreSQL, MariaDB
*and* MySQL (which are quite different in that regard). On all other
databases, it falls back to the behaviour of just storing strings in
text fields. Therefore, it's still different from what is being
proposed here, and also the implementation is quite unclean in the
sense that it needs to check for database backends on the level of
database fields. When implemented in core, this could be done much more
nicely.

Still, it might be an interesting thing to look at when doing this, if
only for the test cases and the weird MySQL/MariaDB things. I'd love to
see this in core and are happy to help wherever I can with my expertise.

Best
Raphael

Am Tue, 2 Apr 2019 04:41:36 -0700 (PDT)
schrieb "Sage M.A." <laym...@gmail.com>:

> Hello, everyone! My name is Sage. I'm a 19-year-old computer science
> student from Indonesia. I'm planning to join the Google Summer of
> Code (GSoC) this year, and I want to contribute to Django. I have
> written a draft for my proposal in this gist

> <https://gist.github.com/laymonage/b53a1acbbab36b77776cd526b48fd2a5>.

[Sage M.A.]

Hi, Raphael.

Ah, yes, I have seen your project, but I guess I forgot to link it in my proposal.

It is very interesting, and I believe it will be helpful for anyone who wants to implement

this. Let's hope we can make this happen. Thanks for the feedback!

Regards,

Sage

[Asif Saif Uddin]

Hi Sage,

Would you mind opening a draft proposal at django/dep repo? If you can implement the POC cross db JSON field, I as a maintainer of django-activity-stream would like to try this with that package.

Looking forward to try your POC. 

Thanks,

Asif

[Sage M.A.]

Hi Asif,

I don't mind at all, but I'm still a bit busy with college stuff now, so I'm not sure when I'll be able to do it.

[Sage M.A.]

Hello everyone,

I would like to thank everyone in here and everyone involved in Django's GSoC participation. This project has been accepted and I will try my best to deliver.

On a side note, do we have any communication channel other than this mailing list and IRC? I don't mind using them, but I guess a faster and easier-to-access channel would be very helpful in the development of this project. I lost context every time I disconnect from IRC, so it's not very desirable.

Or, is django-core-mentorship more appropriate? There hasn't been much going on there, so I'm not really sure.

[part...@gmail.com]

I had the same doubt in my mind, what is the best medium to contact mentors?

[Carlton Gibson]

Hey Sage, Hey Parth. 

First-off welcome on board! 🙂

The goal for the next couple of weeks is to get more deeply involved, whilst I guess you start thinking about your projects too. You can follow here, the dashboard and the timeline — you don't need to be on top of it all!!! — to see what's going on. 

I'll email you off list to get the conversation started. Then I think using django-core-mentorship for group conversations isn't a bad idea, and then we can have you post periodically to django-developers here to report on progress, or if we need to ask questions more widely. 

Well done again. 

Kind Regards,

Carlton