DEP Pre-posal: Re-Designing Django Forms

[Robert Roskam]

Hey All,

Something I've regularly run into as a challenge is the implementation of Django forms work well in the simple use cases, but as the use case grows in complexity, Django forms become more difficult to work with.

I know that's a super general statement, but here's the simplest complex example I can give you. Lets say you're making an application for a home builder, so that their Project Managers can better coordinate the builds. One of the features is the ability to take notes and pictures on anything that's not yet done and specifically if it relates to a specific piece of equipment (AC, furnace, water pump, etc), they can add that too. Below is a moderately simplistic example:

class Note(models.Model):

    project = models.ForeignKey('project_management.Project', related_name="notes")

    equipment = models.ForeignKey('equipment.Equipment', null=True, blank=True, related_name="notes")

    picture = models.FileField(null=True, blank=True)

    is_blocker = models.BooleanField(default=True)

    comment = models.TextField()

    created_by = models.ForeignKey('users.User', verbose_name="Created By")

    created_date = models.DateTimeField(default=timezone.now, verbose_name="Created Date")

class NoteModalForm(forms.ModelForm):

    class Meta:

        fields = ('comment', 'is_blocker','equipment', 'picture')

        model = Note

        labels = {

            'comment': 'Note',

            'is_blocker': 'Blocking Issue',

            'picture': 'Picture',

        }

        widgets = {

            'picture': DragNDropFileButtonInput(button_text='Select file to Upload',

                                                    button_classes='btn btn-bordered uploader'),

        }

General comments first: 

1. I would say there's no better way to accomplish what is currently on that form given the current Form Meta API. (Willing to be challenged on this point, btw.)
   
2. The repetition of picture 3 times over (fields tuple, labels dict, widgets, dict) seems to be inherently not DRY. If this gets very long, then it becomes harder to manage.
3. The API on the Model Form itself behaves not quite like you'd expect initially. You'd expect redeclaring fields directly on a form for it to function like a dictionary update, if the value doesn't exist in the incoming dictionary, it keeps what's there. It actually behaves like re-declaration. This very significant behavior is buried in a note (https://docs.djangoproject.com/en/2.0/topics/forms/modelforms/#overriding-the-default-fields). Additionally, you'll have sources like pydanny basically tell you this is an anti-pattern: https://www.pydanny.com/overloading-form-fields.html
4. The API on Meta leads you to believe initially that you can override lots of things via Meta, and it's difficult to discover what is or is not supported. (I usually dig into django.forms.models, and then wander around until I get to ModelForm.) 
• Here's the list: model, fields, exclude, widgets, localized_fields, labels, help_texts, error_messages, field_classes. 
• What's missing that's on a default field? Required, initial, show_hidden_initial, validators, disabled, label_suffix. 
• Anything not on this list you basically need to override in __init__
5. The Django Form API both does and does not care about the client side implementation. I say it does care because it sets the default values in the DOM if you use the form as recommended in the docs. However, there is no recommended approach for anything beyond this. 

Here are some simple scenarios that are made difficult:

Scenario #1

If I'd like on this form--and this form only--to make picture always required or change the default value of is_blocker, we have to override __init__. And generally it looks like this:

    def __init__(self, *args, **kwargs):

        super(NoteModalForm, self).__init__(*args, **kwargs)

        self.fields['picture'].required = True
        self.fields['is_blocker'].initial = False

Scenario #2

If I have the feature request that notes that are blockers require pictures or notes that have equipment requires pictures, then you have to implement this pattern:

https://docs.djangoproject.com/en/2.0/ref/forms/validation/#cleaning-and-validating-fields-that-depend-on-each-other

Oh, but wait! The best in class approach to this problem would be to inform the user before they submit the form that this is a problem. This catches it only on the server side. How should we implement that? You're on your own to figure it out.

So in summary, I feel like this could be improved. And I'd like to take an whack at it!

Robert Roskam

[Collin Anderson]

I personally use the (undocumented?) formfield() method, which takes the model defaults and lets you override things. I think it's pretty elegant, though maybe it could use some less verbose syntax (maybe have a forms.ModelDefault(label='Note') that does this for you).

class NoteModalForm(forms.ModelForm):

    comment = Note._meta.get_field('comment').formfield(label='Note')

    is_blocker = Note._meta.get_field('is_blocker').formfield(label='Blocking Issue', initial=False)

    picture = Note._meta.get_field('picture').formfield(label='Picture', required=True, widget=DragNDropFileButtonInput(button_text='Select file to Upload', button_classes='btn btn-bordered uploader'))

    class Meta:

        model = Note

        fields = ['comment', 'is_blocker', 'equipment', 'picture']

I should also note, you can change fields without overriding __init__; just use base_fields:

NoteModalForm.base_fields['picture'].required = True

NoteModalForm.base_fields['is_blocker'].initial = False

As far as client-side validation goes, yes, Django really only does client-side validation that's available from plain html. Do you have some ideas for how dependent fields should work? I personally think it would be hard to find a one-size-fits all solution for more complicated cases, but it's probably possible. I think there are some 3rd party libraries that do this. I think a first step would be to natively handle dependent-fields programmatically in the back-end, so the form "knows" that those fields are related. Then maybe django could pass that relationship information to the html as data-* attributes.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8fc29cb6-d059-4e49-bb76-309d0404ecd5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Marc Tamlyn]

Hi Robert,

I have a whole heap of ideas about this, some of which are captured in documentation here: http://django-adapters.readthedocs.io/en/latest/ The code generally doesn't exist yet, but the project is at https://github.com/mjtamlyn/django-adapters.

This is a huge project to achieve everything you mentioned in your email, and it has implications across a large number of Django packages (not least the admin). I don't want to discourage you, but don't underestimate how much work it would be to get a good replacement for forms for the modern web.

Your next steps should be to research, spec and potentially write a DEP.

Marc

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAFO84S5QEgN8tnP2Xk%2BS6UxeNqjNdZ%2B13_J5rRnsDY9g4j88Rw%40mail.gmail.com.

[Robert Roskam]

Hey Marc,

Yeah, I totally agree it's a big job. It's just that the first step for proposing (sry about the previous misspelling) the idea. I saw the issues I've run into, and It just seemed like a better interface could be provided. 

I'll look at what you have worked on.

Robert Roskam

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/8fc29cb6-d059-4e49-bb76-309d0404ecd5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

[Jamesie Pic]

On Thu, Feb 1, 2018 at 12:46 PM, Marc Tamlyn <marc....@gmail.com> wrote:

> This is a huge project to achieve everything you mentioned in your email, and it has implications across a large number of Django packages (not least the admin). I don't want to discourage you, but don't underestimate how much work it would be to get a good replacement for forms for the modern web.

Perhaps we should just be able to swap Forms with WTForms or another python library and bake in ElementUI, even if that means replacing template_name with vue_name in the view generic class, but if we're talking about "modern web" then perhaps it's time for npm to become a first class citizen.

> Your next steps should be to research, spec and potentially write a DEP.

In my recent research it seemed ElementUI the most feature complete UI. It includes ajax file upload which every user expects in the modern web which seems to be the feature which defines feature-completion of a UI framework, compared to what HTML offers out of the box.

Thanks a lot for doing something about this Robert, forms in django definitely needs a major refactoring sprint, typically remove the field/widget layer and rely on one level inheritance that will help a lot for example with material design which displays field.name inside the widget, not possible with current object design.

[Matthew Pava]

> It includes ajax file upload which every user expects in the modern web which seems to be the feature which defines feature-completion of a UI framework, compared to what HTML offers out of the box.

Bandwagon logical fallacy (“every user expects”)

 

In my project, I have no need for file upload.  And I have no need for it in the foreseeable future, and, as such, I do not expect my framework to be able to handle it.

 

I’m neither supportive nor opposed to the idea of changing Django forms, but please avoid logical fallacies in supporting your position.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.
Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1_ESqUA6tUwQxwgastH4XzQ%3D-PBybtq__2yWEuc0OH4BA%40mail.gmail.com.

[Tom Forbes]

> Perhaps we should just be able to swap Forms with WTForms or another python library and bake in ElementUI,

There are a plethora of UI frameworks with different tradeoffs, I really don't think Django sound pick one.

However a stronger integration with the JS-build tools of the day like Yarn, webpack et al would be a good step in the right direction, and more agnostic than just 'use this UI framework we picked for you'.

> perhaps it's time for npm to become a first class citizen.

I think if Django is going to stay relevant to building most/all kinds of web apps this is something that will need to happen. To be fair, npm and the whole frontend ecosystem has only really 'matured' in the last couple of years, it's come a long way since Django started.

--

You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-developers+unsubscribe@googlegroups.com.
To post to this group, send email to django-developers@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-developers/CAC6Op1_ESqUA6tUwQxwgastH4XzQ%3D-PBybtq__2yWEuc0OH4BA%40mail.gmail.com.

[Robert Roskam]

Hey Tom,

My main goal is to improve the Django Form API interface for writing Python code for what Django has historically done: server-side rendered web pages. So any client-validation hooks I might provide would not be fully implemented integrations rather a more convenient integration. (I'm thinking of things like Parsley.js here.)

If I can happen to improve the API for interfacing with JS frameworks, then I'll be happy to do so, but that's honestly not even secondary in importance to me. However, my experience so far as been they assume a REST API on the backend and otherwise want to be in charge of the DOM inside them.

Robert Roskam

On Monday, February 5, 2018 at 5:26:25 PM UTC-5, Tom Forbes wrote:

> Perhaps we should just be able to swap Forms with WTForms or another python library and bake in ElementUI,

There are a plethora of UI frameworks with different tradeoffs, I really don't think Django sound pick one.

However a stronger integration with the JS-build tools of the day like Yarn, webpack et al would be a good step in the right direction, and more agnostic than just 'use this UI framework we picked for you'.

> perhaps it's time for npm to become a first class citizen.

I think if Django is going to stay relevant to building most/all kinds of web apps this is something that will need to happen. To be fair, npm and the whole frontend ecosystem has only really 'matured' in the last couple of years, it's come a long way since Django started.

On 4 Feb 2018 00:36, "Jamesie Pic" <jp...@yourlabs.org> wrote:

On Thu, Feb 1, 2018 at 12:46 PM, Marc Tamlyn <marc....@gmail.com> wrote:

> This is a huge project to achieve everything you mentioned in your email, and it has implications across a large number of Django packages (not least the admin). I don't want to discourage you, but don't underestimate how much work it would be to get a good replacement for forms for the modern web.

Perhaps we should just be able to swap Forms with WTForms or another python library and bake in ElementUI, even if that means replacing template_name with vue_name in the view generic class, but if we're talking about "modern web" then perhaps it's time for npm to become a first class citizen.

> Your next steps should be to research, spec and potentially write a DEP.

In my recent research it seemed ElementUI the most feature complete UI. It includes ajax file upload which every user expects in the modern web which seems to be the feature which defines feature-completion of a UI framework, compared to what HTML offers out of the box.

Thanks a lot for doing something about this Robert, forms in django definitely needs a major refactoring sprint, typically remove the field/widget layer and rely on one level inheritance that will help a lot for example with material design which displays field.name inside the widget, not possible with current object design.

--
You received this message because you are subscribed to the Google Groups "Django developers (Contributions to Django itself)" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-develop...@googlegroups.com.
To post to this group, send email to django-d...@googlegroups.com.

Visit this group at https://groups.google.com/group/django-developers.

[Dmitriy Sintsov]

Hi.

Client-side implementation of fields visual behavior and their validation should be hooked up via document.ready custom JS scripts via form ID or form class. There is no universal solution for that.

When using pure client-side forms from npm, it is possible to map these to Django models via DRF, however there will be no custom server-side widgets then. To me one of the most important parts of Django Forms is the ability to develop your own custom widgets.

Server-side validation of forms can be processed with AJAX, there are few implementations of that, including mine:

https://github.com/Dmitri-Sintsov/django-jinja-knockout/blob/master/docs/forms.rst#ajax-forms-processing

It saves quite enough ot HTTP traffic, because instead of complete re-rendering of submitted form HTML page, only the list of field ID's and their error messages are returned to be highlighed by client-side Javascript code.

It also allows to submit forms in AJAX "popup-like" dialogs:

https://github.com/Dmitri-Sintsov/django-jinja-knockout/blob/db4beeb18296c87af8626980b6a0c91e16f80863/django_jinja_knockout/views/ajax.py#L203

It supports file upload progress bar automatically as well:

https://github.com/Dmitri-Sintsov/django-jinja-knockout/blob/db4beeb18296c87af8626980b6a0c91e16f80863/django_jinja_knockout/static/djk/js/app.js#L1568

It's not so flexible (tied to Bootstrap 3 via Jinja2 macros) but it's possible to generalize such approach.

With client-side Javascript there is always the trouble which library to chose - Angular / Vue / React - cannot suit everyone. I use Knockout.js which is outdated but requires no webpack / npm and works with ES5. It is also fully compatible to server-side DTL / Jinja2 templates (it does not use double curly braces, thus causes no syntax clashes).

Dmitriy