Using Django with Gulp.js

[Some Developer]

I'm using Gulp.js to deal with my JavaScript and SASS files to
concatenate them and minify them but I'm unsure how to merge it with my
Django workflow. I want to be able to automatically minify the files
into one big JavaScript and CSS file to reduce HTTP requests and to
allow easy uploading to a CDN but I need to be able to edit to my base
template so that the development version (which links to the files
individually) can change to linking to the big concatenated files. I was
thinking about writing a simple parser that does the changes
automatically but am unsure what the best method to use is.

Should I use a Python HTML library to edit the HTML file or should I use
regular expressions to do the simple text replacement? I should be able
to call the Python script from a Gulp task to do the actual editing. I'd
also want to change the values back when I move back to the development
version of my site.

Any help is appreciated.

[Cal Leeming [iops.io]]

Personally I use a mixture of Vagrant, Docker, Gulp and makefiles to automate the entire process.

You run the gulp process as a backgrounded job, and then the django runserver in the foreground.. of course, this means gulp is then not interactive.. if you want both to be inter, then you have to use two separate shells, or you can use a tmux script to make it pretty.

You could either keep your static files inside the static dir then build to a sub dir, then upon release you would remove all but the subdir, but this is of course a bit hacky. You can instead keep the static files out of the static dir, and only build out to that dir instead. You do not want to be committing compiled js/css, so you should automate this entire process as part of your CI deployment. You can use something like CircleCI and Docker Hub to automate this.

Pro tip - don't attempt to make all this work automagically with django-pipeline or other tools, it will end in tears. I literally spent over a month prototyping many different workflows, and I'm still not 100% happy with the end outcome. Remember that this can quickly turn into a rabbit hole, and also remember that both Gulp and Grunt have *serious* performance issues with larger file counts (100% cpu when using default settings). I haven't found a single article that has a workflow which has done it "correctly", and even my own has problems... if you have a week of time to sink, then prototype your heart out... otherwise, just do whatever gets the job done. 

As for linking to the concat/minified versions, you'd just use settings.DEBUG in your template and link to .min.css if DEBUG is true, there's no need to use HTML libraries, I'm not sure what your reasoning behind that was. Remember that depending on your level of minification, it might break your source or cause weirdness, so you should always run tests against your minified version as part of your CI build.  Another thing to remember, both Gulp/Grunt do not gracefully handle parser errors by default, so you could end up with your build process exiting but the run server still going. I have seen some people that spawn a process from runserver, but this is particularly nasty imho.

I'm actually due to be doing a video about the way that we do it here, but it's unlikely to be released for another month. 

Hope this helps

Cal

PS) You should consider using your real name, or at least an alias, for friendly/public mailing lists like this one. This isn't full-disclosure :)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users+unsubscribe@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/546479DD.6010407%40googlemail.com.
For more options, visit https://groups.google.com/d/optout.

[Some Developer]

Awesome answer. Thanks.

The only reason I considered using a Python HTML library was because I thought you might need to edit the base template to change the JavaScript and CSS links from the individual files to the main concatenated files when deploying to production. I didn't realise you could check for whether the site was in debug mode within the templates (in hindsight this should have been obvious but I didn't think of it).

I currently don't have a CI set up for deployment. I'll look into using CircleCI and Docker Hub. Both sound interesting and should improve my deployment work flow. I already split out my JavaScript / CSS into a different folder than /static so I should be able to compile and place it all in there for deployment purposes.

I'm pretty new to Gulp.js (just started using it for this project) so I appreciate the hints and tips regarding some of the problems associated with it.

I might have a few more questions regarding this over the next couple of days as I figure out the build cycle / work flow so if you could keep an eye open I'd be grateful :).

As for not using my real name I am subscribed to quite a few open source mailing lists and this is the email account I use to catch them all rather than dumping them all in my main email account (I have over 1GB of email in this account at the moment from various open source projects). I prefer to keep my name out of Google as much as possible for privacy reasons.

[Cal Leeming [iops.io]]

No problem at all.

Also one more thing, to access settings constants from templates you'll need this;

http://stackoverflow.com/questions/433162/can-i-access-constants-in-settings-py-from-templates-in-django

Gulp sadly does have many of the same performance problems that Grunt has, however the difference is that Gulp allows you to configure your build programmatically, where as Grunt is configured using an object. Both have good third party integrations for Sass/uglify etc, so the main decision is personal taste. I would advice using gulp-ruby-sass over gulp-sass as libsass has some pretty nasty compatibility problems right now, especially since the introduction of Sass 3.4 (this is assuming you use Sass).

I've pastebin'd one of my more recent gulpfile.js, which should help give you an idea on the sort of "tricks" that are needed in typical builds. This uses my own JS compiler/packer which, although open sourced and available via npm, is currently in stealth/alpha so I'd recommend using something like RequireJS or AMD style loader, browserify is another alternative. Personally I hate them all from an architectural standpoint, hence why I built my own, but it will get you started.

http://pastebin.com/0tuTMAg5

The concept of pipes in Gulp is actually pretty neat, but just bear in mind that all these tooling have limited maturity, and it's not uncommon for builds to randomly break due to backwards incompatible changes between "minor" version changes, so you might want to consider using hard dependancies (fixed versions) in your package.json. I'd recommend building a new container on each deployment so you can detect such breakages early.

You will probably want to look into using nvm/rvm (node/ruby equivalents of python venv), this allows you to fix your ruby/node versions without worrying about breaking system packages. If you're using Docker, then you have to use some horrible trickery to make rvm/nvm install properly without complaining. Note that RVM depends on system packages sometimes, which it attempts to auto install using sudo.. to get around that, you can run the RVM installs as root first, then re-run them as the targetted user, or you can mess around with "autolibs".

http://pastebin.com/9Q9i2d5z

The whole process is a complete and total pita, the slightest change will break things and it takes a while to construct a recipe which works for your specific project... and it will vary between projects too. However the rewards afterwards are excellent... for example, I push to master and 15 minutes later I have a Docker image ready for deployment which contains literally everything it needs and has been fully unit tested. It even handles it's own syslog dispatch to logentries.

Orchestrating the deployment is a completely different beast, so I'll avoid going into detail on that. But a simple "docker pull" on a standalone box on digitalocean running in a tmux/screen session would suffice... ghetto as hell, but gets the job done until you have time to work on the orchestration. There is also AWS, but it's slow and expensive.

I've gone slightly outside of your original question, but hopefully this will give you some food for thought! I'll link the video once it's up, as all the above is just the unorganised ramblings of a crazy programmer ;)

Cal

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.

To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/5465DDD7.90700%40googlemail.com.