Python / Django slow ? Deciding my next technological stack

[Benj]

Hi,

i'm going to invest lots of time and energy in various web projects (mostly community web sites), and want to pick up a language / framework and invest heavily on it.

I've spent a lot of time evaluating the various options, and narrowed my choice to 2 stacks: C sharp asp.net  MVC or Python / Django.

I'm more attracted to Python / Django combo, because of the Python language, and high level framework Django. I really want to use these.

My only concern is speed. I read that Python can't run concurrent tasks, is this true ? So a multi-processor with hyperthreads won't benefit the stack and even slow it down ?

I have no clue how this translates in reality, but should I expect noticable performance difference on a same server, shall I use Python / Django than if I had C Sharp Asp.net ?

I don't want to invest lots of time and efforts only to discover in the end that the site is slow. 
You that have real world experiences with running sites, what are your conclusions ?

I expect sites to be medium traffic: could be handled by a good dedicated server or average cloud ressources.

Benj

[Alex Mandel]

Use a good WSGI like uwsgi or gunicorn and don't use sqlite as your db.
Those WSGI containers are threaded. Don't confuse running python
directly which is different (and there are ways to use multicores even
then).

Plenty of examples of big websites running django out there.
http://blog.disqus.com/post/62187806135/scaling-django-to-8-billion-page-views

Enjoy,
Alex

[Vijay Khemlani]

It's true that the Global Interpreter Lock will prevent threads from a single process from running concurrently, but your server (uWSGI, gunicorn, etc) will spawn multiple process that don't conflict with the GIL.

Also, if you need some form of multiprocessing celery is a good choice, and one that also doesn't collide with the GIL as it is a different process.

Finally, web apps are mostly limited by I/O (database access), so the actual stack used usually is not that important unless you are scaling a lot (StackOverflow-level "a lot").

I've worked with both Django and ASP.NET MVC (although an older version) and even though ASP.NET MVC is amazing in the level of the IDE (VisualStudio) and query language (Entity Framework / LINQ) it's still a little cumbersome for my taste (and I also prefer Linux enviroments...)

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/54ED0EB9.2020505%40wildintellect.com.

For more options, visit https://groups.google.com/d/optout.

[Andrew Farrell]

As Vijay said, most of the performance of a web app (in any framework) is database access. With django, you can use the Django Debug Toolbar or the db.connection.queries object to find out where those queries are being made and then use the select-related and prefetch-related features of the ORM to reduce them to a minimum.

It is also worth noting that for many applications, the speed at which a programmer is able to work effectively matters more for the success of the project than the speed of the language. Of course, the two aren't completely unrelated; If your unit test suite runs slowly, that will slow down your development workflow.

If you have a need to do high-performance analysis of really large scale data sets, python has a variety of tools like blaze, pandas, and numba and is pretty widely used in scientific computing for this reason.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei0T%2B7HokGppre4z7JhPc6P1%3DkLW%2BJ2o1vAtG0v5DD8MtA%40mail.gmail.com.

[Russell Keith-Magee]

On Wed, Feb 25, 2015 at 7:30 AM, Benj <webk...@gmail.com> wrote:

Hi,

i'm going to invest lots of time and energy in various web projects (mostly community web sites), and want to pick up a language / framework and invest heavily on it.

I've spent a lot of time evaluating the various options, and narrowed my choice to 2 stacks: C sharp asp.net  MVC or Python / Django.

I'm more attracted to Python / Django combo, because of the Python language, and high level framework Django. I really want to use these.

My only concern is speed. I read that Python can't run concurrent tasks, is this true ? So a multi-processor with hyperthreads won't benefit the stack and even slow it down ?

I have no clue how this translates in reality, but should I expect noticable performance difference on a same server, shall I use Python / Django than if I had C Sharp Asp.net ?

I don't want to invest lots of time and efforts only to discover in the end that the site is slow. 
You that have real world experiences with running sites, what are your conclusions ?

Instagram is a Django site. Disqus is a Django site. Many of the Mozilla properties are Django sites. I guarantee that these sites serve more traffic than your site will *ever* serve.

Don't get me wrong - these sites have all had to pay careful attention to the design of their systems, and in many cases have made extensive modifications to "stock" Django to support high performance. However - again - they also serve *much* more traffic than your site will *ever* serve.

C#/ASP.net isn't magical performance sauce. Django and/or the Python GIL isn't a magical performance drain. The choices you make in database schema design, caching strategies, query profiling and algorithm choices will affect your site performance *long* before the limitations of Python or Django start to become the limiting factor.

My advice to you - as long as your candidate frameworks meet basic performance requirements, and there's evidence that they can scale to larger sites (ASP.net, Django, and Ruby on Rails all easily meet these criteria), don't worry about performance. Worry about the language you like to use, the speed of development, the community that exists around the tools, and so on. 

Yours,

Russ Magee %-)

[Tom Evans]

Unless you are producing web-scale sites (gmail, ebay, instagram), the
speed of your website will depend more upon what you do with a
framework than the framework you choose.

If you are producing web-scale sites, then whatever framework you
choose you will need to make the right design decisions and/or
compromises.

Even with the best framework in the world, if you design the
architecture of a website poorly, the website will be slow.

Even less important than the choice of framework is the choice of
hosting container for your framework. If you ever get to the point
where the speed of your wsgi container is the thing that is holding
you back, well done, now you should spend some time looking at it.
Until then, use the way that is easiest for you.

Cheers

Tom

[Nikolas Stevenson-Molnar]

I recently came across a recent article addressing many of common myths
about Python. I recommend giving it a read:
https://www.paypal-engineering.com/2014/12/10/10-myths-of-enterprise-python/

As Tom says, it's mostly a matter of how you use the language. You can
write slow assembly and you can write fast Python. There are certain
areas where writing code in another language will yield significant
performance advantages, but for that most part, much of what you'd want
to do has already been efficiently implemented in C and made available
in Python. E.g., numpy (actually some of this is Fortran, too), scipy,
Pillow (for image processing), et. al.

My general thought is that if it works for Instagram (any many other
high-traffic services), it'll probably work for whatever I'm doing.

_Nik

[Benj]

Thanks for quality answers guys, I'll definitely go the Python route and follow my initial intuition.

[Benj]

Just a little questions guys: when a user upload a file to server, and that file is a litte big and takes 3 seconds to be uploaded... what's the proper way, if possible, not to block the entire system for these 3 seconds so the server can serve other users while the upload is taking place. Should this be done in code with things like asyncio, or server with uwsgi workers ?

[Vijay Khemlani]

The rest of the uWSGI / gunicorn / whatever workers are available to handle the other requests.

If there are so many uploads that all of the workers are busy then there are ways to upload files directly to Amazon S3 (for example) from the browser without going through your server.

--

You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To post to this group, send email to django...@googlegroups.com.
Visit this group at http://groups.google.com/group/django-users.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/cfe76a80-af35-4894-a18d-2717c371235c%40googlegroups.com.

[Avraham Serour]

if you have so many uploads that all of yours workers are busy, so maybe you need more workers...

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/CALn3ei2bnQi0FxpMTtvZ1OAP%3D_Q2k0XJze06mv7vCKLc%3D%2BTikQ%40mail.gmail.com.

[Nikolas Stevenson-Molnar]

On 3/1/2015 7:13 AM, Benj wrote:
> Just a little questions guys: when a user upload a file to server, and
> that file is a litte big and takes 3 seconds to be uploaded... what's
> the proper way, if possible, not to block the entire system for these
> 3 seconds so the server can serve other users while the upload is
> taking place.

Let a web server handle the actual upload. Nginx is really good at this,
it can easily handle thousands of concurrent connections. Once all the
data is actually transferred to the server, only then does the request
get passed on to your Django app. A worker is not needed until that
point. This is the default behavior if you use a Nginx / Gunicorn (or
uWSGI) / Django stack.

You can also use the Nginx Upload Module, and your Django app will never
need to receive the actual data at all. Nginx will handle the upload,
put it in a location of your choosing, then hand the path on the server
off to your app. http://wiki.nginx.org/HttpUploadModule

_Nik

[Ilya Kazakevich]

In modern world any heavy loaded site should have HORIZONTAL SCALING. That means it should support scale by adding new servers to cluster.

With horizontal scaling you should not care about how python is slow on each node, until you have traffic like Facebook or Guthub have. I believe 99% websites work well on Python/Django and you have more chances to face database bottlenecks, than Django/Python bottlenecks. You know Instagram, right? They use Django AFAIK. 

CLR (.NET) should probably be faster in some cases (specially with JIT) and it has better multithreading support and better profiling tools, but even with it you will need to create cluster architecture for heavy sites.

So, choosing between C#/.NET and Python/Django is not about performance, I believe. It is about language, platforms, static/no static typing and so on.

[ijazz jazz]

he appropriate tech stack is crucial for a successful web application. ... Ruby (Ruby on Rails); Python(Django, Flask, Pylons); PHP (Laravel); Java .... Needless to say, removing all bugs requires a lot of time and slows down development.

Best Python Training in Chennai

[ijazz jazz]

Python Training in Bangalore Best Python Training Courses in Bangalore –  Marathahalli, BTM Layout, Rajajinagar, Kalyan Nagar & Jaya Nagar. We train the students from basic level to advanced concepts with a real-time environment. We are the Best Python Training Institute in bangalore.