Django asychronous for millions of users

[Josh moten]

I am creating an auction that will be hosting millions to possibly billions of users so I am trying to figure out which would be the best library to use for scalability at a rapid pace. I am positive to reach a million users in four months. What is the best library and/or package to us for over 100k+ users per second?

[Lars Liedtke]

Hello Josh,

somehow this sounds a bit sketchy to me. Could you please extend your request to your plans to setup such a system. This is not simply done with choosing "the right"  library or package. This has to be a quite elaborate system with multiple kinds of Software, like caches as redis and/or elasticsearch, most propably load balancers and multiple application servers (which could be run with django). But simply sending a question like you did will not solve the prpably appearing problems you'll be facing, if you really will that many users.

Cheers

Lars

Am 30.03.21 um 13:32 schrieb Josh moten:

I am creating an auction that will be hosting millions to possibly billions of users so I am trying to figure out which would be the best library to use for scalability at a rapid pace. I am positive to reach a million users in four months. What is the best library and/or package to us for over 100k+ users per second? --
You received this message because you are subscribed to the Google Groups "Django users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b0ed59a5-5892-4a0e-b159-adc89476da78n%40googlegroups.com.

-- 
---
punkt.de GmbH
Lars Liedtke
.infrastructure

Kaiserallee 13a	
76133 Karlsruhe

Tel. +49 721 9109 500
https://infrastructure.punkt.de
in...@punkt.de

AG Mannheim 108285
Geschäftsführer: Jürgen Egeling, Daniel Lienert, Fabian Stein

[Josh moten]

I am trying to implement the right infrastructure, so that I can be able to support all the users. I am in no way trying to deceive anyone. I am asking because I truly want to make the right decisions from the beginning, so that it will be cost effective. I am thinking of using AWS elastic band, so that proper load balancing will be in place. I am also looking up cacheing as well

Sent from my iPhone

On Mar 30, 2021, at 7:13 PM, Lars Liedtke <lie...@punkt.de> wrote:



You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/7XmCZgz3XzI/unsubscribe.
To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/b93ddf40-0c57-732c-36af-c53728bb7cd4%40punkt.de.

[Lars Liedtke]

Sorry for having been so blunt, but your mail looked to me like some random person asking for how to simply implement something with $framework.

But my remarks had a lot of expirience in them. If you are still not decided on a language or framework then I would suggest you choose the language you are most familiar with and the framework, that offers what you expect from it. Before deciding though you should  also have a look at the integration of caching, because you will definetely need something like e.g. redis and maybe varnish. Additionally you should think about which database you want to use and if this can do clustering.

Django can do all of this and might be the right choice for you, but have a look at how Django "thinks" and decide, if this is a way that you could familiarize yourself well with.

As often in IT there are now real "right" choices, only choices that will be right for you and your usecase.

Cheers

Lars

Am 31.03.21 um 01:28 schrieb Josh moten:

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/10F9EE9C-EC21-4BF3-9C42-490D95940A1A%40gmail.com.

[Derek]

I agree with Lars. You have said very little about your actual situation; my concern would be that if someone was appointed to develop a web-based system that was expected to scale that big and that fast I would expect (in fact, if I was the project funder I would *demand*) that you already have a number of years experience developing, deploying and maintaining such systems - preferably in more than one domain.  There are very smart and knowledgeable people on this mailing list - you may want to consider hiring or partnering with someone like that who can offer you this capability.

[Josh moten]

I truly would love to do that but I know I do not have the funds to hire someone with that much experience. I am disabled so I only get paid once a month and what’s left after bills and groceries is only $200 a month so I am having to bootstrap everything. It would be very disrespectful to ask someone with that much ability and experience to basically be working for free.  I am open to anyone who wish to help. I do have a private GitHub repository for my website so if anyone is interested please let me know.

Again I do not wish to disrespect anyone and I know I am a beginner but I know this can be done Ive only be programming since October of last year and I have made leaps and bounds

Sent from my iPhone

On Mar 31, 2021, at 10:05 AM, Derek <game...@gmail.com> wrote:

I agree with Lars. You have said very little about your actual situation; my concern would be that if someone was appointed to develop a web-based system that was expected to scale that big and that fast I would expect (in fact, if I was the project funder I would *demand*) that you already have a number of years experience developing, deploying and maintaining such systems - preferably in more than one domain.  There are very smart and knowledgeable people on this mailing list - you may want to consider hiring or partnering with someone like that who can offer you this capability.

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/02c94a94-e7a0-4791-87f0-0eab85ba5dd2n%40googlegroups.com.

[Lars Liedtke]

I was on the brink of proposing this before, but if you'd like we could have an online beer together and talk of course without any cost, because we are just having a beer or any other beverage of your liking. Just send me a mail privately and we'll find an appointment

Am 31.03.21 um 18:09 schrieb Josh moten:

To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/4BD733A1-66C7-4ECD-9437-5D8676F6143F%40gmail.com.

[Josh moten]

Yes Lars I would love to do that I just sent you an email just now 

Sent from my iPhone

On Mar 31, 2021, at 2:54 PM, Lars Liedtke <lie...@punkt.de> wrote:



To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/78c08e12-d353-83b2-feb1-9f5b40a8c4ac%40punkt.de.

[Samuel Smith]

> --
> You received this message because you are subscribed to the Google
> Groups "Django users" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to django-users...@googlegroups.com

> <mailto:django-users...@googlegroups.com>.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/django-users/b0ed59a5-5892-4a0e-b159-adc89476da78n%40googlegroups.com

> <https://groups.google.com/d/msgid/django-users/b0ed59a5-5892-4a0e-b159-adc89476da78n%40googlegroups.com?utm_medium=email&utm_source=footer>.

Just being "asychronous" won't get you there. Start small and make sure
you are treating your database right. Use database views to limit
excessive queries and to better aggregate your data instead of trying to
stuff everything into a python object. Use a CDN to host as much static
files as possible and even provide limited caching.

And above all, don't plan out your "scaling" problems until you actually
have enough money flowing to do so. I once worked for a startup that
planned out having over a million customers on one platform. This was
before the days of cheap AWS instances and so we allocated the needed
resources and rack space in a data center and paid for expensive
licensed software that was "web scale". We were spending over 1.5
million a year on hosting and software license costs. After 5 years we
had finally grown to 70,000 customers but that still did not cover the
costs of running the company and after a couple more years, closed shop.

If we would have just started off simple with single django instance
running on a box in a closet somewhere that company would probably still
exist. They had a good product and market, but planning for a
hypothetical future killed it.

Regards,

[Josh moten]

Samuel,

Thank you for that tremendous insight; sad to say you are correct on me trying to plan for the future of scaling. I am so use to having to plan ahead and have a backup for my backup which has a series of more backup to the original plan. I still operate as if I am still in the Marine Corps so thank you for helping me to see that I should first start simple with a simple deployment and later scale as my users grow. The thing I am trying to plan correct for scaling is the asynchronous technology part which from the conversations thus far from Lars and the rest I have found out about load balancer and cacheing. Thank you so much

Sent from my iPhone

> On Apr 1, 2021, at 11:17 AM, Samuel Smith <dja...@net153.net> wrote:

> You received this message because you are subscribed to a topic in the Google Groups "Django users" group.
> To unsubscribe from this topic, visit https://groups.google.com/d/topic/django-users/7XmCZgz3XzI/unsubscribe.

> To unsubscribe from this group and all its topics, send an email to django-users...@googlegroups.com.

> To view this discussion on the web visit https://groups.google.com/d/msgid/django-users/fb8588ae-6297-c6a9-6512-632e9876caa0%40net153.net.