Scaling with Kotti

[David Bain]

We're doing a project now that will need to support about 1000 users actively logged in (but only for 1 month out of the year).

The application is mostly designed to provide private information so lots of specific permissions, so I'm concerned about doing too much caching.

Our plan at the moment looks like this:

• have a dedicated postgres db
• run multiple kotti client instances (possibly on multiple machines)
• benchmark with funkload (because we can simulate logged in users)

I would appreciate comments on my ideas including other things that we should be considering.

[davide moro]

Hi David!

2016-05-11 14:56 GMT+02:00 David Bain <da...@alteroo.com>:

We're doing a project now that will need to support about 1000 users actively logged in (but only for 1 month out of the year).

The application is mostly designed to provide private information so lots of specific permissions, so I'm concerned about doing too much caching.

Our plan at the moment looks like this:

• have a dedicated postgres db
• run multiple kotti client instances (possibly on multiple machines)

You can run multiple kotti client instances and, if needed, using different ini files with different sqlalchemy options depending on what kind of interactions are allowed on that instance in order to tune your performance or if you want you might add different processes, a load balancer, varnish, etc.

For example I've been playing for a while with a private Kotti content management site on one side, a public website for anonymous users and a dedicated Kotti instance for serving image/files (a sort of media.*) with custom Apache/Nginx rewrite rules in order to allow rendering of media files only. Pretty smooth experience.

• benchmark with funkload (because we can simulate logged in users)

You can also have a look at locust.io (sounds interesting: let me know what do you think about that, never tried!).
 

In addition I suggest to adopt newer versions of Kotti and avoid large folders with many items inside: in such case please consider a better balanced content tree or introduce a database index (that might speed up things a lot).

Bye,

davide

I would appreciate comments on my ideas including other things that we should be considering.

--
You received this message because you are subscribed to the Google Groups "Kotti" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kotti+un...@googlegroups.com.
To post to this group, send email to ko...@googlegroups.com.
Visit this group at https://groups.google.com/group/kotti.
To view this discussion on the web visit https://groups.google.com/d/msgid/kotti/3832989e-41df-4705-8bcf-1ba425d4a0ec%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Linkedin: https://it.linkedin.com/in/davidemoro82
Twitter: https://twitter.com/davidemoro
Blog: http://davidemoro.blogspot.it/

[David Bain]

Thanks Davide.... We managed (based on funkload testing) to get to 200 concurrent users with nginx round robin setup and a two kotti client instances.

Will review some of your suggestions.

[Andreas Kaiser]

On 31 May 2016, at 23:43, David Bain wrote:

> Thanks Davide.... We managed (based on funkload testing) to get to 200
> concurrent users with nginx round robin setup and a two kotti client
> instances.

I'd highly recommend using uWSGI with nginx. Let uWSGI do the balancing
(much more efficient than round robin via nginx). Use the uwsgi protocol
and filesystem sockets instead of HTTP sockets. Experiment a bit with
the number of processes and threads (depending on available CPU cores
and RAM).

You need to set "enable-threads = true" to enable threading at all. Most
likely you also want to set "thunder-lock = true". Then try some
combinations of "processes = <int>" and "threads = <int>" and you should
be easily able to serve much more than the desired concurrency on any
decent (virtual) host.


HTH,

Andreas

> On Wednesday, 11 May 2016 07:56:36 UTC-5, David Bain wrote:
>>
>> We're doing a project now that will need to support about 1000 users
>> actively logged in (but only for 1 month out of the year).
>> The application is mostly designed to provide private information so
>> lots
>> of specific permissions, so I'm concerned about doing too much
>> caching.
>>
>> Our plan at the moment looks like this:
>>

>> - have a dedicated postgres db
>> - run multiple kotti client instances (possibly on multiple
>> machines)
>> - benchmark with funkload (because we can simulate logged in

[FFFFFFFab]

avoid large folders with many items inside: in such case please consider a better balanced content tree or introduce a database index (that might speed up things a lot).

Hi, could you explain ? 

[Carmelo Catalano]

Hi FFFFFFFab,

the Kotti content management site on which Davide has been playing has grown to about 6000 items.

Every item (document, image, file etc.) is based on nodes table.

When frontend system renders a page it queries the database to retrieve all page's items.

Because every item is based on nodes table the system query many times the nodes table, so indexing nodes table speed up the system:

ALTER TABLE nodes ADD INDEX(name); ALTER TABLE nodes ADD INDEX(path);

[Andreas Kaiser]

nodes.path has an index by default (unless you're on MySQL, because of its limitations in length). There's also an index on (parent, name), so that another index on name makes little sense, unless your application has lots of where conditions or sorts on name *only*.

Adding indices yourself shouldn't be necessary in most situations.


Andreas

--

Please excuse brevity and typos - sent from a mobile device.

--

You received this message because you are subscribed to the Google Groups "Kotti" group.
To unsubscribe from this group and stop receiving emails from it, send an email to kotti+un...@googlegroups.com.
To post to this group, send email to ko...@googlegroups.com.
Visit this group at https://groups.google.com/group/kotti.

To view this discussion on the web visit https://groups.google.com/d/msgid/kotti/c4f109f0-61cb-4315-83b8-02b3b07850d7%40googlegroups.com.

[FFFFFFFab]

Thanks for your answer. With the default page view, I think the page loading speed is not affected by the number of its children until we explicitly query them. I have a page which has around 17,000 children and I don't see any impact on load speed. But the page loading time increases enormously if I query this page with the "@@content" view that display all children.

[Andreas Kaiser]

On 7 Jun 2016, at 9:18, FFFFFFFab wrote:

> Thanks for your answer. With the default page view, I think the page
> loading speed is not affected by the number of its children until we
> explicitly query them. I have a page which has around 17,000 children
> and I
> don't see any impact on load speed. But the page loading time
> increases
> enormously if I query this page with the "@@content" view that display
> all
> children.

Sure, but in this almost all the time is spent on rendering the
template, not on querying the DB.

Pagination would help here and this is clearly a feature that we want to
have in future versions.


Andreas

[davide moro]

Hi

Il 07/giu/2016 08:49, "Andreas Kaiser" <di...@binary-punks.com> ha scritto:
>
> nodes.path has an index by default (unless you're on MySQL, because of its limitations in length). There's also an index on (parent, name), so that another index on name makes little sense, unless your application has lots of where conditions or sorts on name *only*.

Carmelo: first of all thanks a lot for sharing your experience with Kotti!
Yes, Carmelo is talking about a MySQL based installation. He noticed performance problems with standard @@contents view and a speed up after the indexes introduction, but I don't know if it is only a MySQL related problem. Unfortunately I cannot say more because it wasn't my direct experience.

Cheers,

davide

> To view this discussion on the web visit https://groups.google.com/d/msgid/kotti/5bd45a42-5218-4e50-b4ab-545e6c3852e3%40Andreass-iPhone.