Possible interest in a webcast/presentation about Django site with 40mil+ rows of data??

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Some of you may have noticed, in the last few months I've done quite a few posts/snippets about handling large data sets in Django. At the end of this month (after what seems like a lifetime of trial and error), we're finally going to be releasing a new site which holds around 40mil+ rows of data, grows by about 300-500k rows each day, handles 5GB of uploads per day, and can handle around 1024 requests per second on stress test on a moderately spec'd server.

As the entire thing is written in Django (and a bunch of other open source products), I'd really like to give something back to the community. (stack incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona MySQL/NGINX/supervisord/debian etc)

Therefore, I'd like to see if there would be any interest in webcast in which I would explain how we handle such large amounts of data, the trial and error processes we went through, some really neat tricks we've done to avoid bottlenecks, our own approach to smart content filtering, and some of the valuable lessons we have learned. The webcast would be completely free of charge, last a couple of hours (with a short break) and anyone can attend. I'd also offer up a Q&A session at the end.

If you're interested, please reply on-list so others can see.

Thanks

Cal

[Michał Sawicz]

Dnia 2011-06-22, śro o godzinie 14:15 +0100, Cal Leeming [Simplicity
Media Ltd] pisze:

> If you're interested, please reply on-list so others can see.

Sure, I'd attend.
--
Michał (Saviq) Sawicz <mic...@sawicz.net>

[Thomas Weholt]

Yes! I'm in.

Out of curiosity: When inserting lots of data, how do you do it? Using
the orm? Have you looked at http://pypi.python.org/pypi/dse/2.1.0 ? I
wrote DSE to solve inserting/updating huge sets of data, but if
there's a better way to do it that would be especially interesting to
hear more about ( and sorry for the self promotion ).

Regards,
Thomas

> --
> You received this message because you are subscribed to the Google Groups
> "Django users" group.
> To post to this group, send email to django...@googlegroups.com.
> To unsubscribe from this group, send email to
> django-users...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/django-users?hl=en.
>

--
Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

[Cal Leeming [Simplicity Media Ltd]]

Hey Thomas,

Yeah we actually spoke a little while ago about DSE. In the end, we actually used a custom approach which analyses data in blocks of 50k rows, builds a list of rows which need changing to the same value, then applied them in bulk using update() + F().

Here's our benchmark:

(42.11s) Found 49426 objs (match: 16107) (db writes: 50847) (range: 72300921 ~ 72350921), (avg 13.8 mins/million) - [('is_checked', 49426), ('is_image_blocked', 0), ('has_link', 1420), ('is_spam', 1)]

(44.50s) Found 49481 objs (match: 16448) (db writes: 50764) (range: 72350921 ~ 72400921), (avg 14.6 mins/million) - [('is_checked', 49481), ('is_image_blocked', 0), ('has_link', 1283), ('is_spam', 0)]

(55.78s) Found 49627 objs (match: 18516) (db writes: 50832) (range: 72400921 ~ 72450921), (avg 18.3 mins/million) - [('is_checked', 49627), ('is_image_blocked', 0), ('has_link', 1205), ('is_spam', 0)]

(42.03s) Found 49674 objs (match: 17244) (db writes: 51655) (range: 72450921 ~ 72500921), (avg 13.6 mins/million) - [('is_checked', 49674), ('is_image_blocked', 0), ('has_link', 1971), ('is_spam', 10)]

(51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: 72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

Could you let me know if those benchmarks are better/worse than using DSE? I'd be interested to see the comparison!

Cal

[Shawn Milochik]

Cal,

That sounds awesome. I wish you could present it at DjangoCon US too. :o/

Shawn

[Thomas Weholt]

On Wed, Jun 22, 2011 at 3:36 PM, Cal Leeming [Simplicity Media Ltd]
<cal.l...@simplicitymedialtd.co.uk> wrote:
> Hey Thomas,
> Yeah we actually spoke a little while ago about DSE. In the end, we actually
> used a custom approach which analyses data in blocks of 50k rows, builds a
> list of rows which need changing to the same value, then applied them in
> bulk using update() + F().

Hmmm, what do you mean by "bulk using update() + F()? Something like
"update sometable set somefield1 = somevalue1, somefield2 = somevalue2
where id in (1,2,3 .....)" ? Does "avg 13.8 mins/million" mean you
processed 13.8 million rows pr minute? What kind of hardware did you
use?

Thomas

[Cal Leeming [Simplicity Media Ltd]]

Sorry, let me explain a little better.

(51.98s) Found 49659 objs (match: 16563) (db writes: 51180) (range: 72500921 ~ 72550921), (avg 16.9 mins/million) - [('is_checked', 49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

map(lambda x: (x[0], len(x[1])), _obj_incs.iteritems()) = [('is_checked', 49659), ('is_image_blocked', 0), ('has_link', 1517), ('is_spam', 4)]

In the above example, it has found 49659 rows which need 'is_checked' changing to the value '1' (same principle applied to the other 3), giving a total of 51,130 database writes, split into 4 queries.

Those 4 fields have the IDs assigned to them:

                                    if _f == 'block_images':

                                        _obj_incs.get('is_image_blocked').append(_hit_id)

                                        if _parent_id:

                                            _obj_incs.get('is_image_blocked').append(_parent_id)

Then I loop through those fields, and do an update() using the necessary IDs:

                    # now apply the obj changes in bulk (massive speed improvements)

                    for _key, _value in _obj_incs.iteritems():

                        # update the child object

                        Post.objects.filter(

                            id__in = _value

                        ).update(

                            **{

                                _key : 1

                            }

                        )

So in simple terms, we're not doing 51 thousand update queries, instead we're grouping them into bulk queries based on the row to be updated. It doesn't yet to grouping based on key AND value, simply because we didn't need it at the time, but if we release the code for public use, we'd definitely add this in.

Hope this makes sense, let me know if I didn't explain it very well lol.

Cal

[Cal Leeming [Simplicity Media Ltd]]

Also, the 13.8 minutes per million, is basically a benchmark based on the amount of db writes, and the total amount of time it took to execute (which was 51s).

Please also note, this code is doing a *heavy* amount of content analysis, but if you were to strip that out, the only overheads would be the map/filter/lambda, the time it takes to transmit to MySQL, and the time it takes for MySQL to perform the writes.

The database hardware spec is:

1x X3440 quad core (2 cores assigned to MySQL).

12GB memory (4 GB assigned to MySQL).

/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.

Cal

[Thomas Weholt]

Actually, I started working on something similar, but tried to find
sets of fields, instead of just updating one field pr update, but
didn't finish it because the actual grouping of the fields seem to
take alot of time/cpu/memory. Perhaps if I focused on updating one
field at the time it would be simpler. Might look at it again for DSE
3.0 ;-)

Thomas

[Andre Terra]

Hello, Cal

First of all, congrats on the newborn! The Django community will surely benefit from having yet another success story, especially considering how big this project sounds. Is there any chance you could open-source some of your custom made improvements so that they could eventually be merged to trunk?

I definitely noticed how you mentioned large dbs in the past few months. I, along with many others I assume, would surely like to attend the webcast, with the only impediment being my schedule/timezone.

I recently asked about working with temporary tables for filtering/grouping data from uploads and inserting queries from that temporary table onto a permanent database. To make matters worse, I wanted to make this as flexible as possible (i.e. dynamic models) so that everything could be managed from a web app. Do you have any experience you could share about any of these use cases? As far as I know, there's nothing in the ORM that replicates PostgreSQL's CREATE TEMPORARY TABLE. My experience with SQL is rather limited, but from asking around, it seems like my project could indeed benefit from such a feature. If I had to guess, I would assume other DBMSs would offer something similar, but being limited to Postgres is okay for me, for now, anyway.



Cheers,
André

[Cal Leeming [Simplicity Media Ltd]]

Hmm, that's odd, the grouping (map/reduce/filter/lambda) is extremely quick for me (even on a heavy data set).

My guess is that grouping would need to be done on a combination of field name+value, and would need to allow the user to specify what bulk to use (to prevent MemoryError exception - or find some way to reduce the bulk when MemoryError is encountered).

If you end up introducing it into 3.0, I'll definitely be interested in taking a look at the code :)

Cal

--

[Cal Leeming [Simplicity Media Ltd]]

On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra <andre...@gmail.com> wrote:

Hello, Cal

First of all, congrats on the newborn! The Django community will surely benefit from having yet another success story, especially considering how big this project sounds. Is there any chance you could open-source some of your custom made improvements so that they could eventually be merged to trunk?

Thank you! Yeah, the plan is to release as much of the improvements as open source as possible. Although I'd rely heavily on the community to make them 'patch worthy' for the core, as the amount of spare time I have is somewhat limited. 

The improvements list is growing by the day, and I usually try and post as many snippets as I can, and/or tickets etc. 

It sounds like Thomas's DSE might be the perfect place for the bulk update code too.

 

I definitely noticed how you mentioned large dbs in the past few months. I, along with many others I assume, would surely like to attend the webcast, with the only impediment being my schedule/timezone.

Once we've got a list of all the people who want to attend, I'll send out a mail asking for everyones timezone and availability, so we can figure out what is best for everyone.

 

I recently asked about working with temporary tables for filtering/grouping data from uploads and inserting queries from that temporary table onto a permanent database. To make matters worse, I wanted to make this as flexible as possible (i.e. dynamic models) so that everything could be managed from a web app. Do you have any experience you could share about any of these use cases? As far as I know, there's nothing in the ORM that replicates PostgreSQL's CREATE TEMPORARY TABLE. My experience with SQL is rather limited, but from asking around, it seems like my project could indeed benefit from such a feature. If I had to guess, I would assume other DBMSs would offer something similar, but being limited to Postgres is okay for me, for now, anyway.

I haven't had any exposure to Postgres, but my experience with temporary tables hasn't been a nice one (in regards to MySQL at least). MySQL has many gotchas when it comes to temporary tables and indexing, and on more than one occasion, I found it was actually quicker to analyse/mangle/re-insert the data via Python code, than it was to attempt the modifications within MySQL using a temporary table.

It really does depend on what your data is, and what you want to do with it, which can make planning ahead somewhat tedious lol.

For our stuff, when we need to do bulk modifications, we have a filtering rules list which is ran every hour against new rows (with is_checked=1 set on rows which have been checked). We then use bulk queries of 50k (id >= 0 AND id < 50000), rather than using LIMIT/OFFSET (because LIMIT/OFFSET gets slower and slower the larger the result set). Those queries are analysed/mangled within a transaction, and bulk updated using the method mentioned in the reply to Thomas.

Sadly though, I can't say if the methods we use would be suitable for you, as we haven't tried it against Postgres, and we've only tested it against our own data set + requirements. This is what I mean by trial and error, it's a pain in the ass :)

[Andre Terra]

On Wed, Jun 22, 2011 at 11:47 AM, Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk> wrote:

On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra <andre...@gmail.com> wrote:

Hello, Cal

First of all, congrats on the newborn! The Django community will surely benefit from having yet another success story, especially considering how big this project sounds. Is there any chance you could open-source some of your custom made improvements so that they could eventually be merged to trunk?

Thank you! Yeah, the plan is to release as much of the improvements as open source as possible. Although I'd rely heavily on the community to make them 'patch worthy' for the core, as the amount of spare time I have is somewhat limited. 

The improvements list is growing by the day, and I usually try and post as many snippets as I can, and/or tickets etc. 

It sounds like Thomas's DSE might be the perfect place for the bulk update code too.

Thanks a lot for the quick reply. I'll keep my eyes open for the code, and if unable to contribute with relevant modifications to the patches, I'll at least try to doc and test them!
 
 

I definitely noticed how you mentioned large dbs in the past few months. I, along with many others I assume, would surely like to attend the webcast, with the only impediment being my schedule/timezone.

Once we've got a list of all the people who want to attend, I'll send out a mail asking for everyones timezone and availability, so we can figure out what is best for everyone.

Definitely write me up for the list of attendees, then!
 
 

I recently asked about working with temporary tables for filtering/grouping data from uploads and inserting queries from that temporary table onto a permanent database. To make matters worse, I wanted to make this as flexible as possible (i.e. dynamic models) so that everything could be managed from a web app. Do you have any experience you could share about any of these use cases? As far as I know, there's nothing in the ORM that replicates PostgreSQL's CREATE TEMPORARY TABLE. My experience with SQL is rather limited, but from asking around, it seems like my project could indeed benefit from such a feature. If I had to guess, I would assume other DBMSs would offer something similar, but being limited to Postgres is okay for me, for now, anyway.

I haven't had any exposure to Postgres, but my experience with temporary tables hasn't been a nice one (in regards to MySQL at least). MySQL has many gotchas when it comes to temporary tables and indexing, and on more than one occasion, I found it was actually quicker to analyse/mangle/re-insert the data via Python code, than it was to attempt the modifications within MySQL using a temporary table.

It really does depend on what your data is, and what you want to do with it, which can make planning ahead somewhat tedious lol.

For our stuff, when we need to do bulk modifications, we have a filtering rules list which is ran every hour against new rows (with is_checked=1 set on rows which have been checked). We then use bulk queries of 50k (id >= 0 AND id < 50000), rather than using LIMIT/OFFSET (because LIMIT/OFFSET gets slower and slower the larger the result set). Those queries are analysed/mangled within a transaction, and bulk updated using the method mentioned in the reply to Thomas.

Sadly though, I can't say if the methods we use would be suitable for you, as we haven't tried it against Postgres, and we've only tested it against our own data set + requirements. This is what I mean by trial and error, it's a pain in the ass :)

Thanks again for your enlightening input. Even with our different requirements, this was actually quite relevant as far as solving several doubts I had on how to go about this project.


Cheers,

André

On Wed, Jun 22, 2011 at 10:56 AM, Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk> wrote:

Also, the 13.8 minutes per million, is basically a benchmark based on the amount of db writes, and the total amount of time it took to execute (which was 51s).

Please also note, this code is doing a *heavy* amount of content analysis, but if you were to strip that out, the only overheads would be the map/filter/lambda, the time it takes to transmit to MySQL, and the time it takes for MySQL to perform the writes.

The database hardware spec is:

1x X3440 quad core (2 cores assigned to MySQL).

12GB memory (4 GB assigned to MySQL).

/var/lib/mysql mapped to 2x Intel M3 SSD drives in RAID 1.

Cal

On Wed, Jun 22, 2011 at 2:52 PM, Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk> wrote:

Sorry, let me explain a little better.

(...)

[Ivan Aleman]

On 22 June 2011 08:15, Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk> wrote:

If you're interested, please reply on-list so others can see.

Sweet! Count me in :)

 

--
Iván

[Anurag Chourasia]

I am in :-)

--

[creecode]

Hello SleepyCal,

On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote:

If you're interested, please reply on-list so others can see.

+1

Also if the webcast could be stored for later viewing that would be grand.

Toodle-loooooooooooooo..........
creecode

[Cal Leeming [Simplicity Media Ltd]]

On Wed, Jun 22, 2011 at 4:55 PM, creecode <cree...@gmail.com> wrote:

Hello SleepyCal,

On Wednesday, June 22, 2011 6:15:48 AM UTC-7, SleepyCal wrote:

If you're interested, please reply on-list so others can see.

+1

Also if the webcast could be stored for later viewing that would be grand.

Yup, I'm planning on recording in 1080p and posting on Youtube shortly afterwards.

 

Toodle-loooooooooooooo..........
creecode

--
You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/SZabiWnq_S0J.

[Brian Bouterse]

+1 for making it viewable after the fact

--

You received this message because you are subscribed to the Google Groups "Django users" group.

To view this discussion on the web visit https://groups.google.com/d/msg/django-users/-/SZabiWnq_S0J.

To post to this group, send email to django...@googlegroups.com.
To unsubscribe from this group, send email to django-users...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Brian Bouterse
ITng Services

[creecode]

Hello SleepyCal,

On Wednesday, June 22, 2011 9:00:15 AM UTC-7, SleepyCal wrote:

Yup, I'm planning on recording in 1080p and posting on Youtube shortly afterwards.

Fantastic!

Thank you,

Toodle-loooo...........
creecode

[Malcolm Box]

On 22 June 2011 14:15, Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk> wrote:

Hi all,

Therefore, I'd like to see if there would be any interest in webcast in which I would explain how we handle such large amounts of data, the trial and error processes we went through, some really neat tricks we've done to avoid bottlenecks, our own approach to smart content filtering, and some of the valuable lessons we have learned. The webcast would be completely free of charge, last a couple of hours (with a short break) and anyone can attend. I'd also offer up a Q&A session at the end.

If you're interested, please reply on-list so others can see.

Count me in.

Malcolm

[serek]

Great idea.
I also would like to see this webcast

On Jun 22, 6:00 pm, "Cal Leeming [Simplicity Media Ltd]"
<cal.leem...@simplicitymedialtd.co.uk> wrote:

[Mishen'ka]

I really like the idea, hope i can see it soon.

Interessting thanks

On 22 jun, 15:15, "Cal Leeming [Simplicity Media Ltd]"

[Thomas Weholt]

On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]
<cal.l...@simplicitymedialtd.co.uk> wrote:
>
>
> On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra <andre...@gmail.com> wrote:
>>
>> Hello, Cal
>>
>> First of all, congrats on the newborn! The Django community will surely
>> benefit from having yet another success story, especially considering how
>> big this project sounds. Is there any chance you could open-source some of
>> your custom made improvements so that they could eventually be merged to
>> trunk?
>
> Thank you! Yeah, the plan is to release as much of the improvements as open
> source as possible. Although I'd rely heavily on the community to make them
> 'patch worthy' for the core, as the amount of spare time I have is somewhat
> limited.
> The improvements list is growing by the day, and I usually try and post as
> many snippets as I can, and/or tickets etc.
> It sounds like Thomas's DSE might be the perfect place for the bulk update
> code too.

FYI: Inspired by this discussion I've allready started on a similar
feature ( allthough somewhat simplified ) for DSE v2.2.0 and you're
right; the speed increase is huge using the method described here,
even compared to my current solution ( using cursor.executemany ),
which is considerably faster than the django orm allready. My testing
so far have been using postgresql, not sure how mysql will perform. I
expect to release DSE v.2.2.0 with this feature in the next few days.

[Cal Leeming [Simplicity Media Ltd]]

Nice, seems like there is a lot of positive feedback. 

Here's some further info about the webcast/webinar:

Expected Date: July/August

Expected Time: Somewhere between 9AM - 9PM GMT+0 (UK Time), I'll probably run a webpage vote so you guys can decide.

Via: GoToWebinar - screen share with voip.

Length: 1 hour max for actual presentation, 5 minute break, then another 1 hour max for Q&As.

Presentation Size: 1920x1080 (1080p)

There will be some basic slides (to keep things structured), but the majority of the time will be spent inside the browser + IDE + shell etc.

Cal

--
You received this message because you are subscribed to the Google Groups "Django users" group.

[Cal Leeming [Simplicity Media Ltd]]

Nice!

Nice! Looking forward to seeing this :)

 

--

Mvh/Best regards,
Thomas Weholt
http://www.weholt.org

[william ratcliff]

Definitely looking forward to it!

[Cal Leeming [Simplicity Media Ltd]]

I was thinking.. what would be really nice would be a monkey patch that could catch any inserts/updates/saves from the ORM, and defer them until the user triggered the bulk update some how.. That way, DSE would be completely interchangeable with existing code, making it much more likely for people to adopt it. 

[Władysław Mettler]

Definitely interested. Struggling with scaling a Django/Celery/Postgres/CouchDB/Memcach project now. Would love to see your approach.

Cheers
Vlad

[higs]

> If you're interested, please reply on-list so others can see.
>

would love to see this. thanks for posting.

[Derek]

Carl

Please post here when the recording becomes available... some of us do
not have access to HD-capable bandwidth yet!

Thanks
Derek

[Cal Leeming [Simplicity Media Ltd]]

Hi Derek,

I've had a look and apparently 4-5mbit should be sufficient enough for a HD webinar. If quite a few of you don't have this capability (or screen size), I could look into reducing it down to 1440x900 perhaps? 

Let me know your thoughts guys. (maybe it's something I need to do a web vote for, along with the timezone etc)

Cal

[Chris Calitz]

Sounds really cool. I'm definitely in. 

On 22 Jun 2011, at 14:16, "Cal Leeming [Simplicity Media Ltd]" <cal.l...@simplicitymedialtd.co.uk> wrote:

Hi all,

Some of you may have noticed, in the last few months I've done quite a few posts/snippets about handling large data sets in Django. At the end of this month (after what seems like a lifetime of trial and error), we're finally going to be releasing a new site which holds around 40mil+ rows of data, grows by about 300-500k rows each day, handles 5GB of uploads per day, and can handle around 1024 requests per second on stress test on a moderately spec'd server.

As the entire thing is written in Django (and a bunch of other open source products), I'd really like to give something back to the community. (stack incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona MySQL/NGINX/supervisord/debian etc)

Therefore, I'd like to see if there would be any interest in webcast in which I would explain how we handle such large amounts of data, the trial and error processes we went through, some really neat tricks we've done to avoid bottlenecks, our own approach to smart content filtering, and some of the valuable lessons we have learned. The webcast would be completely free of charge, last a couple of hours (with a short break) and anyone can attend. I'd also offer up a Q&A session at the end.

If you're interested, please reply on-list so others can see.

Thanks

Cal

[John DeRosa]

Me

[Steve Holden]

And, as luck would have it, the US Call For Papers was just published:

  http://djangocon.us/blog/2011/06/22/call-papers/

regards

 Steve

On Wed, Jun 22, 2011 at 9:37 AM, Shawn Milochik <sh...@milochik.com> wrote:

Cal,

That sounds awesome. I wish you could present it at DjangoCon US too. :o/

Shawn

--
You received this message because you are subscribed to the Google Groups "Django users" group.
To post to this group, send email to django...@googlegroups.com.

To unsubscribe from this group, send email to django-users+unsubscribe@googlegroups.com.

For more options, visit this group at http://groups.google.com/group/django-users?hl=en.

--
Steve Holden        +1 571 484 6266  +1 800 494 3119
Holden Web LLC             http://www.holdenweb.com/

[Cal Leeming [Simplicity Media Ltd]]

(sorry, should have sent this reply on-list)

You know, I was actually thinking about that, but the flights would cost way too much (like $1400+) ;/ If the project has interest next year, I'll probably ask about presenting at DjangoCon EU though.

To unsubscribe from this group, send email to django-users...@googlegroups.com.

[Jason]

+1 Very interested.

On Jun 22, 3:47 pm, "Cal Leeming [Simplicity Media Ltd]"

<cal.leem...@simplicitymedialtd.co.uk> wrote:
> Nice, seems like there is a lot of positive feedback.
>
> Here's some further info about the webcast/webinar:
>

> *Expected Date:* July/August
> *Expected Time:* Somewhere between 9AM - 9PM GMT+0 (UK Time), I'll probably

> run a webpage vote so you guys can decide.

> *Via:* GoToWebinar - screen share with voip.
> *Length:* 1 hour max for actual presentation, 5 minute break, then another 1
> hour max for Q&As.
> *Presentation Size: *1920x1080 (1080p)

>
> There will be some basic slides (to keep things structured), but the
> majority of the time will be spent inside the browser + IDE + shell etc.
>
> Cal
>

[Fatrix]

I would like to attend, too.

Thank you in advance!

CU
Fatrix

[Alex Grigorievskiy]

Hi, Cal and everybody,
I'd like to see the webcast too,
thanks very much for that :)

On Jun 23, 2:50 am, "Cal Leeming [Simplicity Media Ltd]"
<cal.leem...@simplicitymedialtd.co.uk> wrote:
> Nice!

>
> On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt <thomas.weh...@gmail.com>wrote:
>
>
>
>
>
>
>
>
>
> > On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

> > <cal.leem...@simplicitymedialtd.co.uk> wrote:
>
> > > On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra <andrete...@gmail.com>

[bruno desthuilliers]

+1

[Nolan Brubaker]

Yes, very interested.  +1

On Fri, Jun 24, 2011 at 9:04 AM, bruno desthuilliers <bruno.des...@gmail.com> wrote:

+1

[Cal Leeming [Simplicity Media Ltd]]

Really glad to see there has been so much interest in this!

If possible, can everyone please use the following form to indicate what week day / timeslot they are available, along with their screen resolution. This will be open until 1st July.

https://spreadsheets0.google.com/spreadsheet/viewform?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ

Whichever week day / timeslot / screen resolution has 100% of the votes, will be the chosen one :) If it's split, we'll look at doing the webcast on two separate occasions. Hopefully this should give everyone a fair chance.

Thanks

Cal

[Cal Leeming [Simplicity Media Ltd]]

Second call for anyone who wants to attend this webcast, 3 days left to place your vote.

Cal

[Andre Terra]

Can't access google spreadsheets through my corporate proxy (go figure), but I will vote later today.

Really interested in watching this!


Andre

[Cal Leeming [Simplicity Media Ltd]]

Wow, that sucks! :X Never known a corp to block google docs

[Matteius]

If I am unable to attend at the scheduled time I absolutely must watch
the resulting recording. Thanks for your efforts.

On Jun 22, 5:50 pm, "Cal Leeming [Simplicity Media Ltd]"
<cal.leem...@simplicitymedialtd.co.uk> wrote:
> Nice!

>
> On Wed, Jun 22, 2011 at 11:39 PM, Thomas Weholt <thomas.weh...@gmail.com>wrote:
>
>
>
>
>
> > On Wed, Jun 22, 2011 at 4:47 PM, Cal Leeming [Simplicity Media Ltd]

> > <cal.leem...@simplicitymedialtd.co.uk> wrote:
>
> > > On Wed, Jun 22, 2011 at 3:25 PM, Andre Terra <andrete...@gmail.com>

[Wagner Vaz]

Please, count me in.

On Jun 22, 2011 10:15 AM, "Cal Leeming [Simplicity Media Ltd]" <cal.l...@simplicitymedialtd.co.uk> wrote:
> Hi all,
>
> Some of you may have noticed, in the last few months I've done quite a few
> posts/snippets about handling large data sets in Django. At the end of this
> month (after what seems like a lifetime of trial and error), we're finally
> going to be releasing a new site which holds around 40mil+ rows of data,
> grows by about 300-500k rows each day, handles 5GB of uploads per day, and
> can handle around 1024 requests per second on stress test on a moderately
> spec'd server.
>
> As the entire thing is written in Django (and a bunch of other open source
> products), I'd really like to give something back to the community. (stack
> incls Celery/RabbitMQ/Sphinx SE/PYQuery/Percona
> MySQL/NGINX/supervisord/debian etc)
>
> Therefore, I'd like to see if there would be any interest in webcast in
> which I would explain how we handle such large amounts of data, the trial
> and error processes we went through, some really neat tricks we've done to
> avoid bottlenecks, our own approach to smart content filtering, and some of
> the valuable lessons we have learned. The webcast would be completely free
> of charge, last a couple of hours (with a short break) and anyone can
> attend. I'd also offer up a Q&A session at the end.
>
> If you're interested, please reply on-list so others can see.
>
> Thanks
>
> Cal
>

[Cal Leeming [Simplicity Media Ltd]]

Hey all,

Last call for registering your interest in this webcast (28 votes so far).

If you haven't done so already, please visit the following URL:

https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0

Poll ends tomorrow, at which point we will decide upon a fixed time/date(s).

Cal

[Benedict Verheyen]

> Hey all,
>
> Last call for registering your interest in this webcast (28 votes so far).
>
> If you haven't done so already, please visit the following URL:
>
> https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0
> <https://spreadsheets.google.com/spreadsheet/viewform?hl=en_GB&formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ#gid=0>
>
> Poll ends tomorrow, at which point we will decide upon a fixed time/date(s).
>

I'm interested, visited the URL above already.
Time isn't going to be that big of an issue for me as I'm only 1 timezone away :)

Cheers,
Benedict

[Cal Leeming [Simplicity Media Ltd]]

Hi guys,

Alright, vote is closed and here are the results:

https://spreadsheets.google.com/spreadsheet/gform?key=0AiciLOVZqDWqdENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c&hl=en_GB&gridId=0#chart

It looks like I might have to do two separate webcasts to cater for all timezones, but we'll see.

I'll post up a fixed time/date (or two) shortly.

Cal

[Cal Leeming [Simplicity Media Ltd]]

Oops, sorry guys, I pasted the wrong link.

Here you go:

https://spreadsheets0.google.com/spreadsheet/viewanalytics?formkey=dENyOVFSSkhSYnhBLVZGTktiN1Z3Y2c6MQ

[gradja]

> If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella

[Andre Santos]

Maybe someone can record it and upload somewhere for the ones who cant be online when you do the presentation...

2011/7/3 gradja <graztou...@yahoo.fr>

> If you're interested, please reply on-list so others can see.

Count me in, and thanks to share your experience with others.
Graziella

[Cal Leeming [Simplicity Media Ltd]]

This has already been discussed earlier in the thread :)

[Andre Santos]

Oh, sorry :).

2011/7/3 Cal Leeming [Simplicity Media Ltd] <cal.l...@simplicitymedialtd.co.uk>

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Great response to this, 45 registered votes in total.

The webcast will take place on Monday 29th August 2011 - (Minimum resolution 1920x1080)

However, because the time zone results are almost split 50/50, I'm going to allow users to select from two time slots, if the results are still split 50/50, then I'll do both an afternoon and evening session.

Please use this form to register your place.

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ

Cal

[Venkatraman S]

Can you please ..please...please..please record this session!?

-V

[Cal Leeming [Simplicity Media Ltd]]

Of course :) I'll make sure to take a high quality recording and stick it on YT.

Cal

On Tue, Jul 12, 2011 at 6:31 PM, Venkatraman S <venk...@gmail.com> wrote:

Can you please ..please...please..please record this session!?

-V

--

[AlexH]

> If you're interested, please reply on-list so others can see.

Would be very interested in attending. Thanks!

[Andre Terra]

Unfortunately, I'm on GMT-0300 and none of those timezones work for me. Looking forward to the recorded version, though!

Cheers,

André

--

[Cal Leeming [Simplicity Media Ltd]]

Second call for register your attendance on this webcast, if you haven't already done so.

Thanks

Cal

[nicolas HERSOG]

I won t be able to follow the live session but I can't wait to watch your record on YT.

Thx!

[Cal Leeming [Simplicity Media Ltd]]

Last call on this guys:

https://spreadsheets.google.com/a/foxwhisper.co.uk/spreadsheet/viewform?hl=en_US&formkey=dENPX3BMUUFMbi1CbElwV3BvOEdkNmc6MQ#gid=0

If you want to register your attendance, please do so now.

Thanks

[Cal Leeming [Simplicity Media Ltd]]

Hi all,

Okay, good news and bad news. 

The bad news, is that the site/project which sparked this webcast to be made, has had some legal complications and is being shut down - less than 1 month after being released :L

The good news, is that the webcast will still be going ahead.

For those of you that want to see the site it was related to, it's http://www.iliketochan.com . 

The site itself was a complete 4chan archive, dating back to 2010, peaking at 50 million posts and 17 million images.

Particularly sad about this because so much time and effort went into the site, but at the same time, it has been a great learning experience (both in code and business). So being able to share that on the webcast will be of some small comfort :)

Cal

[Cal Leeming [Simplicity Media Ltd]]

Hey guys,

I'm sending out the webcast invitations now (should receive them in about 10 minutes from GoToMeeting). Last chance to jump on if you haven't already.

Cal

[william ratcliff]

Any problem if we register for both?

[Cal Leeming [Simplicity Media Ltd]]

Nope, we're allowed up to 100 attendees per time slot, and we are at 55 at the moment, so feel free to register for both.

[Cal Leeming [Simplicity Media Ltd]]

Bloody thing, had to send it directly to the list cos it doesn't have the option of bulk inviting people. How annoying :X

Anyone who didn't receive the invite email, please let me know.

Cal