DAL size limit for query?

[goome]

Hello
is there any size limit in DAL about query?
i explain:
i have a script that stop running.
I thought it was a problem with modwsgi becouse i got the error message 'truncatet or oversize respoinse header' (so that i post this question : https://groups.google.com/forum/#!topic/web2py/CRsEWe_mmSc)
but now i realized the problem raise even in the web2py shell, exactly when querying:
This is from my pc:
"""
tabella = 'STK'
In [3]:  query = legacy_db[tabella]
 legacy_db(query).select()

Out[5]: <Rows (32144)>
""

this from the server, where the data in the db is are more:
"""In [1]: tabella = 'STK'

In [2]: query =legacy_db[tabella]
In [4]: legacy_db(query).select()

Killed
root@rb:/home/www-data/web2py#
"""
i still can query through 'executesql' :

n [1]: rows = legacy_db.executesql("select * from STK")

In [2]: len(rows)
Out[2]: 155370

The insertion in the table, by means of DAL and a cronjob, still is working

Any help?
Thanks

[Niphlod]

a) step out apache as soon as you can

b) what if you do it in a shell ?

c) the only limit is the available memory

d) if you hit that limit, and you don't need update_record() and delete_record(), you can save some with select(...cacheable=True)

e) even more can be saved with executesql(as_dict=True)

[Marcello Console]

when trying in the shell, the shell itself got killed :

>> In [4]: legacy_db(query).select()
>>
>> Killed
>> root@rb:/home/www-data/web2py#

Even using executesql
(but i can query sqlite directly)
"step out apache": Does this mean stopping apache?

Thank you a lot for your help

> --
> Resources:
> - http://web2py.com
> - http://web2py.com/book (Documentation)
> - http://github.com/web2py/web2py (Source code)
> - https://code.google.com/p/web2py/issues/list (Report Issues)
> ---
> You received this message because you are subscribed to a topic in the
> Google Groups "web2py-users" group.
> To unsubscribe from this topic, visit
> https://groups.google.com/d/topic/web2py/nqUnVx62GE0/unsubscribe.
> To unsubscribe from this group and all its topics, send an email to
> web2py+un...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Dave S]

On Wednesday, August 3, 2016 at 2:33:35 PM UTC-7, goome wrote:

when trying in the shell, the shell itself got killed :
>> In [4]: legacy_db(query).select()
>>
>> Killed
>> root@rb:/home/www-data/web2py#

Is that using web2py in a bash/python shell  (i.,e., -M -S myapp) ?
Or in the web shell (admin app feature)?   The latter is deprecated.

Even using executesql
(but i can query sqlite directly)
"step out apache": Does this mean stopping apache?

Probably just using the shell to avoid having apache doing things like killing the thread.
 

Thank you a lot for your help

If you are using -S myapp, do you have top (on linux) or taskmgr (on windders) monitoring the system state?

/dps

 

[Marcello Console]

2016-08-04 1:08 GMT+02:00, Dave S <snide...@gmail.com>:
>
>
> On Wednesday, August 3, 2016 at 2:33:35 PM UTC-7, goome wrote:
>>
>> when trying in the shell, the shell itself got killed :
>> >> In [4]: legacy_db(query).select()
>> >>
>> >> Killed
>> >> root@rb:/home/www-data/web2py#
>>
>>
> Is that using web2py in a bash/python shell (i.,e., -M -S myapp) ?

yes, this one

[c...@cemeren.com]

You may try using iterselect() instead of select() if the issue is memory related.

The link to the book is below.

http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer?search=Using-an-iterator-based-select-for-lower-memory-use#Using-an-iterator-based-select-for-lower-memory-use

[Marcello Console]

thank you for your answer and sorry for the delay, i had been some days away.

at this time the code that fails is:
tabella = request.args[0]
response.headers['Content-Type']='text/x-csv'
response.headers['Content-Disposition']='attachment;filename="report.csv" '
dati = legacy_db().select(legacy_db[tabella].ALL)
s = cStringIO.StringIO()
dati.export_to_csv_file(s)


so i should use iterselect when defining dati, but when i try something like:
"""
for row in legacy_db(legacy_db.COD2.id > 0).iterselect():
....: rtn = row
....:
""
i get :
""
---------------------------------------------------------------------------
AttributeError Traceback (most recent call last)
/home/www-data/web2py/applications/PROVE/models/menu.py in <module>()
----> 1 for row in legacy_db(legacy_db.COD2.id > 0).iterselect():
2 rtn = row
3

AttributeError: 'Set' object has no attribute 'iterselect'

""
moreover, i should 'append' the row to a pydal.objects.Rows
as to say
dati += row
but how can i do this?

Thank you a lot again

2016-08-04 9:52 GMT+02:00, c...@cemeren.com <c...@cemeren.com>:
> You may try using iterselect() instead of select() if the issue is memory
> related.
>
> The link to the book is below.
>
> http://web2py.com/books/default/chapter/29/06/the-database-abstraction-layer?search=Using-an-iterator-based-select-for-lower-memory-use#Using-an-iterator-based-select-for-lower-memory-use
>
> On Thursday, August 4, 2016 at 10:35:13 AM UTC+3, goome wrote:
>>

>> 2016-08-04 1:08 GMT+02:00, Dave S <snide...@gmail.com <javascript:>>:

>> >
>> >
>> > On Wednesday, August 3, 2016 at 2:33:35 PM UTC-7, goome wrote:
>> >>
>> >> when trying in the shell, the shell itself got killed :
>> >> >> In [4]: legacy_db(query).select()
>> >> >>
>> >> >> Killed
>> >> >> root@rb:/home/www-data/web2py#
>> >>
>> >>
>> > Is that using web2py in a bash/python shell (i.,e., -M -S myapp) ?
>> yes, this one
>>
>

[Dave S]

On Saturday, August 27, 2016 at 3:29:58 AM UTC-7, goome wrote:

thank you for your answer and sorry for the delay, i had been some days away.

at this time the code that fails is:
tabella = request.args[0]
response.headers['Content-Type']='text/x-csv'
response.headers['Content-Disposition']='attachment;filename="report.csv" '
dati = legacy_db().select(legacy_db[tabella].ALL)
s = cStringIO.StringIO()
dati.export_to_csv_file(s)


so i should use iterselect when defining dati, but when i try something like:
"""
for row in legacy_db(legacy_db.COD2.id > 0).iterselect():
   ....:     rtn = row
   ....:
""
i get :
""
---------------------------------------------------------------------------
AttributeError                            Traceback (most recent call last)
/home/www-data/web2py/applications/PROVE/models/menu.py in <module>()
----> 1 for row in legacy_db(legacy_db.COD2.id > 0).iterselect():
      2     rtn = row
      3

AttributeError: 'Set' object has no attribute 'iterselect'

""

You may need to show us your define_table statement(s), but it looks like you're writing down a query object and asking for iterselect on it, as in the book example.  One of the DB gurus may be needed.
 

moreover, i should 'append' the row to a pydal.objects.Rows
as to say
dati += row
but how can i do this?

That one looks easy:  data.append(row)
 

Thank you a lot again

Good luck!

/dps
 

[Marcello Console]

>
>> moreover, i should 'append' the row to a pydal.objects.Rows
>> as to say
>> dati += row
>> but how can i do this?
>>
>>
> That one looks easy: data.append(row)

but how can i create first 'data' as a pydal.objects.Rows?

>
>
>> Thank you a lot again
>>
>>
> Good luck!
>
> /dps
>
>
>>

>> 2016-08-04 9:52 GMT+02:00, c...@cemeren.com <javascript:> <
>> c...@cemeren.com <javascript:>>:

>> > web2py+un...@googlegroups.com <javascript:>.

[Dave S]

On Monday, August 29, 2016 at 1:25:33 PM UTC-7, goome wrote:

>
>> moreover, i should 'append' the row to a pydal.objects.Rows
>> as to say
>> dati += row
>> but how can i do this?
>>
>>
> That one looks easy:  data.append(row)

but how can i create first 'data' as a  pydal.objects.Rows?

Well, that's the answer to your first question, which I can't provide.  It may help other helpers though if you show your define_table() statements.

/dps