shove does not store data as expected

[Alex]

Dear all,
I'm trying to use the shove module (http://pypi.python.org/pypi/shove)
for a simple script. The script read a CSV file ad store the data.
When I check the content of the "store" object (instance of Shove)
*before* I close it, the data are all there but when I close and re-
open it some data are lost. How it is possible? There is something
wrong in my code or I didn't understand how shove works?

Thanks in advance.

Here is a sample of my code:

DBNAME = "file://store.db"

csv_file ="test.csv"
cities = csv.reader(open(csv_file), delimiter=";")
store = Shove(DBNAME, compress=True)
for city,region,code in cities:
entry_list = store.setdefault(region, [])
data = 'just a sample'
entry = {city:data}
entry_list.append(entry)

print "----Before closing----"
for k,v in store.iteritems():
print "Key->", k
print "Value->", v
print "----"
store.close()

store = Shove(DBNAME, compress=True)
print "----After closing and re-opening----"
for k,v in store.iteritems():
print "Key->", k
print "Value->", v
print "----"

Here is the output

----Before closing----
Key-> Marche
Value-> [{'Ancona': 'just a sample'}, {'Ascoli Piceno': 'just a
sample'}]
Key-> Piemonte
Value-> [{'Alessandria': 'just a sample'}, {'Asti': 'just a sample'}]
Key-> Puglia
Value-> [{'Bari': 'just a sample'}, {'Barletta-Andria-Trani': 'just a
sample'}]
Key-> Valle d'Aosta
Value-> [{'Aosta': 'just a sample'}]
----
----After closing and re-opening----
Key-> Marche
Value-> [{'Ancona': 'just a sample'}]
Key-> Piemonte
Value-> [{'Alessandria': 'just a sample'}]
Key-> Puglia
Value-> [{'Bari': 'just a sample'}, {'Barletta-Andria-Trani': 'just a
sample'}]
Key-> Valle d'Aosta
Value-> [{'Aosta': 'just a sample'}]
----

Here is the content of the CSV file:

Alessandria;Piemonte;AL
Ancona;Marche;AN
Aosta;Valle d'Aosta;AO
Ascoli Piceno;Marche;AP
Asti;Piemonte;AT
Bari;Puglia;BA
Barletta-Andria-Trani;Puglia;BT

[Chris Rebert]

On Wed, Apr 21, 2010 at 2:51 AM, Alex <metallo...@gmail.com> wrote:
> I'm trying to use the shove module (http://pypi.python.org/pypi/shove)
> for a simple script. The script read a CSV file ad store the data.
> When I check the content of the "store" object (instance of Shove)
> *before* I close it, the data are all there but when I close and re-
> open it some data are lost. How it is possible? There is something
> wrong in my code or I didn't understand how shove works?
>
> Thanks in advance.
>
> Here is a sample of my code:
>
> DBNAME = "file://store.db"
>
>    csv_file ="test.csv"
>    cities = csv.reader(open(csv_file), delimiter=";")
>    store = Shove(DBNAME, compress=True)
>    for city,region,code in cities:
>        entry_list = store.setdefault(region, [])
>        data = 'just a sample'
>        entry = {city:data}
>        entry_list.append(entry)

If `shove` is like the std lib `shelve` module, the following might
fix the problem:

#untested
for city,region,code in cities:
entry_list = store.get(region, [])

data = 'just a sample'
entry = {city:data}
entry_list.append(entry)

store[region] = entry_list

Explanation:
The explicit assignment back to the `store` pseudo-dictionary lets it
properly update its internal state to reflect the change to the value
(in this case, the list) associated with the region key. In your
original version, you merely modified the list in-place, and (due to
the way Python works) `store` has no way of knowing that you've done
that and thus doesn't do the necessary bookkeeping, hence the behavior
you're observing.

See also: http://docs.python.org/library/shelve.html#example

And further note that shove seems to be beta and (apart from
docstrings in the source code) undocumented.

Cheers,
Chris
--
http://blog.rebertia.com

[Alex]

On 21 Apr, 12:36, Chris Rebert <c...@rebertia.com> wrote:
[cut]

>
> Explanation:
> The explicit assignment back to the `store` pseudo-dictionary lets it
> properly update its internal state to reflect the change to the value
> (in this case, the list) associated with the region key. In your
> original version, you merely modified the list in-place, and (due to
> the way Python works) `store` has no way of knowing that you've done
> that and thus doesn't do the necessary bookkeeping, hence the behavior
> you're observing.
>
> See also:http://docs.python.org/library/shelve.html#example
>
> And further note that shove seems to be beta and (apart from
> docstrings in the source code) undocumented.

Thanks a lot for the clear explanation. It works!
I will read the docs more carefully next time :-)

Alex