Merging two dictionaries
Douglas Garstang
I have the two dictionaries below. How can I merge them, such that:
1. The cluster dictionary contains the additional elements from the
default dictionary.
2. Nothing is removed from the cluster dictionary.
The idea here is that the two dictionaries are read from different
files where, if the value isn't found in the cluster dictionary, it's
pulled from the default one, and I can have a new dictionary
reflecting this. The update() method on dictionaries doesn't seem to
work. The resulting dictionary always seems to be the one passed as a
parameter.
default = {
'cluster': {
'platform': {
'elements': {
'data_sources': {
'elements': {
'db_min_pool_size': 10
},
},
},
},
}
}
cluster = {
'cluster': {
'name': 'Customer 1',
'description': 'Production',
'environment': 'production',
'platform': {
'elements': {
'data_source': {
'elements': {
'username': 'username',
'password': 'password'
},
},
},
},
}
}
The resulting dictionary would therefore look like this:
new_dict = {
'cluster': {
'name': 'Customer 1',
'description': 'Production',
'environment': 'production',
'platform': {
'elements': {
'data_source': {
'elements': {
'username': 'username',
'password': 'password',
'db_min_pool_size': 10 # This was added from
the default.
},
},
},
},
}
}
Thanks,
Doug.
--
Regards,
Douglas Garstang
http://www.linkedin.com/in/garstang
Email: doug.g...@gmail.com
Cell: +1-805-340-5627
Gary Herron
> On Sun, Aug 1, 2010 at 10:58 PM, Gary Herron<ghe...@islandtraining.com> wrote:
>> going on. Here I've replaced all the distractions of your dictionary
>> nesting with a simple (string) value. Now when you try to update
>>
>>
>>>>> default = {'cluster': 'some_value'}
>>>>> cluster = {'cluster': 'another_value'}
>>>>> cluster.update(default)
>>>>> print cluster
>>>>>
>> {'cluster': 'some_value'}
>>
>> If you read up on what update is supposed to do, this is correct -- keys in
>> default are inserted into cluster -- replacing values if they already exist.
>>
>> I believe update is not what you want for two reasons:
>>
>> 1. It's doubtful that you want a default to replace an existing value, and
>> that's what update does.
>>
>> 2. I get the distinct impression that you are expecting the update to be
>> applied recursively down through the hierarchy. Such is not the case.
>>
>>
>>
>>
>> And I just have to ask: Of what use whatsoever is a dictionary (hierarchy)
>> that contains *one* single value which needs a sequence of 6 keys to access?
>>
>> default['cluster']['platform']['elements']['data_sources']['elements']['db_min_pool_size']
>>
>>>>> 10
>>>>>
>> Seems absurd unless there is lots more going on here.
>>
> Thanks. Any particular reason you replied off-list?
>
Huh? Oh hell. My mistake. (This is now back on the list -- where it
should have been to start with.)
> Anyway, I'm trying to model a cluster of servers in a yaml file that
> gets edited by humans and a tree structure makes it easier to
> understand the context of each invidual key. If it was arrange in a
> flat fashion, each key would have to be longer in order to make it
> unique and provide some context as to what the user was actually
> editing.
>
> I actually didn't paste the whole dictionary. I cut it down to make it
> easier to explain. When you see the full version, the multiple levels
> make more sense. Tried various approaches so far, and none work. I
> can't traverse the tree recursively because each time you recurse, you
> lose the absolute position of the key your currently at, and then
> there's no way to update the values.
>
> Doug.
>
Ok. Thanks for simplifying things before sending the question out to
the list. You probably wouldn't have gotten a response otherwise.
I'm not sure I believe the reasoning for the inability to recurse. It
seems rather simple to recurse through the structures in tandem, adding
any key:value found in the default to the other if not already present.
Gary Herron
Douglas Garstang
Actually, I had issues with trying recurse through the structures in
tandem too. This didn't work:
for a,b,c,d in ( cluster.iteritems(), default.iteritems() ):
... do something ...
It returns an unpack error.
Doug.
Chris Rebert
<doug.g...@gmail.com> wrote:
> Actually, I had issues with trying recurse through the structures in
> tandem too. This didn't work:
>
> for a,b,c,d in ( cluster.iteritems(), default.iteritems() ):
> ... do something ...
>
> It returns an unpack error.
Well, yeah. That for-loop has several problems:
- You're iterating over the items of a 2-tuple. It's just like:
for a,b,c,d in [1, 2]:
It's not treated any differently just because the items happen to be
iterators themselves. The iterators aren't automagically iterated
through in parallel just by putting them in a tuple. That would
require a zip().
- iteritems() returns a sequence of 2-tuples. Even when zipped, these
tuples don't get magically unpacked and repacked into 4-tuples:
for a, b, c, d in zip([(1,2), (3,4)], [(5,6), (7,8)]):
# still fails; can't unpack 2 separate tuples (i.e. (1,2) (5,6) )
directly into 4 variables; the nesting is wrong
- iteritems() returns the keys in an arbitrary order; the two
iteritems() calls won't be in any way "synchronized" so the keys match
up
Cheers,
Chris
--
http://blog.rebertia.com
Paul Rubin
> default = {...
> 'data_sources': { ...
> cluster = {...
> 'data_source': { ...
Did you want both of those to say the same thing instead of one
of them being 'data_source' and the other 'data_sources' ?
If yes, then the following works for me:
def merge(cluster, default):
# destructively merge default into cluster
for k,v in cluster.iteritems():
if k in default and type(v)==dict:
assert type(default(k))==dict
merge(v,default[k])
for k,v in default.iteritems():
if k not in cluster:
cluster[k] = v
Peter Otten
> I have the two dictionaries below. How can I merge them, such that:
>
> 1. The cluster dictionary contains the additional elements from the
> default dictionary.
> 2. Nothing is removed from the cluster dictionary.
def inplace_merge(default, cluster):
assert isinstance(default, dict)
assert isinstance(cluster, dict)
d = set(default)
c = set(cluster)
default_only = d - c
both = d & c
for key in both:
dv = default[key]
cv = cluster[key]
if isinstance(cv, dict):
inplace_merge(dv, cv)
cluster.update((dk, default[dk]) for dk in default_only)
should work once you've fixed your example dicts.
Peter
Douglas Garstang
> http://mail.python.org/mailman/listinfo/python-list
>
Hmmm, using that gives me:
Traceback (most recent call last):
File "./test4.py", line 48, in ?
merge(cluster, default)
File "./test4.py", line 42, in merge
assert type(default(k))==dict
TypeError: 'dict' object is not callable
where line 42 is 'assert type(default(k))==dict', and the inputs are:
default = {
'cluster': {
'platform': {
'elements': {
'data_sources': {
'elements': {
'db_min_pool_size': 10
},
},
},
},
}
}
cluster = {
'cluster': {
'name': 'Customer 1',
'description': 'Customer Production',
'environment': 'production',
'platform': {
'elements': {
'data_source': {
'elements': {
'username': 'username',
'password': 'password'
},
},
},
},
}
}
and it's called with:
merge(cluster, default)
Doug.
Peter Otten
> On Mon, Aug 2, 2010 at 12:47 AM, Paul Rubin <no.e...@nospam.invalid>
wrote:
>> If yes, then the following works for me:
>>
>> def merge(cluster, default):
>> # destructively merge default into cluster
>> for k,v in cluster.iteritems():
>> if k in default and type(v)==dict:
>> assert type(default(k))==dict
>> merge(v,default[k])
>> for k,v in default.iteritems():
>> if k not in cluster:
>> cluster[k] = v
>> --
>> http://mail.python.org/mailman/listinfo/python-list
>>
>
> Hmmm, using that gives me:
>
> Traceback (most recent call last):
> File "./test4.py", line 48, in ?
> merge(cluster, default)
> File "./test4.py", line 42, in merge
> assert type(default(k))==dict
> TypeError: 'dict' object is not callable
>
> where line 42 is 'assert type(default(k))==dict', and the inputs are:
Not making an attempt to understand the code that you are about to use?
default(k)
should be
default[k]
Peter
Douglas Garstang
Paul Rubin
> where line 42 is 'assert type(default(k))==dict', and the inputs are:
Woops, cut and paste error. default(k) should say default[k]. Or you
could remove the assertion altogether.