Kevin meeting notes
2 views
Skip to first unread message
Jason Priem
May 7, 2012, 11:10:09 AM5/7/12
to total-im...@googlegroups.com
Kevin and I had a good meeting this morning. A few points:
1. Heather and Jason should refrain from adding much to the core code. This makes it harder for Kevin to merge new changes. Kevin is going to merge the new providers branch today. I think we'll be okay, because most of the code Heather and I will be writing is just for providers, using the already–defined API. But it's good for me and Heather to remember for the next week that things will go more smoothly if we let Kevin have free rein.
2. The new queuing structure, which parallelizes not just providers but items within each provider, works great, saving much time. Well done Kevin!
3. Work on saving metric values as metrics snaps in the database, instead of attached to the item doc, will proceed soon. This will allow us to get rid of some relatively unpleasant locking mechanisms that are currently needed to deal with update complex.
4. Kevin is concerned that creating items in a factory method is unpythonic, and suggests a more ORM style solution. This is an interesting idea (especially since flask has a nice couchDB ORM module already). Jason suggests that this might be a good thing to pursue later, after this sprint. The factory method works fine for now, and nicely decouples item creation from all its dependencies.
J
Sent from my iPad
1. Heather and Jason should refrain from adding much to the core code. This makes it harder for Kevin to merge new changes. Kevin is going to merge the new providers branch today. I think we'll be okay, because most of the code Heather and I will be writing is just for providers, using the already–defined API. But it's good for me and Heather to remember for the next week that things will go more smoothly if we let Kevin have free rein.
2. The new queuing structure, which parallelizes not just providers but items within each provider, works great, saving much time. Well done Kevin!
3. Work on saving metric values as metrics snaps in the database, instead of attached to the item doc, will proceed soon. This will allow us to get rid of some relatively unpleasant locking mechanisms that are currently needed to deal with update complex.
4. Kevin is concerned that creating items in a factory method is unpythonic, and suggests a more ORM style solution. This is an interesting idea (especially since flask has a nice couchDB ORM module already). Jason suggests that this might be a good thing to pursue later, after this sprint. The factory method works fine for now, and nicely decouples item creation from all its dependencies.
J
Sent from my iPad
Kevin Campbell
May 8, 2012, 12:13:53 PM5/8/12
to total-im...@googlegroups.com
Jason,
I think an immediate issue which is there is that we aren't necessarily writing a single document to couch anymore. I was assuming we'd end up with a calling convention live follows:
This would then result in a call to MetricSnap.save() for all snap objects, which themselves would only write to the database if they are dirty. The metric snap objects themselves would be saveables, and Item.save() would need to be overloaded to be aware of it's own internal structure.
Alternatively, I could set an attribute on a Saveable object called 'subobjects'. The save() method of Saveable would then ignore the subobjects when calling as_dict(), but subsequently call save() on each subobject. Again, I think we need some sort of _modified attribute. If we are noting which objects are dirty, we will end up with say 30+ database writes per Item save, and we're also back to having collisions.
Having the factory load and instantiate MetricSnaps when loading an Item from the database is no problem, doing that with a view is fine.
K
On Mon, May 7, 2012 at 4:10 PM, Jason Priem <wordsl...@gmail.com> wrote:
4. Kevin is concerned that creating items in a factory method is unpythonic, and suggests a more ORM style solution. This is an interesting idea (especially since flask has a nice couchDB ORM module already). Jason suggests that this might be a good thing to pursue later, after this sprint. The factory method works fine for now, and nicely decouples item creation from all its dependencies.
I think an immediate issue which is there is that we aren't necessarily writing a single document to couch anymore. I was assuming we'd end up with a calling convention live follows:
snap = MetricSnap('github:watchers', timestamp, 14)
Item.add_metric_snap(snap)
Item.save()
Item.add_metric_snap(snap)
Item.save()
This would then result in a call to MetricSnap.save() for all snap objects, which themselves would only write to the database if they are dirty. The metric snap objects themselves would be saveables, and Item.save() would need to be overloaded to be aware of it's own internal structure.
Alternatively, I could set an attribute on a Saveable object called 'subobjects'. The save() method of Saveable would then ignore the subobjects when calling as_dict(), but subsequently call save() on each subobject. Again, I think we need some sort of _modified attribute. If we are noting which objects are dirty, we will end up with say 30+ database writes per Item save, and we're also back to having collisions.
Having the factory load and instantiate MetricSnaps when loading an Item from the database is no problem, doing that with a view is fine.
K
Jason Priem
May 8, 2012, 1:59:51 PM5/8/12
to total-im...@googlegroups.com
Hey Kevin,
--
Jason Priem
UNC Royster Scholar
School of Information and Library Science
University of North Carolina at Chapel Hill
I'm not clear why we even need to be putting snaps in the item at all. The Providers give calling code metrics and their values. Seems like the whole advantage of saving snaps separately is that then we don't have to mess with working them into the object. Why can't calling code do something like:
metrics = provider.metrics(alias)
snap = {} # make a simple, light dict rather than an object;
for name, val in metrics.iteritems():
snap['name'] = name
snap['value'] = val
snap['ts'] = time.time()
snap['item_id'] = item.id
dao.save(snap)
This would work fine, I think, and be much simpler. The only time we ever need the item to contain its snaps is when we display it, and the itemFactory takes care of that.
j
--
You received this message because you are subscribed to the Google Groups "total-impact-dev" group.
To post to this group, send email to total-im...@googlegroups.com.
To unsubscribe from this group, send email to total-impact-d...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/total-impact-dev?hl=en.
Jason Priem
UNC Royster Scholar
School of Information and Library Science
University of North Carolina at Chapel Hill
Kevin Campbell
May 8, 2012, 2:21:03 PM5/8/12
to total-im...@googlegroups.com
Jason,
You mean entirely detach item and metric? I can certainly do it like that.
In which case, loading the item from the factory would populate the metrics no different from how it currently does it as it would get the data from a view.
Regards,
Kevin
You mean entirely detach item and metric? I can certainly do it like that.
In which case, loading the item from the factory would populate the metrics no different from how it currently does it as it would get the data from a view.
Regards,
Kevin
Jason Priem
May 8, 2012, 3:37:55 PM5/8/12
to total-im...@googlegroups.com
Yep, I think that's the way to go...looks like nothing but win from here. Would be happy to reconsider if i'm missing an important downside of some kind, though.
j
Reply all
Reply to author
Forward
0 new messages