Dynamic (on-the-fly) creation/modification of models and tables

[rsco...@gmail.com]

Well, as a first post to this list I figured I should make it a good one -- hopefully this qualifies as such. ;-)

I've had a low-level curiosity for how this might be done using peewee (outside of generating code, importing as module, etc.) as it can be useful in isolated situations. As such, I've cruised the code and poked at some live instances upon occasion (including just a few minutes ago).

All of this is because I now have a potential use for such an endeavor, at least it seems like it *may* be a good option for the problem.

So _one_ possibility for this kind of hacking is the importation of crappy or unknown data. Basically to create an in-memory staging table that can be used for analyses using SQL instead of iterating lists and dictionaries.

Here's a story line to illustrate: 

Joe: "Hey Bob, client sent us the data for their new WhizBang project but we have no documentation, schema, not even a header it seems."

Bob: "WTF?, we gotta get some better clients man -- I need a life"

Joe: "Yeah. Well 'rich' clients and 'smart' clients are not always the same entity... so they're going to pay us to figure it out."

Bob: "Oh, this isn't a weekend of comping my time then."

Joe: "Nope, full shop-rate"

Bob: "What, the full $10 an hour? Your kidding?"

Bob: "Good thing i figured out how to hack peewee for in-process table creation -- I'll just read all the data into SQlite as text columns, query their contents for types and uniqueness, and refine from there. I can copy the data into fresh tables on the fly."

While fun to write, I gotta get back to the day-job... it pays better than my non-code script-writing. And since I play the role of both parties these days, and I could have used something like this a while ago, I have been tinkering with the idea. Now, with another batch of "mystery-meat" looming on the horizon a variation of the above might just be handy... esp. since it is a bit more complex than the Bob 'n Joe show.

Anyway, I've run out of time for the moment to research options so I thought I'd post a question regarding what details need to be maintained for peewee to remain happy. 

I'm at the point where the instantiation and insertion of fields into the model._meta.tables dictionary is lacking a reference to the base class I believe. So when peewee and the back-end go to compile and then create a table "UnhappyThings(TM)" happen. Since I have a full collection of "UnhapyThings(TM)", with spares, here I am. 8-)

TIA, for any and all suggestions.

S

PS. Charles: I've been watching peewee iterate over the past few years and your 2.0 is a good jump forward, 'grats. I've found a few details here 'n there that I'll pull together a patch once my present nightmare has ceased.

[Charles Leifer]

Thank you for your interest in the project -- I'm glad you're happy with the changes in 2.0, I too feel like its a step in the right direction.

So having never been faced with this scenario I'm having a hard time picturing how it would go. It all comes down to how structured your data is.  If its already in a relational db and you have a dump of it, just import it and run pwiz to generate the schema... If on the other hand its a CSV file, as I'm inferring, your approach seems pretty heavy.

If you want to create models dynamically though, there should be nothing stopping you.  Adding fields to model._meta.fields is the wrong way to do it -- instead just instantiate your fields, then call ``field_inst.add_to_class(ModelClass, 'field_name')`` and I believe that should work.

--
You received this message because you are subscribed to the Google Groups "peewee-orm" group.
To unsubscribe from this group and stop receiving emails from it, send an email to peewee-orm+...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.