Stalker

[Erkan Özgür Yılmaz]

Hey everybody,

I was not planning to write this before next year or so, but in a recent chain of emotional reactions I've started to think about that the time has come.

My name is **Erkan Ozgur Yilmaz**, probably a couple of you chubs would know/recognize me from my not anymore very active blog or from this group. I'm the writer/architect (always loved this title) of a python library called **Stalker**, which happens to be an Open Source Production Asset Management Library, that sounds like a bold description, and I've always scared about the heavy weight of this project title.

Stalker is written to solve things that I define to be a problem in my everyday work. It has started to be the library to fit in to the way we work in our studio or others may work in their studios. I've always tried abstract it from the studio it is developed in and always tried to look from a much wider perspective to keep it useful for most of us.

If you start getting excited about it, I've written a lot of things in to the not quite finished documentation of Stalker and it is a good idea to start reading the API tutorial page at http://pythonhosted.org//stalker/tutorial.html (probably the installation page needs to be updated, but it is not so much different than installing an ordinary python library with setuptools).

The source code can be found in https://github.com/eoyilmaz/stalker

We are using Stalker along with other libraries like **Stalker Pyramid** (Pyramid based WebApp) and **Anima** (Pipeline tools developed on top of Stalker) in our current feature film project in Anima Istanbul.

I don't consider **Stalker Pyramid** and **Anima** as solid as Stalker it self but they are a bunch of example that shows how Stalker can be used in a Web application and in PyQt4/PySide UIs (interested in PyQt4/PySide UIs for Maya, Nuke, Fusion, Houdini, Photoshop check "anima.env" and "anima.ui" modules).

Although Stalker became a mature and production stable (developed with TDD practices from day 0, currently has around 1500 tests and growing), I don't think that I'm going to stop developing it, it will grow older with me.

So, please use it, test it and most importantly enjoy it.

Cheers,

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

[Chad Dombrova]

Erkan,

I read through your tutorial and checked out some of the code.  This is really good stuff.  The code is super-clean, well organized, and well documented.  I also really like the sqlachemy mixin design.  Very nice work.  Anyone out there who is looking to setup a custom production tracker should have a look at this, even if just to learn from it.

Your models as a great starting point, but most studios will of course want to associate additional data with shots, projects, sequences, etc.  I'd like to hear how if you support this or plan to support it.

I noticed that the docstrings are all well-formatted for sphinx:  do you have the full api docs generated somewhere?  it would be great to see it on https://readthedocs.org/

It would also be helpful if you had a little sandboxed demo of stalker-pyramid running. 

Thanks for open sourcing this.

chad.

[Nils Lerin]

That's incredible Erkan! I've been planning on developing something very similar to stalker for some time now but seeing how stalker seems to do exactly what I wanted I might continue from there. You don't happen to have a video or screenshot demonstrating the Anima interfaces?

Thanks for sharing it!

/Nils

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/06444F30-DEB0-4385-910C-DDEB5D98B6D1%40gmail.com.

For more options, visit https://groups.google.com/d/optout.

[Erkan Özgür Yılmaz]

Hi Chad,

It is my honor to hear a python experts thoughts on Stalker. Thank you very much for your compliments, I tried to write it as clean as possible.

Customizing Stalker is possible by instantiating the SOM classes that you think that fits to your needs. Also before doing that I am very open to new ideas to be implemented directly to Stalker it self.

I'll upload the documentation to readthedocs.org or you can read the whole API documentation from http://pythonhosted.org/stalker/.

For Stalker Pyramid, I don't think that it is ready to be announced right now. Currently, it is shaping according to our daily needs and what me and my wife is coding in to it is a little bit quick and dirty. We probably will write it from scratch after the production of the current feature animation project is completed. But I can upload screen shots of it, you probably will enjoy the UI, UI is good but the backend is not in good shape.

Cheers

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

[Erkan Özgür Yılmaz]

Hey Nils,

Go ahead, fork Stalker and Anima in GitHub (Stalker Pyramid is not on GitHub yet).

I've not written any documentation on installing Anima along with Stalker yet and sadly there are no screen casts, I will try to do it as soon as I've time.

What you need to know about Stalker, Stalker Pyramid and Anima is the level of generality of each of them:

Stalker (LGPLv2) > Stalker Pyramid (LGPLv2) > Anima (BSD)

So, Stalker is the most generalized one, I tried to make it as generic as possible.

But, I think with v0.2.5 I had to introduce what I've called the **Task Status Workflow**. It is the workflow of statuses that a task/asset/shot/sequence can have on their life time.

This part is not that much open yet. That is, Stalker will force you to follow a scheme, you can read it from http://pythonhosted.org//stalker/generated/stalker.models.task.Task.html#stalker.models.task.Task

Also the Ticket statuses are fixed and hard coded, but it is a little bit more open then Tasks (you can define the ticket workflow in config.py as a python dictionary).

But I plan to merge the both status workflows of Tasks and Tickets in to something that is managed by a custom class which may be called StatusManager. So Stalker will have a default status manager for Tasks and then one for Tickets but then you can setup your installation to use your own custom StatusManager.

Stalker Pyramid is tried to be written as generic as possible again, but I think we have failed on that. It is currently working only with PostgreSQL, even SQLite3 is not supported. And we didn't follow TDD practices at all, cause we were learning how to create a web application let alone how to test it. Also we were not aware how to test javascripts etc. That's way it is full of regressions. And we are very good at adding new buggy code on to it everyday.

I plan to do a rewrite on Stalker Pyramid so it will have a good backend somewhere around next year or so.

Anima, is fully customized to fit in to the pipeline we are trying to create here in Anima Istanbul (and our pipeline is having its baby steps right now). So it is tailored for our needs, I don't think that it is going to fit 100% to any other studios needs.

Anima is licensed under BSD and I think I should convert it to MIT, so you guys can use it as a quick and dirty place to copy and paste the code from, it may help you at least tackle with the problems that I've had (like displaying one cell of a QTreeView as a QButton or displaying/fetching data on demand etc).

So again please fork them, and play with them.

Best,

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

[Chad Dombrova]

It is my honor to hear a python experts thoughts on Stalker. Thank you very much for your compliments, I tried to write it as clean as possible.

All well deserved.

I'll upload the documentation to readthedocs.org or you can read the whole API documentation from http://pythonhosted.org/stalker/.

either site is fine.  what I was referring to is the auto-generated API documentation (think doxygen), which can be creating using autodoc / autosummary  for sphinx. 

For Stalker Pyramid, I don't think that it is ready to be announced right now. Currently, it is shaping according to our daily needs and what me and my wife is coding in to it is a little bit quick and dirty. We probably will write it from scratch after the production of the current feature animation project is completed. But I can upload screen shots of it, you probably will enjoy the UI, UI is good but the backend is not in good shape.

Luma is working on some web components which might work nicely with Stalker.  We are using the Meteor web framework, which runs on Node.js and provides some incredibly powerful features.  I'm not a web expert -- I'm just managing the project -- but what I like about it is that we get full "reactivity" (when the db backend is updated the UI updates and vice versa) for free.  Meteor also encourages best-practices like modular, reusable, and test-driven designs, wherein most of the heavy lifting is done in javascript, and the UI components are instantiated and configured per application using a light-weight templating language much like jinja2.

Currently we're building out some of the lowest level components based on some popular open-source projects:  UI widgets and styling based on bootstrap.js, and data filtering/sorting based on datatables.js.  You can see them both in action here: http://jquery-datatables.meteor.com/.  The datatables component is able to filter and sort through HUGE amounts of data very rapidly, and as new data is added to the database the contents of the tables will be updated in real-time.  Soon we'll have some examples of how to style cells per data-type, e.g. displaying timestamp data as a calendar date selector, image paths as embedded images, integers as spin-boxes, etc.

The big question is how do you use sqlAlchemy / python models in a meteor / javascript web application?  

First some background: Meteor relies heavily on MongoDB to provide its reactivity.  As a NoSQL database, Mongo uses  structured json documents instead of normalized relational tables, like postgres.  You can think of these NoSQL documents as "pre-joined" relational data; joining across a foreign key in your relational database is like adding a nested sub-document to your json document.   

Back to the question: We're currently working on a component which uses use sqlAlchemy models like those in Stalker to create a 2-way sync between postgres and mongo.  This allows your applications to access the same data in either style -- relational or document-based -- depending on their need, with full read/write support for both styles.  So, your pipeline code can use a sqlAlchemy DOM, while your web application uses Meteor's DOM (based on mongodb), and when you commit a change to postgres via sqlAlchemy, all web clients viewing that data will be immediately updated.

My dream was always to write our next-gen internal web tools in python -- using flask, pyramid, and/or tornado -- and we still do this for basic apps, but the advantages of Meteor and Node have convinced me that it's worth adding a new language to our repetoire.  So, if you're willing to dig into some javascript, you might consider taking a look.  Austin, our web developer, pays close attention to the github repos and will be glad to point you in the right direction.  Meteor has a steep learning curve, but we're hoping that we can make getting started with our components as simple as running some code generators and writing a few templates.

chad.

[Erkan Özgür Yılmaz]

Hey Chad,

Thats very interesting stuff, I was planing to use gevent along with pyramid but I think I also should heavily evaluate Meteor (though this mongodb <-> sqlalchemy is scarering me a little bit).

Also I'm not sure that I was clear about the docs, the auto generated docs for Stalker are in http://pythonhosted.org/stalker/ just slide down to the very end of the page to start seeing the api doc links.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/696CA5EE-8534-4F80-B236-974186A58676%40gmail.com.

[Justin Israel]

On the topic of introducing other languages in a studio, has anyone had experience trying Go in a production VFX pipeline? I've brought it in for one tool rewrite, and one web service so far, and have been really hoping to see it pick up as a standard in pipeline. Benefits that relate specifically to our world are things like:

* Easy syntax that caters to Python developers

* Compiled, static binaries that are easy to deploy 

* Fast compilation so it makes iterating version fast when in a crunch or needing to fix fast

* Performance on par with C++/Java 

* Statically typed language prevents those pesky bugs that can come out of python when someone doesn't have a test that covers every single code branch. So generally when you mess up...you find out at compile time, instead of in production when you see "Unbound local variable 'foo' does not exist". 

* Super strong support for writing servers/services, both in the standard library, and the number of 3rd party packages

* Core concurrency concepts

I've managed to get a couple other co-workers stoked on the idea, so hopefully I will see some more tools come about. To me, it seems like it is an extremely good fit for pipelines

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGNmyx5D-H-_BuTej8tbT_Pd5Ta4wYPGYxG%3DMCe8jb8xL-Ji9A%40mail.gmail.com.

[Eduardo Grana]

Hello Erkan!

This is great. Thank you very much.

Hope this is not too much of an off topic question,

since maybe it is more related to task juggler.

Most of the time, in the productions i work we try

to make the task as paralell as possible, so we try

to put thing in a way that when some task is started

the dependent tasks can start using as input a wip of

the parent task. Of couse, in order to finish the dependent

task, the parent task must be finished.

Is it possible to handle this 'double dependecy' where the

start of the task depends on one status of the parent task,

and the end of the task depends on other status of the

parent task?

Hope this makes sense, otherwise please let me know! XD

Cheers,

Eduardo

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1OKd0VxLT2juZ4pB0GuDg3E3zb43g7cvob2tw_RpRbPA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Eduardo Graña
www.eduardograna.com.ar

[Erkan Özgür Yılmaz]

Hi Eduardo,

Before going any more detail, let me clear something, the start and end dates of a parent task is gathered from the start and end dates of the children tasks.

What you should refer to should be the "dependent task" then it is possible in Stalker (as it is in TaskJuggler), just set the dependency target to "onstart" and give a "gap duration".

The following is a code snippet showing this (without the boiler plate of creating a project and other stuff):

m = Task(name="Model")
r = Task(name="Rig", depends=[m])

# the m.depends is a list of tasks that
# this one depends to,
#
# where as the m.task_depends_to is
# a list of TaskDependency instances
# which holds the details of the
# task to task relation

dp = r.task_depends_to[0]
dp.dependency_target = "onstart"
dp.gap_timing = 4
dp.gap_unit = "d"
dp.gap_model = "duration"

# the default is effort which
# considers the given duration
# as work time, where "duration"
# is calendar duration

The above code will schedule the rig task so it will start after 4 calender days of the start of the model task.

If we didn't change the task relation Stalker will schedule them so rig follows the end of model (at least if their resources are available on that dates).

I think this answers your question.

Oh, one more thing, after reading your question again I get the idea that you are refering the statuses of tasks as the driving force for the other task to start, if that is correct then I should make it clear again that task statuses are the result of other actions, they do not change anything, and it is not possible yo change the status to some certain other statuses, you should read the Task class documentation in https://pythonhosted.org/stalker/generated/stalker.models.task.Task.html#stalker.models.task.Task

I mean, you do not change the task status by hand (other then on some edge conditions which are very rare and I fix them as soon as I find them), but take some actions like completing a dependent task, entering a time log or requesting a review or a revision etc.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6GrnGLE6yDO5Px%3D3skii2V14O-q2rH0Hfa02eEEJ5xaGjww%40mail.gmail.com.

[Chad Dombrova]

Thats very interesting stuff, I was planing to use gevent along with pyramid but I think I also should heavily evaluate Meteor (though this mongodb <-> sqlalchemy is scarering me a little bit).

So far the mongodb <---> postgresdb prototype seems to be working well.  once we have the connector in a testable state, I'll let you know.  Even without meteor, I see the ability to have simultaneous access your data via SQL or NoSQL as a form of future-proofing.

Meteor seems to have a lot of momentum right now.  To get an idea of its meteoric rise (pun intended), the repo has been starred 12,458 times on github and has almost 1,800 forks in just a few years (compare that to Pyramid's 1,372 and 472, and you get an idea of relative adoption).  In 2012 they got a first round of venture capital to the tune of $11.2 million, backed by some huge names. 

One more question about stalker.  I was looking at this part of the tutorial: http://pythonhosted.org//stalker/tutorial.html#part-vi-asset-management

A FilenameTemplate defines how to generate a path for a Version instance.  How do you go the other direction: lookup a Version instance and its associated task from a file path?

Also I'm not sure that I was clear about the docs, the auto generated docs for Stalker are in http://pythonhosted.org/stalker/ just slide down to the very end of the page to start seeing the api doc links.

ah, whoops. I stopped scrolling when I got to the changelog. 

chad.

[Erkan Özgür Yılmaz]

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

On Sun, May 11, 2014 at 9:46 PM, Chad Dombrova <cha...@gmail.com> wrote:

Thats very interesting stuff, I was planing to use gevent along with pyramid but I think I also should heavily evaluate Meteor (though this mongodb <-> sqlalchemy is scarering me a little bit).

So far the mongodb <---> postgresdb prototype seems to be working well.  once we have the connector in a testable state, I'll let you know.  Even without meteor, I see the ability to have simultaneous access your data via SQL or NoSQL as a form of future-proofing.

Meteor seems to have a lot of momentum right now.  To get an idea of its meteoric rise (pun intended), the repo has been starred 12,458 times on github and has almost 1,800 forks in just a few years (compare that to Pyramid's 1,372 and 472, and you get an idea of relative adoption).  In 2012 they got a first round of venture capital to the tune of $11.2 million, backed by some huge names. 

Those are great numbers and I would be appreciated to be informed about the state of the mongodb to postgresdb bridge.

 

One more question about stalker.  I was looking at this part of the tutorial: http://pythonhosted.org//stalker/tutorial.html#part-vi-asset-management

A FilenameTemplate defines how to generate a path for a Version instance.  How do you go the other direction: lookup a Version instance and its associated task from a file path?

The path of a Version instance is stored in DB with the Repository path is stripped out, so it is basically a repository root relative path.

What I do when I have a full path is, first to find the Repository, strip the repo parth from the full path and then search for a Version with that string.

I do it in Anima as follows (this is a snippet from the anima.env.base.EnvironmentBase class):

class EnvironmentBase(object):

    @classmethod
    def find_repo(cls, path):
        """returns the repository from the given path

        :param str path: path in a repository
        :return: stalker.models.repository.Repository
        """
        # first find the repository
        from stalker import Repository

        repos = Repository.query.all()
        found_repo = None
        for repo in repos:
            if path.startswith(repo.path) \
               or path.startswith(repo.windows_path) \
               or path.startswith(repo.linux_path) \
               or path.startswith(repo.osx_path):
                found_repo = repo
                break
        return found_repo

    def trim_repo_path(self, path):
        """Trims the repository path value from the given path

        :param path: The path that wanted to be trimmed
        :return: str
        """
        # get the repo first
        repo = self.find_repo(path)

        if not repo:
            return path

        # then try to trim the path
        if path.startswith(repo.path):
            return path[len(repo.path):]
        elif path.startswith(repo.windows_path):
            return path[len(repo.windows_path):]
        elif path.startswith(repo.linux_path):
            return path[len(repo.linux_path):]
        elif path.startswith(repo.osx_path):
            return path[len(repo.osx_path):]
        return path

    def get_version_from_full_path(self, full_path):
        """Finds the Version instance from the given full_path value.

        Finds and returns a :class:`~stalker.models.version.Version` instance
        from the given full_path value.

        Returns None if it can't find any matching.

        :param full_path: The full_path of the desired
            :class:`~stalker.models.version.Version` instance.

        :return: :class:`~stalker.models.version.Version`
        """
        # convert '\\' to '/'
        full_path = os.path.normpath(full_path).replace('\\', '/')

        # trim repo path
        full_path_trimmed = self.trim_repo_path(full_path)
        logger.debug('full_path_trimmed: %s' % full_path_trimmed)

        from stalker import Version

        # try to get a version with that info
        version = Version.query\
            .filter(Version.full_path == full_path_trimmed).first()
        return version


Even though it is working great, I don't think that it is the best way of doing it, I should probably implement a nicer SQL query to Stalker and be sure that the resultant Version is really stored in that repository (meaning version.task.project.repository == repo)

 

Also I'm not sure that I was clear about the docs, the auto generated docs for Stalker are in http://pythonhosted.org/stalker/ just slide down to the very end of the page to start seeing the api doc links.

ah, whoops. I stopped scrolling when I got to the changelog. 

chad.

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7GTSdgaosdNrsHr2tepamMpWVVhEv_1e-o3EN-8LC-8Ng%40mail.gmail.com.

[Chad Dombrova]

Another question regarding sqlalchemy and the inheritance model:  http://pythonhosted.org/stalker/design.html#inheritance-diagram

which of these classes correspond to a table in the database?  all but the mixins?

does each table have a foreign key to the 'id' field of its super-class table?  for example, does the Shot table have a foreign key to Task.id, and the Task table have a foreign key to Entity.id, and so on?

thanks,

chad.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGNmyx6rD68Xbf-CvLbPMhnjhhtZfijTajHNq36bnEYRsCEKEw%40mail.gmail.com.

[Fredrik Averpil]

Looks very interesting for command line tools. Are all these packages included, or do they have to be downloaded/installed prior to using?

http://golang.org/pkg/

It looks like third party Go packages are cross platform, text based (non-binary). Is this so?

That would be awesome for easy deployment in a multi platform pipeline.

// Fredrik

[Erkan Özgür Yılmaz]

Yes, exactly as you've said, all of the classes except mixins corresponds to a table and Shots.id -> Tasks.id -> Entities.id -> SimpleEntities.id all are foreign keys to the next. So querying a SimpleEntity will may return a mixed list of other entity types, this is the power of SQLAlchemy and Joined Table Inheritence model and it is working very well.

Let me point some other important information, because Assets, Shots and Sequences are derived from Tasks, creating a bunch of assets and shots or sequences and querying them by using Task class (using Task.query) is working very nicely. You do not care about if the data is an Asset or Shot until you need to know it. This helped us a lot when writing the web templates, we just wrote a template for the Task and it worked well with the other types.

Also another point Shots are not in anyway forced to be related to a Sequence. In fact a Shot can be connected to more than one Sequence (think about a shot is being the flashback cut in another sequence so it is appearing twice in the movie etc.).

Because of the freedom in the parent/child relation of the Tasks, you can setup things like a Shot is the parent of an Asset or the reverse like an Asset is the parent of a Shot or something meaningless like a Shot is a parent of a Sequence etc. you are free to do that. That brings a lot of power when customizing the hierarchy. This feature alone makes Stalker to fit in a lot different workflows.

E.Ozgur Yilmaz
eoyilmaz.blogspot.com

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7GHiFK8n%2B8JgcKmam3B9RLcpmDRh6RO8KXyfpoSDoPT2Q%40mail.gmail.com.

[Justin Israel]

(Sorry, I realized after I posted my side-topic that I might hijack the thread with it. I can move it to a new thread if need be)

@Fredrik 

Ya what you are pointing at is the standard library, so it is all included. Go compiles to native binary, but the source is all cross platform. You can even cross-compile from one machine. I have built Linux and Windows binaries, from my OSX box. Its heavily suited for writing servers/services, but also command line tools. One of the command line tools I replaced at work was originally written in Python and was heavily accessed. Writing it in code reduced file ops from the hundreds of thousands per invocation, down to a couple hundred. 

It has a built in package installer:  go get [some repository]

So installing new packages is kinda like using pip with python. Except once you compile your binary, it is static so you don't have the equivalent of a PYTHONPATH afterwards, since everything is in the single binary executable. 

Again, we can move this to another thread if we want to continue this convo. 

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGNmyx742NAKn_AncS9ZuswpMa%3D6BvP39WNy%2Bf4K5ev3Bb3bvQ%40mail.gmail.com.

[Shih Hwa Lai]

http://www.meteorpedia.com/read/DDP_Clients#Python

I wonder if we can use a python DDP client as data source instead. This way Stalker server needs only send JSON respones to meteor client.

[Chad Dombrova]

On Thu, May 29, 2014 at 8:54 AM, Shih Hwa Lai <dsh...@gmail.com> wrote:

http://www.meteorpedia.com/read/DDP_Clients#Python

I wonder if we can use a python DDP client as data source instead. This way Stalker server needs only send JSON respones to meteor client.

that's the plan :)

we're still chipping away at this both from the meteor side and from the sqlalchemy-to-mongo side.  Will let you all know when we have something ready to show.  Hopefully in a month.

-chad.

[Marcus Ottosson]

that’s the plan :)

The plan for Stalker? Are you helping Erkan develop this? That’s awesome!

​

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7E%2Bg6T%2BC%2B63pC7-xYZ_3RLr_8LUOMx1V%2B-mfgerJrveRA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

Chad,

I'm soon done with a small addition to stalker, and after that I'm adding the possibility of adding additional data to any such entity in stalker. Personally, I'm going to store dictionaries (keys and values) / JSON style but I'm probably going to just allow anyone to store whatever (TEXT). Did you have anything particular in mind here?

// Fredrik

[Chad Dombrova]

On Thu, May 29, 2014 at 12:43 PM, Marcus Ottosson <konstr...@gmail.com> wrote:

that’s the plan :)

The plan for Stalker? Are you helping Erkan develop this? That’s awesome!

We're working on a web UI built using meteor, which is what Shih Hwa Lai was referring to.  Our front-end will work with any relational database that has well defined schema (i.e. defined using something like sqlalchemy or bookshelf.js), so I think it will be a perfect complement to Stalker.

chad.

 

[Fredrik Averpil]

Actually, Erkan suggested PickleType so I might go for that.

// Fredrik

[Justin Israel]

Would PickleType imply that you are storing Python-specific data in the database? Maybe json is better since it means the data is retrievable by any type of client? Unless of course the data is always being served by a python application server to the clients. 

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWNg0uhqqJbJMfbarLH_rLyGq052J%3DO%3DjJYP4-gUzSHWnw%40mail.gmail.com.

[Erkan Özgür Yılmaz]

@Shih Hwa Lai: Stalker is simply the model part of an MV, MTV or MVC application and it uses SQLAlchemy as the ORM. So your suggestion is more related with SQLAlchemy itself and as Chad said he and his team is working on an interface similar to what you have suggested and I'm very excited about what they are going present at the end.

Also I want to thank to Fredrik about his contribution to Stalker, he did a very good job while he was extending Stalker according to his studios needs, it was something that I've always dreamed of.

And what he is going to do probably next week is to add support for storing arbitrary Python objects (anything that is able to be pickled) inside any entity in Stalker, which will extend Stalker in a way that makes it fit in to more different pipelines. Again thank you for your contribution.

Ozgur
eoyilmaz.blogspot.com

[Fredrik Averpil]

Justin, that's a very good point. That's why initially I was thinking of just storing plain text. But being able to store Python objects is also quite powerful. I think I might ending up implementing both as two separate attributes on the entity.

Erkan, Hehe... you're being way too modest. I couldn't have done it without your excellent help. :) And yes, the plan is to implement the metadata stuff next week.

Having had some time to get to know stalker now, I can't believe how well written and well thought out the whole thing is.

// Fredrik

[Marcus Ottosson]

Ah sorry, missed the quoted part. Thanks for clearing that up!

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGq9Q7HBqDNP9FQ9OAg%2B4FvmqZhkZ12pV1Sws494jfh%2ByijW1A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Nils Lerin]

Regarding the addition of storing TEXT such as JSON. Whouldn't that just mean adding a attribute/column to a entity of type text to hold your data? Or am I missing something?

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOD5OX54VKFscf%3D%2Brdw8mrf9NYZN6GJi93VvfV4eo%2Ban%3Dg%40mail.gmail.com.

[Marcus Ottosson]

I'm assuming they're referring to having serialisation/de-serialisation happen server-side/automatically. As anything, even Py objects, could get stored as text anywhere.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGkBUFfP9oo93_qRbNQYMtofAFMQbSAGR1cz-%2BU_YndvX2ehbA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

Yes.

[Chad Dombrova]

I've got to admit, I don't love the idea of storing pickle strings in a database. There are so many ways that it can go wrong based on differences between the environment on the machine that writes the pickle and the one that reads it.  Pickles are a great short term storage format, but they are pretty horrible for long term storage (changed the name of a module and all your data breaks). I would not want to build my pipeline around something so fragile. 

If custom columns are needed the first thing I would look into is ensuring that the Stalker base classes are easily extended with new sqlAlchemy properties. The second thing I would look at is storing data as json: Postgres actually has a json field type and support for this is going to get even better with version 9.3, allowing keys to be used as indices in queries a la mongo. 

My 2 c

-chad

—
Sent from Mailbox

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAGkBUFfP9oo93_qRbNQYMtofAFMQbSAGR1cz-%2BU_YndvX2ehbA%40mail.gmail.com.

[Tony Barbieri]

I agree with Chad and Justin.  I would avoid storing a pickles for all the reasons listed above.  Storing a data type that can only be read by a specific language in the db seems to be getting too specific.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/1401463325588.4512584d%40Nodemailer.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Marcus Ottosson]

I'm with Chad on this one.

In fact, I'd stay away from pickle altogether; if for no other reason than to enforce thought into what data you eventually end up persisting/send across a network. Pickle is much to kind in what you're allowed to send/store.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/1401463325588.4512584d%40Nodemailer.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Fredrik Averpil]

You are probably completely right. I wasn't thinking of storing anything but plain text initially. I'm using MySQL (not Postgres) and to my knowledge there's no special JSON field type so it'll just be plain text.

// Fredrik

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODwy6hVSgh0u%2BoVwC0tk13kn03S%3Dkap%2BsnGthFDKxV4gg%40mail.gmail.com.

[Marcus Ottosson]

Storing plain-text as pickled Python objects?

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAD%3DwhWPKiqdQ5Ra%3DXCBHtYfgt5GrKioZENPRH8ip3xXR4OEnbA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Yea totally agree with Chad's detailed reasons not to put pickle data in a database. It is much better suited for a private data store used specifically by a python application, and not a generic storage format. I think Postgres also has other really awesome field types like being able to store dictionaries that are all indexed on the keys, as well as indexed arrays. Mysql doesn't have this stuff I don't think. 

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAiBO8Zn%2BAAbrZpWSn9Cw0NSJFuoyo08upz8obVsXkbKw%40mail.gmail.com.

[Erkan Özgür Yılmaz]

I see, you are all right about pickled data and I've had issues with them.

I think I should consider using only plain json formatted text as custom data where needed.

Ozgur
eoyilmaz.blogspot.com

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0Y2XO80-rDeNNA_%3DBnqis0cTvZZbAWxu5cEtMp9-J0Jw%40mail.gmail.com.