[Maya-Python] Simple Web Service to track scripts usage

[Eduardo Grana]

Hello People!

I was thinking of doing a simple server for tracking the usage of some

tools via web, something that,

- accepts data and that you could retrieve it later in block.

- give response to calls with simple arguments

Any advice on what type of server to implement?

Any advice on how or where to host it?

Thanks in advance,

Eduardo

-- 

Eduardo Graña
www.eduardograna.com.ar

[Justin Israel]

Depending on how you want to expose it you could use a client library that accesses something like ElasticSearch, or mongodb, or Redis. They are document or key/value-based and would have flexible storage for the data.
Or you could have a really simple service exposing an API using some micro Web framework like flask or bottle.
If you want to have the freedom to modify the back end without impacting your running apps too much then the service route gives you that separation where all of the logic is in the service and not the client.

--
You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6Gr%3D1_cOL%3D6zkRdU0L5yAqdfHgqmM%3DLZ-5XmszKj08yyueg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

[Marcus Ottosson]

Hi Eduardo,

I’ve got nothing in the pre-packaged arena to recommend, but if your looking to roll your own system, read on.

• accepts data and that you could retrieve it later in block.

Before having a server that can accept data, have you considered how to transmit your data? For instance, if you’re looking at rolling your own system, then one of the pattern you could use is called PUB/SUB, for publish/subscribe. I.e. your methods/functions publishes data and then something is out there listening. Also called producer and consumer.

Once the consumer has collected data, say in a mongodb database or in a simple text file, you can provide access to this datastore via a web interface, command-line or pyqt gui. Depending on your use, it may be helpful to think of keeping the production and consumption of your data separate; the opposite being finding a pre-packaged web-server and designing your messages for it specifically.

If separate, say in a text-file, retrieving the data is a mere open(path, 'r'), this is then equally applicable to both your web-server, and tools.

The production of message could look something like this:

 ___________
|           |
|   start   |--> publish "A: starting MyTool"
|___________|
      |
 _____v_____
|           |
|   meth1   |--> publish "A: running MyCommand"
|___________|
      |
 _____v_____
|           |
|   meth2   |--> publish "A: wrapping up MyTool"
|___________|

Each message potentially transmitting dates and user and machine stats etc. You’d then have a consumer, listening for anything tagged “A”.

       __________
      |          |
("A"--|  listen  |
      |__________|

The important thing to note being that producing messages is a non-blocking operation and that there mustn’t be a consumer/listener at the other end; that is optional.

give response to calls with simple arguments

For this, there is the REQ/REP pattern, for request/reply, in which you could either send a message, synchonously, and await an arbitrary response:

>>> unpacked_message = {'status': 'ok', 'command': 'givz_cat'}
>>> packed_message = json.dumps(unpacked_message)
>>> socket.send(packed_message)
>>> packed_reply = socket.recv()
# Blocks here
>>> unpacked_reply = json.loads(packed_reply)
>>> process(reply)

Or you could go for something like the RPC pattern which mimics calling upon procedures as though they were local:

>>> reply = myobj.givz_cat()
>>> process(reply)

Each perform similar tasks (i.e. the bundling and serialising of commands) whereas an RPC masks it under a familiar, higher-level interface.

And finally, to make any of this possible, you can either roll your own low-level messaging mechanism, which in the case of REQ/REP isn’t that big of a deal (with e.g. socket module) depending on your aspirations, but for PUB/SUB would make life slightly more difficult. For this or both, I would suggest a MOM, or Message-Oriented Middleware, such as ZeroMQ. Whichever you choose, there are tutorials and use-cases in the ZeroMQ guide.

Hope it helps.

Best,
Marcus

​

[Eduardo Grana]

Thanks Justin and Marcus for your replies.

I was thinking on something that i could use 'out of the box',

something that was already made and hosted, where i just

code for the logic on the server, and then use it like

a soap service or a wsdl, something like this example

http://www.diveintopython.net/soap_web_services/introspection.htmlhttp://www.diveintopython.net/soap_web_services/introspection.html

Sounds like something reasonable?

Thanks again!

Eduardo

--

You received this message because you are subscribed to the Google Groups "Python Programming for Autodesk Maya" group.
To unsubscribe from this group and stop receiving emails from it, send an email to python_inside_m...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBC0NrHqYbkNDm%2BDHTnxeYtRUpafv70tKqT9ZSqqgrDOA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Eduardo Graña
www.eduardograna.com.ar

[Marcus Ottosson]

Ah, in that case, there was a thread put up a while back about sentry. Haven't used it, but maybe it's close to what you're looking for?

https://pypi.python.org/pypi/sentry

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6GrnQKYtjvamUZW_aopazG%3DJTBbOxV31kuUndCczBgfeTKA%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

Looks cools. I couldn't tell at a glance if it was fully paid service or if the Pypi package is a self hosting open source version.

For something a bit more generic that let's you visualize basically anything in your custom data you can also check out Kibana:
http://www.elasticsearch.org/overview/kibana/

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAtaygYqhAQV9RfYVXbEqcWE%2BctCuNicsdw%2Bb2vMmiPOw%40mail.gmail.com.

[Marcus Ottosson]

At the end of the day though, both of these seem rather heavy handed and probably better suited for internet-based/big-data applications. Monitoring tool usage probably has less pre-packaged alternatives simply because it may considered something rather straightforward.

Eduardo, you would probably be better off taking the time to understand how logging works and how to distribute it, as you'd probably spend an equal amount of time figuring out either of these products and still be none the wiser about what the users of your tools are up to.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA0FU-Vue0pYUAwgQ8ZKb0bim%2BQoA3MhJ-6XycWXPLZc7w%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Justin Israel]

I don't see this as heavy handed at all (unless you are paying for something). It is relatively simple to spin up an instance of elastic and start throwing arbitrary data at it.
We have people here at the studio in different departments that use both Kibana and Grafana (graphite) to visualize data that we are chucking into elastic. Once you have it set up you are just making different graphs and aggregations. Or making a query to say "give me all messages for application foo between this date range and with 'warning'  in the message"

It would probably be more work to use a basic logging system and then have to use your own tools to parse and visualize the data.

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOAJ7DUfCvEJQiR_rBcfrQJYTapmfqEJ%3D-cENgFmDmom8A%40mail.gmail.com.

[Marcus Ottosson]

You are always paying for something - cost comes in the flavours "juicy-money" and "tasty-time" :) 

Eduardo, do you have any requirements for your system? How many users do you expect per tool?

My comments are based on there being 1-3 TDs at your facility and there being around 5-10 artists/tool. If you're closer to 50 TDs and 500 artists per tool, my advice would put you in a battle against Goliath. :) In this scenario, I would instead opt for Justin's advice.

With more details, I think we could all come to a reasonable conclusion to your goals.

Best,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAPGFgA1nSn%2BY4vSO4gBt7Ntd8U%3D9c0w2XN78rmbjxgvfYv%3DOhg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Eduardo Grana]

Hey Guys!

Thanks so much for all the information you gave me.

I'll have to spend some time testing all these options.

Sentry seems the easiest thing, although you probably

end up paying...

I think i'll try to get a hosting and try some stuff in there,

yesterday i made an account in pythonanywhere, and

i was able to try flask, sounds promising. Gonna try django too.

The use of the system its a little weird,

I usually work as a freelance, in several studios at the same time,

so its actually just me as td, but there are several facilities, and

several users. Also there are users taking some scripts home to work on weekends.

So the idea is to be able to track changes on the enviroment and the effects

on the scripts, also to measure the effectiveness of the different features based on usage.

Hope this description is usefull!

Cheers!

Eduardo

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOBRBPc04Pt5RhhHX4Kkpez3Z4XH5gJK_C2te3Qhsxxf%3Dw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Eduardo Graña
www.eduardograna.com.ar

[Tony Barbieri]

Sentry offers an open source, self hosted option.  You have to set it up yourself of course and will only receive support from their forums.  You could set it up on a personal server somewhere and have all the scripts at the various facilities report into it.

I would double check with the facilities before hand to make sure they are okay with their data streaming out of their internal networks with whatever setup you end up doing.  You never know what potentially sensitive data could get transmitted (project names in folder paths/values sent to scripts, etc.).  You don't want to be liable for potentially NDA information leaking out...

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6Grk76rT7rnODi60qHjMMvZ8Km9WbaWJbSuhGeqJbjSuvFg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
-tony

[Marcus Ottosson]

Hey Eduardo,

That's quite useful, thanks. Since it's just you, I don't think you need to bother with a remote server and can instead host it on your own machine. You could then store either with your IP or a DNS to your machine directly in your tools, and have them "phone home" whenever an action takes place. It would be both free and fully under your control.

However, depending on your situation, users working from home probably won't mind, but the studios you work against might. Having anything communicate with the outside world from within an organisation is usually a big no-no, as it means confidential information could leak out and potentially end up in the wrong hands.

You could safe-guard your scripts and your communication, but you'd probably end up paying more than you bargain for (in terms of time versus value).

Best,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6Grk76rT7rnODi60qHjMMvZ8Km9WbaWJbSuhGeqJbjSuvFg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Marcus Ottosson]

+1 for Tony's comment. :)

--

Marcus Ottosson
konstr...@gmail.com

[Eduardo Grana]

Hello Guys,

Thanks again for all the feedback!

Let me clarify first that I'm very respectfull of NDA and all agreements i sign,

some of them even extended to the worlds yet to discover.

The agreements i have with the companies allow me to gather

this information, in case something break, i can fix it as soon as possible.

At the end, it's just me getting the information. I'll have to keep that info safe, of course.

Cheers!

Eduardo

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmODzjMtSC22skeZ_i%3DpJ6hHNY-kMQGN7RBWiDLdCp5d_sg%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Eduardo Graña
www.eduardograna.com.ar

[Marcus Ottosson]

Well, it isn't quite that easy. :) I'm also quite concerned that any facility would in fact let you do this, when you yourself doesn't seem aware of the risks.

Opening up any link to the outside world through your scripts makes the facility vulnerable to attacks; primarily though the port that you have them open for you. You can have your scripts encrypt messages that they send, if not, the data may be intercepted by third-parties on its way from the facility over to you. And depending on your encryption, it may still be intercepted; leaving you in violation of your NDA, even though you did nothing explicit to leak any information.

The second, possibly more dangerous risk is the open port itself. Your scripts are written in Python (I'm assuming) and Python can be modified at run-time; making any facility communication via any of your scripts possibly vulnerable to injection; such as Command Injection; meaning that an attacker gains as much control over the client computer as the Python interpreted was given when started (most likely the same level as the corresponding user).

Bottom line is, if you aren't confident in securing your communication across the internet, you could end up getting blamed for damage you never foresaw which may cost both you and your company(ies) quite a lot of both time and money.

Stay safe. :)

Best,

Marcus

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6GrkxbZczKdVKCGtjy%3DaY20tkReg0JXnYC4rrvGSXdY--xw%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

Marcus Ottosson
konstr...@gmail.com

[Marcus Ottosson]

I should mention that, the way this is usually handled is that all logging runs internal to the facility, where nothing is exposed to the internet. You are in quite a difficult (and unique) spot for distributed logging as you are working remotely, and your users too.

--

Marcus Ottosson
konstr...@gmail.com

[Eduardo Grana]

Hey Marcus, you are probably right.

I'll have to weight the benefits and the risks.

Maybe for now i'll test it with free scripts for now.

Sentry offered ssl encription of data, I'm wondering how could that work

if you do your own server.

Maybe I could do a local logging on the facilities, and log myself into.

Although somebody could break into that too.

I'll have to think this better.

Thanks again.

Eduardo

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CAFRtmOCrduVgBV7hcs2ZVwt8AjicCFNAmQY4HL3KxZG8Kgbj_A%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
Eduardo Graña
www.eduardograna.com.ar

[Nils Lerin]

Hi

How about setting up a MySQL server and create a create- or write-only user that your tools can use? Many hosting solutions provide MySQL out of the box. You don't risk unwanted personell from reading your data unless they get a hold of your admin-password or site admin details. Also you can use MySQL workbench or phpmyadmin to view your data without writing your own tools. The drawback? You have to learn sql to add data to the database or make queries. This can get a bit tricky if you want to make detailed queries. Also, in contrast to mongodb and elastic, it's not document based and entities must follow a strict schema. Sorry if you already know everything about SQL databases, without implying that I do. Just throwing it in here.

/Nils

To view this discussion on the web visit https://groups.google.com/d/msgid/python_inside_maya/CACt6Grm0Ong%3DRFc_sSTqUqP_ruVfqZWoye%2B46NeH0ucvhbZ0VA%40mail.gmail.com.