Simple solution needed for http access to API

652 views
Skip to first unread message

Raffael Cavaliere

unread,
Feb 21, 2017, 9:30:48 PM2/21/17
to shotgun-dev
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 

Peter Shinners

unread,
Feb 21, 2017, 10:03:17 PM2/21/17
to shotg...@shotgunsoftware.com
We have an internal web app that queries directly to the Shotgun server with ajax. I just looked at the code used to talk to the server. It's uglier and bulkier than I remembered, which makes it hard to share in this reply. It has allowed us to write code that runs on a web page like this,
window.Shotgun.GetThumbnail = function(entityType, guid, callback) {
  return window.Shotgun.Read(entityType, ['id', 'image'], 
      [{'path': 'sg_assetmgt_id', 'relation': 'is', 'values': [guid]}], 
      function(result) {
          callback(result[0].image);
  });
};


On 02/21/2017 06:30 PM, Raffael Cavaliere wrote:
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 
--
--
To unsubscribe, email shotgun-dev...@shotgunsoftware.com
For other options go to http://groups.google.com/a/shotgunsoftware.com/group/shotgun-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev...@shotgunsoftware.com.

Kirill Kovalevskiy

unread,
Feb 22, 2017, 3:38:41 AM2/22/17
to shotg...@shotgunsoftware.com
I can look at the shotgun-api3 Python source and see how requests are build. Then you can construct your own request from JavaScript or whatever language you need. However, It would be nice to have an official JavaScript API. 

I attached a bash script example of "read" request that list specified shotgun entity. Change your site API key and its name in the script and then run it like this

./sg_req_examle.sh Project


Kirill

On Tue, Feb 21, 2017 at 6:30 PM Raffael Cavaliere <raffael....@telmatik.com> wrote:
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 

--
sg_req_example.sh

Rob Aitchison

unread,
Feb 22, 2017, 8:56:46 AM2/22/17
to shotg...@shotgunsoftware.com
What kind of statistics are you trying to display?

On Wed, Feb 22, 2017 at 3:38 AM, Kirill Kovalevskiy <koval...@gmail.com> wrote:
I can look at the shotgun-api3 Python source and see how requests are build. Then you can construct your own request from JavaScript or whatever language you need. However, It would be nice to have an official JavaScript API. 

I attached a bash script example of "read" request that list specified shotgun entity. Change your site API key and its name in the script and then run it like this

./sg_req_examle.sh Project


Kirill


On Tue, Feb 21, 2017 at 6:30 PM Raffael Cavaliere <raffael.cavaliere@telmatik.com> wrote:
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 

--
--

For other options go to http://groups.google.com/a/shotgunsoftware.com/group/shotgun-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev+unsubscribe@shotgunsoftware.com.

--
--
To unsubscribe, email shotgun-dev+unsubscribe@shotgunsoftware.com

For other options go to http://groups.google.com/a/shotgunsoftware.com/group/shotgun-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev+unsubscribe@shotgunsoftware.com.

Matt Daw

unread,
Feb 22, 2017, 9:04:44 AM2/22/17
to shotg...@shotgunsoftware.com
We're thinking about our next steps on APIs now, and an official pure HTTP API is high up on our list. I'm aiming to put a general platform roadmap update together for Siggraph...

In the meantime, it is possible to use the underlying JSON transport that the Python API uses (as Kirill suggests). We can't officially support that, and we reserve the right to make breaking changes since it's currently an implementation detail rather than a public API. But in practice we won't be making breaking changes regularly.

Matt

On Wed, Feb 22, 2017 at 3:38 AM, Kirill Kovalevskiy <koval...@gmail.com> wrote:
I can look at the shotgun-api3 Python source and see how requests are build. Then you can construct your own request from JavaScript or whatever language you need. However, It would be nice to have an official JavaScript API. 

I attached a bash script example of "read" request that list specified shotgun entity. Change your site API key and its name in the script and then run it like this

./sg_req_examle.sh Project


Kirill

On Tue, Feb 21, 2017 at 6:30 PM Raffael Cavaliere <raffael.cavaliere@telmatik.com> wrote:
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 

--
--
---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev+unsubscribe@shotgunsoftware.com.

--
--
To unsubscribe, email shotgun-dev+unsubscribe@shotgunsoftware.com

Michael Oliver

unread,
Feb 22, 2017, 12:40:59 PM2/22/17
to shotgun-dev
Another option that I used to hack something together is to use Flask with Chart.JS.  https://pythonspot.com/flask-and-great-looking-charts-using-chart-js/

Then you can use the SG python API and make pretty charts / show statistics in a html webpage.

Raffael Cavaliere

unread,
Jun 27, 2017, 3:49:53 AM6/27/17
to shotgun-dev
Thanks to all for your comments. great community here.
I finally decided to implement the whole thing as a REST service using Django REST Framework. I am in the process of exposing all the objects using the find method. Then I am loading all this beautiful data in MS PowerBI. Pretty cool. I was thinking of hosting the django server on the cloud somewhere (bitnami apps).

Stéphane Deverly

unread,
Jun 27, 2017, 4:14:49 AM6/27/17
to shotg...@shotgunsoftware.com
On 27 Jun 2017, at 09:49, Raffael Cavaliere <raffaelc...@gmail.com> wrote:

Thanks to all for your comments.  great community here.
I finally decided to implement the whole thing as a REST service using Django REST Framework.  I am in the process of exposing all the objects using the find method.  Then I am loading all this beautiful data in MS PowerBI.  Pretty cool.  I was thinking of hosting the django server on the cloud somewhere (bitnami apps).

Hello,
if you plan to host your Django server somewhere, Heroku can be a good choice.
It supports Django apps out of the Box, is very easy to use, use git for releases, and even provide some free instances for your tests.
Just my 2 cents!


Owen Nelson

unread,
Jun 27, 2017, 12:38:14 PM6/27/17
to shotgun-dev
We've been doing exactly this (Django REST Framework wrapper web APIs) for a few years at Laika. It works well enough, but recently I've been looking at ways to increase throughput. Our shotgun queries are often measured in seconds rather than milliseconds, and as such we need a nicely threaded or async solution to keep things moving through while we wait for shotgun to respond.

I just completed a proof of concept for our next generation of wrapper APIs which uses Jython to execute shotgun queries in threads (via Scala futures), all from a Play framework web service. Maybe this will give you some ideas for how to address scale as your need increases. In my experience, when you start exposing this data in more convenient ways, the appetite increases! 

Raffael Cavaliere

unread,
Jun 27, 2017, 1:36:56 PM6/27/17
to shotgun-dev
Nice !

This is why I posted this yesterday.  I am currently using a small test project database, so everything is pretty fast, plus I am not messing around in PowerBI with multiple transformations and cross-queries, but once this will be in the hands of the management people, who knows what they will do...  The thing is, in PowerBI, the user has nothing to do while he waits for the data to be collected... Once you hit the refresh button, the only purpose of the application is to retrieve the data and refresh the report....

I was thinking more of using some kind of caching mechanism inside the wrapper, i.e. creating a model for all shotgun objects and querying the models instead of shotgun.  But this would imply a lot more storage on my cloud django server.

Owen Nelson

unread,
Jun 27, 2017, 3:13:29 PM6/27/17
to shotg...@shotgunsoftware.com
For endpoints with pre-determined queries behind them (no filtering exposed to the caller), which are also considered non-volatile stuff (for us we have things that change infrequently in shotgun, such as crew, details about pipeline step templates such as colors associated with types of tasks, etc) we use memcache via Django's lower level cache API with different expiry times per endpoint.

For endpoints that take a really long time to load, we do something called cache priming.
A good example of this is reading the work schedule/non-working days which can take over 45 seconds, we run a cron job that invalidates the cache and re-caches the latest value before the expiry time, meaning for most users, this data will load in ~10ms instead of 45 seconds.

Django has a lot of options for cache backends, memcache is just one. You could do this with files on disk, redis, etc. Whatever is convenient. I'd avoid using the in-memory cache for prod though. It's basically an intentional memory leak if you don't have a good prune/eject policy in place.


Owen Nelson

--

Christopher Spears

unread,
Jan 2, 2018, 7:37:17 PM1/2/18
to shotgun-dev
I know this was posted a while ago, but I was wondering if there has been any progress on making a HTTP API.


On Wednesday, February 22, 2017 at 6:04:44 AM UTC-8, Matt Daw wrote:
We're thinking about our next steps on APIs now, and an official pure HTTP API is high up on our list. I'm aiming to put a general platform roadmap update together for Siggraph...

In the meantime, it is possible to use the underlying JSON transport that the Python API uses (as Kirill suggests). We can't officially support that, and we reserve the right to make breaking changes since it's currently an implementation detail rather than a public API. But in practice we won't be making breaking changes regularly.

Matt
On Wed, Feb 22, 2017 at 3:38 AM, Kirill Kovalevskiy <koval...@gmail.com> wrote:
I can look at the shotgun-api3 Python source and see how requests are build. Then you can construct your own request from JavaScript or whatever language you need. However, It would be nice to have an official JavaScript API. 

I attached a bash script example of "read" request that list specified shotgun entity. Change your site API key and its name in the script and then run it like this

./sg_req_examle.sh Project


Kirill

On Tue, Feb 21, 2017 at 6:30 PM Raffael Cavaliere <raffael....@telmatik.com> wrote:
I would like to access shotgun project data from a simple html page, mostly displaying statistics.  I was wondering if anyone had developed a bridge from the python API to a REST or JSON http-oriented service ? 

--
--
To unsubscribe, email shotgun-dev...@shotgunsoftware.com
---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev...@shotgunsoftware.com.

--
--
To unsubscribe, email shotgun-dev...@shotgunsoftware.com

Lee Hollingworth

unread,
Jan 2, 2018, 8:19:57 PM1/2/18
to shotgun-dev
Hi Christopher,

We're actively working on a REST API for Shotgun and should have that available soon. You can get more details by viewing our recent webinar where we shared our roadmap plans with our clients. The details on the REST API start at about the 32 minute mark, but I encourage you to watch the whole video.

Password: q4webinar

Let me know if you want more info.

Cheers,
Lee

Christopher Spears

unread,
Jan 2, 2018, 8:49:00 PM1/2/18
to shotgun-dev
Awesome!  I'll  take a look.

Kyle Hayes

unread,
Jan 3, 2018, 10:51:00 AM1/3/18
to shotg...@shotgunsoftware.com
I'm glad to see this in the roadmap especially with support for webhooks.

I'm just now seeing this thread overall and am curious to ask of folks who are using Django REST to wrap the current API as to why you went with Django? Was it a familiarity choice or was it something that interested you specifically about DRF? Did you also look at options like Eve or Flask?

Cheers,
Kyle

--
--
To unsubscribe, email shotgun-dev+unsubscribe@shotgunsoftware.com

For other options go to http://groups.google.com/a/shotgunsoftware.com/group/shotgun-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev+unsubscribe@shotgunsoftware.com.

Rob Aitchison

unread,
Jan 4, 2018, 11:29:00 AM1/4/18
to shotgun-dev
Thanks for sharing this webinar! Is there a beta program for REST / webhooks early access?
Cheers!

Christopher Spears

unread,
Jan 4, 2018, 1:38:54 PM1/4/18
to shotgun-dev
I would be interested in a beta program as well.

Thanks!

Lee Hollingworth

unread,
Jan 5, 2018, 12:33:58 PM1/5/18
to shotgun-dev
There is no beta yet. We will be sure to announce it here once there is.
Cheers,
Lee

Owen Nelson

unread,
Jan 5, 2018, 2:20:40 PM1/5/18
to shotg...@shotgunsoftware.com
I'm just now seeing this thread overall and am curious to ask of folks who are using Django REST to wrap the current API as to why you went with Django? Was it a familiarity choice or was it something that interested you specifically about DRF? 
> Did you also look at options like Eve or Flask?

I can speak to this a little bit. I have no experience with Eve, but have spent a bunch of time with both Django and Flask.

Part of this may just be down to personal preference, but I've found Django to provide a nice scaffolding to which the surrounding ecosystem can attach modular pieces of functionality (via "apps"). 
This is a feature we've exploited a great deal in delivering separate packages containing "apps" that can provide View Class mixins which provide convenience methods for running "canned" shotgun queries, adding properties that give access to api client connections which are configured for you via Django settings system, caching policies which can leverage our memcache pool in production, and a local in-process cache during development. We use LDAP to handle auth, and Django's pluggable auth backends made it trivial to integrate this with Django REST Framework such that we could accept user credentials from a single page web application and convert that into a token which could then be used to identify a user and manage access privileges on a per-endpoint basis.

While REST Framework provides the most value add when working with the Django ORM and an application-local database, rather than a third party api client, we still got a lot of value from the rest of the features mentioned above.
Had we set out to solve a less specific problem during early development of our REST service, and a little more developer resources to throw at the project, we might have attempted to implement a custom Django DB Backend that used the shotgun client api instead of a regular DB driver object, which would have opened up a bunch of interesting opportunities by putting us "back on the rails" meaning we could leverage more DRF features for free (automatic filter building via query string parsing, pagination, et al)

All of this is possible with Flask as well, but you have to strap some extra pieces together to make it happen, especially when it comes to configuration switching, identity management, and trading out different caching strategies depending on the deployment/environment. I also found Flask's answer to modular app design (blueprints) to be very lacking. In our service, we have a series of apps, each representing different problem spaces. They all use a mix of shotgun data and app-local database data, and using blueprints for this sort of segmentation gets ugly real quick. Bottom line, I found Django's project structure and layout to be the most collaborative.

Generally speaking, I'll reach for Flask if the project will fit in the palm of my hand, otherwise I'm always going to reach for Django (assuming I _need_ to use Python).
Whenever the REST interface from Shotgun arrives, I might rebuild the shotgun specific endpoints in Scala, running all the shotgun queries async -- this should give me some better response times if I can run multiple shotgun queries in parallel, then compose the result in my response...

Hope that helps explain some of the rationale!



Owen Nelson

Kyle Hayes

unread,
Jan 5, 2018, 4:41:58 PM1/5/18
to shotg...@shotgunsoftware.com
Thank you, Owen—this is very interesting. I've not done any heavy development in Django for about 5 years. I do a lot of client/server JavaScript these days but have been using Flask a lot as well for Python primarily due to the projects consisting mostly of APIs. However, some of our projects are user interface + API but we would typically keep those as separate projects.

I'm most intrigued by your use of Django's apps. I didn't fully understand the power of that paradigm in Django—I was also pretty new to Python at the time.

Thank you for sharing!

Kyle Hayes
Lead Sr. Software Engineer, Walt Disney Animation Studios

On Fri, Jan 5, 2018 at 11:20 AM, Owen Nelson <one...@gmail.com> wrote:
I'm just now seeing this thread overall and am curious to ask of folks who are using Django REST to wrap the current API as to why you went with Django? Was it a familiarity choice or was it something that interested you specifically about DRF? 
> Did you also look at options like Eve or Flask?

I can speak to this a little bit. I have no experience with Eve, but have spent a bunch of time with both Django and Flask.

Part of this may just be down to personal preference, but I've found Django to provide a nice scaffolding to which the surrounding ecosystem can attach modular pieces of functionality (via "apps"). 
This is a feature we've exploited a great deal in delivering separate packages containing "apps" that can provide View Class mixins which provide convenience methods for running "canned" shotgun queries, adding properties that give access to api client connections which are configured for you via Django settings system, caching policies which can leverage our memcache pool in production, and a local in-process cache during development. We use LDAP to handle auth, and Django's pluggable auth backends made it trivial to integrate this with Django REST Framework such that we could accept user credentials from a single page web application and convert that into a token which could then be used to identify a user and manage access privileges on a per-endpoint basis.

While REST Framework provides the most value add when working with the Django ORM and an application-local database, rather than a third party api client, we still got a lot of value from the rest of the features mentioned above.
Had we set out to solve a less specific problem during early development of our REST service, and a little more developer resources to throw at the project, we might have attempted to implement a custom Django DB Backend that used the shotgun client api instead of a regular DB driver object, which would have opened up a bunch of interesting opportunities by putting us "back on the rails" meaning we could leverage more DRF features for free (automatic filter building via query string parsing, pagination, et al)

All of this is possible with Flask as well, but you have to strap some extra pieces together to make it happen, especially when it comes to configuration switching, identity management, and trading out different caching strategies depending on the deployment/environment. I also found Flask's answer to modular app design (blueprints) to be very lacking. In our service, we have a series of apps, each representing different problem spaces. They all use a mix of shotgun data and app-local database data, and using blueprints for this sort of segmentation gets ugly real quick. Bottom line, I found Django's project structure and layout to be the most collaborative.

Generally speaking, I'll reach for Flask if the project will fit in the palm of my hand, otherwise I'm always going to reach for Django (assuming I _need_ to use Python).
Whenever the REST interface from Shotgun arrives, I might rebuild the shotgun specific endpoints in Scala, running all the shotgun queries async -- this should give me some better response times if I can run multiple shotgun queries in parallel, then compose the result in my response...

Hope that helps explain some of the rationale!



Owen Nelson

Julien Brasseur

unread,
Mar 12, 2018, 10:42:35 AM3/12/18
to shotgun-dev, raffael....@telmatik.com
I'm really excited for this REST api, i need it for a project. Keep us updated @Lee !
Message has been deleted

Andrew Watson

unread,
Mar 26, 2018, 1:44:37 PM3/26/18
to shotgun-dev, raffael....@telmatik.com, julien.b...@gmail.com
Is there an estimate for when this API will enter beta?

Rob

unread,
Mar 27, 2018, 10:29:06 AM3/27/18
to shotg...@shotgunsoftware.com
Hi all! I’ve been eagerly waiting to be able to make this announcement. With Shotgun 7.8.0 the REST API is available as a beta. You can email sup...@shotgunsoftware.com to ask for access. 

More info here:
I’m excited to get feedback from everybody to help shape the API for general release. 

-r

On Tue, Mar 27, 2018 at 8:14 AM Andrew Watson <pyr...@gmail.com> wrote:
Is there an estimate for when this API will be in beta?
Message has been deleted

Lukáš Bílek

unread,
Mar 30, 2018, 9:00:38 AM3/30/18
to shotg...@shotgunsoftware.com
Maybe this is not REST API but what about to try GraphQL?

On Mon, Mar 26, 2018 at 6:42 PM, Andrew Watson <pyr...@gmail.com> wrote:
Is there an estimate for when this API will be in beta?

On Wednesday, January 3, 2018 at 1:19:57 AM UTC, Lee Hollingworth wrote:

--
--
To unsubscribe, email shotgun-dev+unsubscribe@shotgunsoftware.com

For other options go to http://groups.google.com/a/shotgunsoftware.com/group/shotgun-dev?hl=en

---
You received this message because you are subscribed to the Google Groups "shotgun-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to shotgun-dev+unsubscribe@shotgunsoftware.com.



--
Lukas Bilek

Pipeline TD
Giant Animation

Rob Blau

unread,
Mar 30, 2018, 12:14:12 PM3/30/18
to Thomas Eskénazi, shotgun-dev
Hi Thomas,

The REST API will support CORS.  We are planning on adding a security setting that will give control over the CORS header to a site admin, but that is not yet implemented.


-r

On Mar 30, 2018, at 6:03 AM, Thomas Eskénazi <e...@mikrosimage.eu> wrote:


Hi Rob,

Thanks for the update !

Can you telle us if the REST API will support CORS ?

I was not able to find this information elsewhere.

Best Regards,

Thomas

Rob Blau

unread,
Mar 30, 2018, 12:24:16 PM3/30/18
to shotgun-dev
Hi Lukáš,

We are definitely thinking about GraphQL, and there may be a public GraphQL interface at some point in the future.  It is an compelling way of being able to query more complex data from Shotgun with fewer round trips, but we first want to get the REST API out there, complete, and make sure everybody is happy with it before we move on to another API.

-r
Reply all
Reply to author
Forward
0 new messages