Is it possible to load a model from Google Drive?

[Nitin Pasumarthy]

Hi tfjs community,

To publicize our work internally and get feedback, I want to port our proprietary Python model to tfjs and host it as an internal static (standard HTML & JS without any build tools) website via Google App Script. As the model is confidential, I cannot save it in a public location and load it from there. A vague idea is,

• Download the model folder from drive as a Blob and save it in either localStorage or IndexedDB
• Load from either (if I'm able to save it in the right format) using loadLayersModel API

Any better ways to achieve this? 

https://js.tensorflow.org/api/latest/#io.browserFiles is always an option, but I'm seeing if I can give a seamless experience to users, so that they dont have to download the model to local disk from Drive and re-upload it the first time.

[Jason Mayes]

Hi there!

I am actually in the process of writing a codelab for hosting a custom TFJS model for deployment to Firebase Hosting which supports permissions on assets uploaded so that one must be logged in to view such files. This may be of interest to you.

Drive I think does not support relative URLs eg if I have model.json and model.bin in the same directory, the JS cant just go to /model.bin to grab .bin so may cause issues. Also CORS headers may be an issue for cross origin resources - I have not checked that for Drive before. 

Let me know if you would be interested in the firebase tutorial once it is ready.

Jason

--
You received this message because you are subscribed to the Google Groups "TensorFlow.js Discussion" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tfjs+uns...@tensorflow.org.
To view this discussion on the web visit https://groups.google.com/a/tensorflow.org/d/msgid/tfjs/e12f66fb-0de7-4f0e-8b12-d47d3b26156fn%40tensorflow.org.

--

•  Jason Mayes   •  Senior Developer Advocate, TensorFlow.js   •  Google   •   Follow my discoveries or contribute your own

This email may be confidential and privileged. If you received this communication by mistake, please don't forward it to anyone else, please erase all copies and attachments, and please let me know that it has gone to the wrong person.

[Nitin Pasumarthy]

Thanks Jason. Will give this a try :)

 

--

Nitin @ LinkedIn

[Jason Mayes]

Hi Nitin, good news, my codelab is now published if you would like to be one of the first to try it! It should walk you through how to host a model on Firebase hosting. Just be sure to look into setting up the authentication for your hosting to prevent anyone being able to view the deployed files shared below.

1) Firebase Auth: https://firebase.google.com/docs/auth/web/start

2) Codelab to use firebase hosting with TFJS model:  https://codelabs.developers.google.com/codelabs/tensorflowjs-firebase-hosting-model/#0  

Jason

[Nitin Pasumarthy]

Thanks Jason. Worked like a charm!

 

--

Nitin @ LinkedIn

[Nitin Pasumarthy]

Though firebase deployment and auth were simple enough to follow (thanks to your codelab), they apparently are not acceptable as firebase URL would be still be public in my case. In app script, the URL is only accessible from within the company. Any suggestions on the below approach on how to properly load the weights (.bin) file using Drive and app-script?

I was able to load the model.json from Drive and print the summary like so,

google.script.run.withSuccessHandler(modelJsonDataStr => {

    const modelJson = new File([modelJsonDataStr], "model.json", { type: "application/json" })

    const model = await tf.loadLayersModel(tf.io.browserFiles([modelJson]))

    model.summary();

}).withFailureHandler(onFailure).loadModelFromDrive();

and in the g-script file,

function loadModelFromDrive() {

  const f = DriveApp.getFileById("modeljson-file-id-from-drive")

  const modeljson = f.getAs("application/json").getDataAsString();

  return modeljson

}

_______________

But I'm unable to convert the Blob or Byte[] returned by g-script to a type that decodeWeights() of tf.loadLayersModel accepts. I tried something like,

google.script.run.withSuccessHandler(modelFiles => {

    const modelJson = new File([modelFiles[0]], "model.json", {type: "application/json"})

    const modelWeights = new File([new Blob(modelFiles[1])], "group1-shard1of1.bin")

    const model = tf.loadLayersModel(tf.io.browserFiles([modelJson, modelWeights])).then(model => {

        model.summary();

    })

}).withFailureHandler(onFailure).loadModelFromDrive();

and updated the loadModelFromDrive to return both these files,

function loadModelFromDrive() {

  const f = DriveApp.getFileById("1xboldJQ9rcaYF-Y6JxptrhI5fLFjoHoR")

  const modeljson = f.getAs("application/json").getDataAsString();

  const f2 = DriveApp.getFileById("1WGP-rc6gy0gRdGMFmTyl88LDjFeN7PHs")

  const weights = f.getBlob().getBytes()

  return [modeljson, weights]

}

tf.io.decodeWeights doesn't like the native JS Blob or Float32Array types. All the dtypes in my model are "float32" only.

Sorry for such a long message. I think this is another interesting way to deploy model, which sadly is the only possibility in my case :)

Thanks in advance for the help.

[Jason Mayes]

Let me also check in with the Firebase team about permissions that would make it work like we expected for only authed users in case I am missing something here.

[Nitin Pasumarthy]

Thanks Jason. I have a simple HTML file that loads the model, transforms the raw user input into model input, makes a prediction and prints the result. To secure it, I used firebase auth with Google sign-in to download the model only when users from my company sign-in. But as the URL of the model files are baked into the HTML (in my current design), anyone can curl them directly. I can think of a few solutions here,

1. Secure the model files with firebase storage with authentication rules
2. Design a server side session based app to only show model page after login
3. Figure out a way to translate Blob file from Google Drive to a format that decodeWeights() of tf.loadLayersModel accepts

I have permissions to use (3) today as Google App Script takes care of server based authorization. I feel this is possible and we are close in making this work; that too in a non-hacky way. 

(1) is easy but I have to get my entire app reviewed by the security team here, which is an involved process. (2) is a complicated solution IMO. 

--

Nitin @ LinkedIn

[Jason Mayes]

Understood. Thanks for the context and I shall still follow up with the Firebase team in case I have missed something about file level authentication. It is also good to know about your needs here in case this is something that can be addressed in the future too if not supported currently. I will certainly pass on the feedback. Thanks Nitin!

[Jason Mayes]

PS just a quick update, it was also suggested that  Google Cloud Storage with added IAM or Security rules could give you the granularity you need to have file based permissions - though I wonder if it keeps the relative URL nature you need for web stuff. I can also try and ask some GCP folk about that. It seems firebase hosting does not have file level permissions as you suspected.

[Nitin Pasumarthy]

Thanks Jason! GCP is a great idea and do you think we can overcome the relative URL nature required for model files with,

const model = await tf.loadLayersModel(tf.io.browserFiles([modelJson, modelWeights])) approach?

We can manually download modelJson and modelWeights files using fetch API or similar and create JS File objects. But I'm unable to make this work as shared in earlier emails, because I dont understand the internals of loadLayersModel

 

--

Nitin @ LinkedIn

[Jason Mayes]

I noticed that  https://js.tensorflow.org/api/latest/#io.browserFiles specified only the first bin file location (eg  uploadWeightsInput.files[0] ) . I just want to check it supports sharded bin files too if you have multiple bin files where each can be specified as you would need to generate a unique authed url for each for the GCP / Firebase security to work. If it can accept multiple files then we should be good. If not, we have same problem as before where it will assume relative URL for other bin files. Let me check on that and get back to you.

If that is supported however then you can generate the unique authed URL from GCP or Firebase for each file you need to load and set the URL to expire say a few minutes later so if someone posts the resulting URL publicly it would become invalid after that time at least.

If that few minutes of replay attack (should the unique url be shared by a malicious signed in user) is not acceptable then the only other way to do this would be to write your own thin server layer for auth eg using regular cookies etc which you send request to, and if request is from valid user, you then send back the file (eg traditional app engine style approach). However, more work involved to set that infrastructure up, though for that I do not see any replay attack in that narrow time window being possible.

In all of these cases proposed though it should be noted that if you had malicious authenticated users - that valid authenticated user could still grab the .json and .bin files that ultimately get downloaded to client side and save them locally and then share  manually without any URLs. The only way to prevent that is to never let the model leave the server and instead have some sort of websocket API or such to use with fairly realtime results depending on latency from client to server. Model security on client side is something on our minds but I believe to do it properly it needs browser level support to prevent a client side user from inspecting the network panel and grabbing files that came down in the page requests so may be a while before we see such a standard emerge.

J

[Ping Yu]

Hi Nitin

We can add a weightUrlTranslationFunc to the loadOptions of the loader API, in this function you will receive the weight file name as input, you need to generate the authenticated URL as output.

I will create a PR and cc you on that.

Ping

[Nitin Pasumarthy]

Jason,

Looks like browserFiles API accepts an array of >=0 binary files in order as per their documentation - Optionally, One or more binary files containing the binary weights. These files must have names that match the paths in the weightsManifest contained by the aforementioned JSON file, or errors will be thrown during loading.   

Signed URLs is a great idea for model files. I can get this reviewed and give it a try. The site is secured by firebase auth but for authorization of the site itself, I need to think of a session based scheme to only load the web page if successful. It is safe to assume authorized LinkedIn employees would not perform malicious activities. Yeah, browser model security is a hard and interesting problem. Thanks again for looking into this and making tfjs accessible to everyone.

Ping,

That would be great. Thank you :)

 

--

Nitin @ LinkedIn

[Jason Mayes]

Awesome, thanks for the update. Let us know how it goes :-)

[Nitin Pasumarthy]

Was able to finally make this work! And guess what the issue was? Copy - Paste 🤣

Pasted the snippet where this happened. Wrote a short article on this approach - Securely deploy your Tensorflow JS model via Google App Script.

function loadModelFromDrive() {

  const f = DriveApp.getFileById("1xboldJQ9rcaYF-Y6JxptrhI5fLFjoHoR")

  const modeljson = f.getAs("application/json").getDataAsString();

  const f2 = DriveApp.getFileById("1WGP-rc6gy0gRdGMFmTyl88LDjFeN7PHs")

  const weights = f.getBlob().getBytes()  // this should have been f2.getBlob().getBytes(). Had I used Google's clasp CLI to develop it my IDE, f2 would have been highlighted as an unused variable 😅 

  return [modeljson, weights]

}

Thanks a lot Jason and Ping for helping me out :)