Triggering Python Cloud Functions via the Firestore Emulator

[Brad Dwyer]

I know this isn't currently an officially supported feature (only Node functions can get linked to your Firebase project and get triggered by the Firestore Emulator). But I wanted to lay out my use-case and see if anyone has any suggestions for how I could improve the workflow because it's becoming pretty cumbersome.

Most of our Cloud Functions are written in Node, but one in particular needs access to functions in scikit-image (and in the future we will need parity with some other image processing and analysis code only available in Python packages).

We've been able to setup this python Cloud Function to trigger via Firestore events manually via the Google Cloud Functions. But it's completely disjointed from the rest of our app. Deploys are handled separately. And our local environment (with the Firestore Emulator) is unable to trigger the function.

So in order to test any changes to this function we have to deploy it and test via triggering a production Firestore event. Not only is this problematic because we're testing in production currently but the change->deploy->trigger loop is frustratingly slow.

What I'm looking for is a way to run and trigger this locally. I'm considering wrapping it in a Node Firebase Function that calls out to the python interpreter. But that seems incredibly hacky (and I'm not sure if I'd be able to effectively link the Python packages with the interpreter if they're deployed with the node function).

Any other suggestions for how to improve this workflow? Or any hacky ways to get this to work in local development mode?

PS - also looking for a way to mock Cloud Storage (and its associated triggers) locally as well but that's been causing me less pain than this Python thing.

[Sam Stern]

Hey Brad,

Thanks for all the feedback, that's a very specific but valid use case and I can see how it's a pain to test.  As you mentioned there's no support for Python in the Firebase emulators and it's not likely to happen in the future because most of the tricks we use to achieve local emulation are fairly Node-specific.  This page from the Google Cloud docs has some good information on how to write unit or integration tests for Cloud Functions written in Python:
https://cloud.google.com/functions/docs/testing/test-background#storage-triggered_functions

I linked specifically to the section on Storage-triggered functions since that should be very similar to Firestore triggered functions.  I would suggest running the function a few times in production (as you've done) with different inputs so you can create a realistic set of mocks that you can use as unit tests for the function in the future.  If your function also writes back to Cloud Firestore, you can set the "FIRESTORE_EMULATOR_HOST=locahost:8080" environment variable to point at the Cloud Firestore emulator so that your outgoing writes don't affect production.

Hope that's helpful!

- Sam

--
You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/9012d80d-7757-4858-a9d7-5a7843667c04%40googlegroups.com.

[John Carter]

Hi Sam, are you able to explain a little further as to what Node-specific tricks are used for this?  Given the release of the python functions framework ( https://github.com/GoogleCloudPlatform/functions-framework-python ) I was digging into how Node registers emulated functions with the firestore emulator to see what would be needed to do this from python.

(apologies, my background is Python, my Node knowledge is limited).

From what I can see FunctionsEmulator.addFirestoreTrigger does a JSON PUT of the bundle to http://localhost:${firestorePort}/emulator/v1/projects/${projectId}/triggers/${definition.name}

eg http://localhost:8080/emulator/v1/projects/myproject/triggers/my_function

where the bundle (for project id myproject) is

{'eventTrigger': {'resource': 'projects/myproject/databases/(default)/documents/messages/{documentId}',
  'eventType': 'providers/cloud.firestore/eventTypes/document.create',
  'service': 'firestore.googleapis.com'}}

This makes the firestore emulator do a HTTP POST to port 5001 on firestore document create.

I've tried to imitate this by:

1) running the java emulator manually with the same argument as per firebase emulators:start --only  functions,firestore:

java -Duser.language=en -jar /home/johnc/.cache/firebase/emulators/cloud-firestore-emulator-v1.10.2.jar --host localhost --port 8080 --rules /home/johnc/Projects/myproject/firestore.rules --functions_emulator localhost:5001 --webchannel_port 8081

2) then running a local python function with:

functions-framework --signature_type event --debug true --target hello_event --port=5001

3) setting up the trigger with a PUT as per above (which returns 200).

But I'm obviously missing something, since functions-framework doesn't receive any HTTP requests when I then do a firestore write to the emulator (eg as per the python quickstart https://firebase.google.com/docs/firestore/quickstart ).

Any suggestions of what I might be missing?

[Sam Stern]

Hey John,

Sorry for the super slow reply here, I was on vacation all of last week.  I have to say I am impressed by how much you've dug into this!

Here's what I mean by cool JavaScript tricks. When you use the Firebase CLI to emulate a Cloud Function we launch a node subprocess that runs this code we call the runtime:

https://github.com/firebase/firebase-tools/blob/master/src/emulator/functionsEmulatorRuntime.ts

Because it's JavaScript we can do things like monkey-patching to make admin.firestore() calls automatically point at the emulator rather than production, or detect when you try to make an external HTTP call and warn you.  We're trying to optimize for a really smooth local development experience, where functions-framework is trying to optimize for a local experience that is most similar to production.

So that said ... what you tried to do looks totally sane and I'm not sure why it doesn't work.  Can you open a new GitHub issue on functions-framework-python and cc me?  I can chime in about the Firebase parts and they can comment about what the functions framework can/can't do.

- Sam

--

You received this message because you are subscribed to the Google Groups "Firebase Google Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to firebase-tal...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/firebase-talk/8d03bc3f-c985-47a8-90bc-8ed63a78752b%40googlegroups.com.