Logging all notebook activity sent to the kernel

2,394 views
Skip to first unread message

Hussein Elgridly

unread,
Jul 16, 2017, 11:51:27 AM7/16/17
to Project Jupyter
Hi folks,

We're standing up a Jupyter (Python right now, will get R later) stack and everything is going well so far. The wrinkle is that users will be talking to federal data and our auditors are getting nervous; they want to know what people are doing.

If my understanding of the Jupyter architecture is correct, the notebook server ultimately bundles up the cell contents and sends them to the backing kernel through 0MQ. Can we hook into this somewhere and write all such messages to a log?

If there is a config flag that already does this I am ecstatic; if you can point me to "make a pull request somewhere around here" that'd work too.

Thanks,
-Hussein

Thomas Kluyver

unread,
Jul 19, 2017, 7:44:53 AM7/19/17
to Project Jupyter
Hi Hussein,

There isn't a config option for it, but your understanding is correct, and it should be possible to log what is being executed. You can see the messaging protocol description here:
https://jupyter-client.readthedocs.io/en/latest/messaging.html

The simplest way to do this in some regards is to connect another client to the kernel, which should broadcast each code cell it receives as an execute_input message. However, this depends on the kernel actually doing so, and you'd still need to know when a new kernel was starting up.

Perhaps a more robust way would be to insert a bit of extra code into the notebook server. The execute_request message is sent over a websocket by the browser, received by the server, reserialised and sent over ZMQ to the kernel. The code doing that is here:
https://github.com/jupyter/notebook/blob/5.0.0/notebook/services/kernels/handlers.py#L278

That assumes that your users only have Jupyter access through the notebook; if they can use other Jupyter frontends, like the Qt console, then we'd need to get the messages out somewhere else.

Best wishes,
Thomas

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+unsubscribe@googlegroups.com.
To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/44e0bcc5-fa2c-4d52-8b9d-00f21ba0bb19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Hussein Elgridly

unread,
Jul 19, 2017, 9:17:58 PM7/19/17
to jup...@googlegroups.com
Hi Thomas,

This is great. As you suggest I suspect the best thing for us to do here would be to add some logging code in the on_message function you linked. Our users are limited to the notebooks UI only so that should be sufficient for our needs.

I may resurrect this thread in a couple of months if we continue down this path to ask if you'd like a PR once we've got something working.

Thank you so much!
-Hussein

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/sLKCCBwlKEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+unsubscribe@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

Thomas Kluyver

unread,
Jul 20, 2017, 3:20:20 AM7/20/17
to Project Jupyter
Hi Hussein,

Please do drop us a note when you've got something working - even if we decide not to add it as an option in the notebook, it could well be a useful example for people who want to do similar things.

Best wishes,
Thomas

To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAJHWZXPaKReooRnU%3DFGkgTEk4Z5TJ46WMa609WU%3DCeNrPBMGLQ%40mail.gmail.com.

Jason Grout

unread,
Jul 20, 2017, 10:19:51 PM7/20/17
to Project Jupyter
A long time ago, we discussed making it easy to plug in/configure transformers for any message that came through. That would make it easy to log, but it would also make it easy to, for example, inspect messages for large images, strip them out and put them into a database, leaving a link in the message, etc. It would be really nice to have some easy way to configure such things in the notebook server.

Thanks,

Jason


To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/44e0bcc5-fa2c-4d52-8b9d-00f21ba0bb19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/sLKCCBwlKEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAJHWZXPaKReooRnU%3DFGkgTEk4Z5TJ46WMa609WU%3DCeNrPBMGLQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jupyter+u...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/CAOvn4qijYgtxSYeDJxWDv7E9VWL7k29H5mH8%2BWnYK%2Bw-Ob8VBA%40mail.gmail.com.

Leo Gallucci

unread,
Mar 27, 2020, 12:52:12 PM3/27/20
to Project Jupyter
How did you solve this Hussein?
To unsubscribe from this group and stop receiving emails from it, send an email to jup...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/jupyter/44e0bcc5-fa2c-4d52-8b9d-00f21ba0bb19%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to a topic in the Google Groups "Project Jupyter" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/jupyter/sLKCCBwlKEc/unsubscribe.
To unsubscribe from this group and all its topics, send an email to jup...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jup...@googlegroups.com.

To post to this group, send email to jup...@googlegroups.com.

--
You received this message because you are subscribed to the Google Groups "Project Jupyter" group.
To unsubscribe from this group and stop receiving emails from it, send an email to jup...@googlegroups.com.

Hussein Elgridly

unread,
Mar 27, 2020, 1:34:14 PM3/27/20
to Project Jupyter
We ended up satisfying our auditors via other means -- specifically by monitoring network usage and alerting on significant egress (and in some cases limiting data transfer to only within a VPC).
Reply all
Reply to author
Forward
0 new messages