What are the prepended headers and isolation primitives in NaClSendDatagram?

[George Robinson]

Greetings!

I have two questions about the NaClSendDatagram and NaClReceiveDatagram functions in Native Client. 

First, I've constructed a demo application where the untrusted code can communicate with the trusted code (and vice-versa) via the NaClSendDatagram and NaClReceiveDatagram functions. In our experimentation with this demo we've noticed that Native Client prepends a 16 byte header when communicating from untrusted code to trusted code, but not vice versa.

For example, if we send the int 0x03 via NaClSendDatagram in the untrusted code we can hexdump the 4 bytes in the I/O vector: 0x03 0x00 0x00 0x00:

int
main(int argc, char **argv)
{
        int ret;

        NaClIOVec iov;
       
        /*
         * The NaClMessageHeader is a struct which has these attributes:
         *
         * NaClIOVec*   iov
         * size_t               iov_length
         * NaClHandle*  handles
         * size_t               handle_count
         * int                  flags
         */
        NaClMessageHeader hdr = {&iov, 1, NULL, 0, 0};

        struct mystruct m = {0x3};

        assert(argc == 2);
        imcfd = atoi(argv[1]);

        iov.base = &m;
        iov.length = sizeof(m);
       
        ret = NaClSendDatagram(imcfd, &hdr, 0);
        assert(ret != -1);
        hexdump(iov.base, iov.length);

        return 0;
}

But, when we receive this datagram in the trusted code:

ret = NaClReceiveDatagram(handlepair[1], &hdr, 0);
CHECK(ret != -1);
mystruct = (struct mystruct *) ((char *) iov.base + 16);
printf("Received in trusted code:");
hexdump(iov.base, ret);

we read: 0x01 0xffffffde 0xffffffc0 0xffffffd3 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x03 0x00 0x00 0x00 such there is 16 bytes of header prepended to the IO vector. 

I've traced NaClSendDatagram down to the imc_sendmsg system call number 63, where it's address is loaded in the service runtime and used as a function pointer: 

#define NACL_SYSCALL_ADDR(syscall_number) \
 (NACL_SYSCALL_START_ADDR + (syscall_number << NACL_SYSCALL_BLOCK_SHIFT))

Despite this, I still can't figure out where this header is prepended and what it means. If anyone knows where I can find the code for this that would be super!

Second question is whether the IO vector is copied between untrusted code and trusted code, and vice versa? I can see some references to this in /src/trusted/service_runtime/sys_imc.c for NaClSysImcSendmsg and NaClSysImcReceivemsg? Is it safe to presume that this is the imc code that's invoked for both trusted and untrusted code on all platforms, including within a NaCl sandbox?

Thanks!

George

[Mark Seaborn]

If you want to use IMC to communicate between trusted and untrusted code, NaClSend/ReceiveDatagram() are the wrong functions to use.

Instead, you'd want to use:

 * In untrusted code:  the imc_sendmsg()/recvmsg() syscalls, using FD numbers.

 * In trusted code:  the SendMsg()/RecvMsg() methods of a NaClDesc, using NaClDesc* pointers.

(It's unfortunate that there isn't a consistent interface to use in both trusted and untrusted code.)

NaClSend/ReceiveDatagram() are low-level layers of IMC's implementation which are only used in trusted code.  The fact that there is a definition for them for untrusted code is quite misleading, because those definitions aren't useful and shouldn't be being used.

I could answer your question further and explain how IMC works internally if you really want, but what's the context of what you're trying to do here?  Do you just want to communicate between trusted and untrusted code within a host-OS process, or do you want cross-process communication as well?

Cheers,

Mark

--
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to native-client-di...@googlegroups.com.
To post to this group, send email to native-cli...@googlegroups.com.
Visit this group at http://groups.google.com/group/native-client-discuss.
For more options, visit https://groups.google.com/d/optout.

[George Robinson]

I see!

Our goal is to send a message from trusted code to untrusted code, and then for the trusted code to send a message back to trusted code after doing whatever it needed to do with the message - like a request-response cycle. - all within a SINGLE host-os process. We have no need for IPC. 

We had at first used imc_sendmsg() in the untrusted code and then SendMsg() in the trusted code - but we were unable to initiate communication from the trusted code to the untrusted code. Rather, we found that we had to call imc_recvmsg() in the untrusted code, which invoked a callback in the trusted code (like in the custom_desc test) upon which the trusted code could then write into untrusted memory. Is this possible? As a result of these difficulties we found we were able to initiate communication from the trusted code to the untrusted code via NaClSendDatagram and NaClReceiveDatagram.

Would you be able to explain what you mean by "because those definitions aren't useful and shouldn't be being used"? Is it that these definitions are unsafe?

If it wouldn't consume too much of your time, a brief tour of the internals of IMC would be great! We're currently reading through the Native Client source to understand how it works - however we've only just reached the system call invocation in the trampoline routine for untrusted code (i.e. how the system call number calculates an offset in service runtime which is called like a function pointer etc).

 

Many thanks!

To unsubscribe from this group and stop receiving emails from it, send an email to native-client-discuss+unsub...@googlegroups.com.

[George Robinson]

Hi Mark,

I had the idea to trace these two calls, NaClSendDatagram in trusted code and then NaClSendDatagram in the untrusted code. I inserted a printf statement in the implementation for NaClSendDatagram in shared/imc/nacl/nacl_imc.cc and shared/imc/linux/nacl_imc.cc.

It would appear that the implementation of NaClSendDatagram in shared/imc/linux/nacl_imc.cc is called when sending messages from trusted code to untrusted code, whereas the implementation of NaClSendDatagarm in shared/imc/nacl/nacl_imc.cc (a wrapper around imc_sendmsg):

int NaClSendDatagram(NaClHandle handle, const NaClMessageHeader* message,
                     int flags) {
  return imc_sendmsg(handle, (const NaClImcMsgHdr *) message,
                     flags);
}

Would you be able to comment on this?

Kind regards,
George

[Mark Seaborn]

On 2 July 2015 at 02:28, George Robinson <george.ro...@ucl.ac.uk> wrote:

I see!

Our goal is to send a message from trusted code to untrusted code, and then for the trusted code to send a message back to trusted code after doing whatever it needed to do with the message - like a request-response cycle. - all within a SINGLE host-os process. We have no need for IPC. 

We had at first used imc_sendmsg() in the untrusted code and then SendMsg() in the trusted code - but we were unable to initiate communication from the trusted code to the untrusted code. Rather, we found that we had to call imc_recvmsg() in the untrusted code, which invoked a callback in the trusted code (like in the custom_desc test) upon which the trusted code could then write into untrusted memory. Is this possible? As a result of these difficulties we found we were able to initiate communication from the trusted code to the untrusted code via NaClSendDatagram and NaClReceiveDatagram.

It's hard to answer because I can't tell what you have been trying in your code.  It would help if you posted your code, or relevant snippets of it.

In trusted code, were you calling the SendMsg()/RecvMsg() methods (the ones defined in nacl_desc_base.h), or were you trying to implement them (using nacl_desc_custom.h)?

Would you be able to explain what you mean by "because those definitions aren't useful and shouldn't be being used"? Is it that these definitions are unsafe?

What do you mean by unsafe?

If you mean "unsafe" as in "untrusted code could break out of the sandbox", then no, they're not unsafe:  The purpose of the NaCl sandbox is that you can link any code into an untrusted nexe and it shouldn't be able to break out of the sandbox.

Or do you mean "unsafe" as in "might do something the untrusted code doesn't expect"?

If it wouldn't consume too much of your time, a brief tour of the internals of IMC would be great! We're currently reading through the Native Client source to understand how it works - however we've only just reached the system call invocation in the trampoline routine for untrusted code (i.e. how the system call number calculates an offset in service runtime which is called like a function pointer etc).

You could start by looking at this (quite old) documentation of what IMC provides:

https://code.google.com/p/nativeclient/wiki/IMCSockets

(That will eventually disappear, but it's mirrored here: https://github.com/mseaborn/nacl-wiki/blob/master/IMCSockets.wiki)

Cheers,

Mark

[George Robinson]

Hi Mark,

The code I pasted above is snippets from our latest attempt - using NaClSendDatagram and NaClReceiveDatagram to send messages from the trusted code to the untrusted code, and then back again.

Trusted code: http://pastie.org/10283823
Untrusted code: http://pastie.org/10283822

However, our first attempt was to use the imc_sendmsg/imc_recvmsg and SendMsg and RecvMsg callbacks, like in the custom_desc test. Since they are function pointers, I presume they are infact callbacks. Therefore, we had code that looks like this:

Trusted code: http://pastie.org/10283930
Untrusted code: http://pastie.org/10283853

The issue with this code is that in order to send a message from the trusted code to the untrusted code, we must invoke imc_recvmsg() first in the untrusted code. We would like to be able to send messages to the untrusted code without having to first call imc_recvmsg() - like we do in the NaClSendDatagram example.

Can we call these functions directly, or must they be invoked as callbacks from imc_sendmsg and imc_recvmsg? If we can, how can we do this?

Since in a prior message you wrote:

NaClSend/ReceiveDatagram() are low-level layers of IMC's implementation which are only used in trusted code.

I concluded that these were not fit for use in untrusted code, such that the untrusted code might be able to write into trusted memory, etc.

Kind regards

George