An idea - add MD5 Hash of the nexe to the nmf file

[Assaf Raman]

Hi,

An issue that was very common when I was working on the OGRE port is
that the nexe file didn't update some of the times when I updated it
on the server.

If an MD5 Hash was part of the nmf per nexe file - the part of the
browser that loads the nexe could have checked the file against the
MD5 Hash - and if needed - downloaded the more updated version from
the server.

While on the subject – I would also recommend to add an option to
store the nexe in a compressed zip file (or preferably 7z file) - to
save download time – the OGRE nexe file is more the 20 mg in size –
downloading a compressed version can save time and bandwidth.

Also on the subject – an option to digitally sign a nexe file could be
nice.
Possibly all of the options are implemented in NaCl – so if that is
the case – I will be happy for any links or information.

Regards,

Assaf

[Ivan Krasin]

On Thu, Aug 25, 2011 at 12:07 AM, Assaf Raman <assaf...@gmail.com> wrote:
> Hi,
>
> An issue that was very common when I was working on the OGRE port is
> that the nexe file didn't update some of the times when I updated it
> on the server.
>
> If an MD5 Hash was part of the nmf per nexe file - the part of the
> browser that loads the nexe could have checked the file against the
> MD5 Hash - and if needed - downloaded the more updated version from
> the server.

If you name your nexe file like
myapp.6016f134731d580c727584b6b671935405856924.nexe, where this long
number is the hash (you can use a shorter hash, if you want), the
problem with caching would be resolved w/o any support from NaCl
plugin. Am I correct?

>
> While on the subject – I would also recommend to add an option to
> store the nexe in a compressed zip file (or preferably 7z file) - to
> save download time – the OGRE nexe file is more the 20 mg in size –
> downloading a compressed version can save time and bandwidth.

HTTP protocol supports different kinds of compressing. gzip is
supported by all the browsers (and since you need to run your NaCl
module in Chrome, it supports gzip for sure). So, the problem of
compression could be resolved on the server side. Would it work for
you?

>
> Also on the subject – an option to digitally sign a nexe file could be
> nice.

Would HTTPS address the problem of identity? Or, you want to be able
to distribute a nacl module across the internet, serve it from
arbitrary server and be able to validate identity? If yes, what's the
use case?

Please, note that I don't want to say that the proposed features are
useless. I just want to understand why do you need them.

Ivan Krasin

> Possibly all of the options are implemented in NaCl – so if that is
> the case – I will be happy for any links or information.
>
> Regards,
>
> Assaf
>

> --
> You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.
> To post to this group, send email to native-cli...@googlegroups.com.
> To unsubscribe from this group, send email to native-client-di...@googlegroups.com.
> For more options, visit this group at http://groups.google.com/group/native-client-discuss?hl=en.
>
>

[Ivan Krasin]

On Thu, Aug 25, 2011 at 12:07 AM, Assaf Raman <assaf...@gmail.com> wrote:

> Hi,
>
> An issue that was very common when I was working on the OGRE port is
> that the nexe file didn't update some of the times when I updated it
> on the server.
>
> If an MD5 Hash was part of the nmf per nexe file - the part of the
> browser that loads the nexe could have checked the file against the
> MD5 Hash - and if needed - downloaded the more updated version from
> the server.

Also, you describe the functionality that is very similar to
http://en.wikipedia.org/wiki/Cache_manifest_in_HTML5

>
> While on the subject – I would also recommend to add an option to
> store the nexe in a compressed zip file (or preferably 7z file) - to
> save download time – the OGRE nexe file is more the 20 mg in size –
> downloading a compressed version can save time and bandwidth.
>
> Also on the subject – an option to digitally sign a nexe file could be
> nice.
> Possibly all of the options are implemented in NaCl – so if that is
> the case – I will be happy for any links or information.
>
> Regards,
>
> Assaf
>

[Assaf Raman]

Hi,

Here is the use case I am thinking of - a game engine.
Scenario 1 - I put the nexe on my site and everyone point to my site
in their html, if they point an nmf file on my site - I can gzip the
nexe, change the name of the nexe file each time I update the version
(so it will get downloaded), I assume that the nmf file is never
cached (else I have an issue here) - so I only have the signature
issue - is there a way for the javascript that point to the nmf file
on my site to check that the https certificate is my company's one? If
so - https solves this issue also - and this scenario fully solved. If
I can't check the https certificate - there is a risk of a link
highjack attack because of cross site issues.

Scenario 2 - I give out nexe files to my developer community and they
use them on their sites - issues: gzip will not always be available
(not all site hosting services supports it), same goes for https,
regarding the solution renaming the files all the time, this makes you
edit the nmf every time you update the nexe, I guess we can live with
that, but if the nexe file was corrupted in the download - what is the
solution for that? How will you know?

One other note regarding gzip - it has a big cpu toll on the server in
many cases, why not have the file zipped in the first place? also -
gzip compression is not as good as 7z, and the download time here in
this case is significant for the app start time, so why not have the
best compression possible?

BTW: is it possible for the javascript to point to an nmf file that
points to an nexe created in the nexe file system space (or smiler)?
If so I could write a small nexe that downloads an MD5 Hashed, zipped
and signed nexe, then make the javascript on the page start it up.
Then I will have all the flexibility I need. This may be the best
option - and will require the least amount of work from the google
developers.

This e-mail is not as clear as I wanted, but I think it explains the
spirit of the issue.

Regards,

Assaf

On Aug 25, 10:29 am, Ivan Krasin <imkra...@gmail.com> wrote:

> On Thu, Aug 25, 2011 at 12:07 AM, Assaf Raman <assafra...@gmail.com> wrote:
> > Hi,
>
> > An issue that was very common when I was working on the OGRE port is
> > that the nexe file didn't update some of the times when I updated it
> > on the server.
>
> > If an MD5 Hash was part of the nmf per nexe file - the part of the
> > browser that loads the nexe could have checked the file against the
> > MD5 Hash - and if needed - downloaded the more updated version from
> > the server.
>

> Also, you describe the functionality that is very similar tohttp://en.wikipedia.org/wiki/Cache_manifest_in_HTML5

[Ivan Krasin]

On Thu, Aug 25, 2011 at 1:03 AM, Assaf Raman <assaf...@gmail.com> wrote:
> Hi,
>
> Here is the use case I am thinking of - a game engine.
> Scenario 1 - I put the nexe on my site and everyone point to my site
> in their html, if they point an nmf  file on my site - I can gzip the
> nexe, change the name of the nexe file each time I update the version
> (so it will get downloaded), I assume that the nmf file is never
> cached (else I have an issue here) - so I only have the signature
> issue - is there a way for the javascript that point to the nmf file
> on my site to check that the https certificate is my company's one? If
> so - https solves this issue also - and this scenario fully solved. If
> I can't check the https certificate - there is a risk of a link
> highjack attack because of cross site issues.

AFAIK, it's currently not possible to load nexe file from different
origin. Have you checked that?
I would be surprised if that's already supported, because the security
consequences of cross-origin scenario are not clear.

>
> Scenario 2 - I give out nexe files to my developer community and they
> use them on their sites - issues: gzip will not always be available
> (not all site hosting services supports it), same goes for https,
> regarding the solution renaming the files all the time, this makes you
> edit the nmf every time you update the nexe, I guess we can live with
> that

Why not generate nmf on the fly? It's just ~100-300 bytes.
If you don't like dynamic stuff, it's easy to write a tool that would
generate nmf at publication phase.

>but if the nexe file was corrupted in the download - what is the
> solution for that? How will you know?

I agree that checking the file for the integrity is important.
I don't think that NaCl has a solution for that now. Probably, it
should become a feature request to NaCl.

>
> One other note regarding gzip - it has a big cpu toll on the server in
> many cases, why not have the file zipped in the first place? also -
> gzip compression is not as good as 7z, and the download time here in
> this case is significant for the app start time, so why not have the
> best compression possible?

The drawbacks of implementing 7z in NaCl plugins is the security. It's
not really a good idea to increase the size of the code you have to
trust. Probably, it could be addressed if 7z decoder would run inside
NaCl itself.

>
> BTW: is it possible for the javascript to point to an nmf file that
> points to an nexe created in the nexe file system space (or smiler)?

Not sure what you had in mind here. Could you please explain a little bit more?

> If so I could write a small nexe that downloads an MD5 Hashed, zipped
> and signed nexe, then make the javascript on the page start it up.

This is a very good point. I think that something like that would be
available inside NaCl at some point. But now I don't think that it's
possible... Probably, I'm wrong.

-- krasin

[Assaf Raman]

> > BTW: is it possible for the javascript to point to an nmf file that
> > points to an nexe created in the nexe file system space (or smiler)?
>
> Not sure what you had in mind here. Could you please explain a little bit more?
>

The idea is to have a small bootstrap nexe that will downloaded a
compressed and signed nexe, extract it to somewhere javascript can
access it, the message backt to the javascript to start that
downloaded nexe.

[Ivan Krasin]

In fact, I've understood the whole idea, but I have failed to
understand what's "nexe file system space". Probably, it does not
matter.

[Assaf Raman]

On Aug 25, 11:32 am, Ivan Krasin <imkra...@gmail.com> wrote:

> On Thu, Aug 25, 2011 at 1:30 AM, Assaf Raman <assafra...@gmail.com> wrote:
>
> >> > BTW: is it possible for the javascript to point to an nmf file that
> >> > points to an nexe created in the nexe file system space (or smiler)?
>
> >> Not sure what you had in mind here. Could you please explain a little bit more?
>
> > The idea is to have a small bootstrap nexe that will downloaded a
> > compressed and signed nexe, extract it to somewhere javascript can
> > access it, the message backt to the javascript to start that
> > downloaded nexe.
>
> In fact, I've understood the whole idea, but I have failed to
> understand what's "nexe file system space". Probably, it does not
> matter.
>
>

NaCl have a very limited file access, you have to use pepper api to
access it, I meant that. Is ""pepper file system space" the right
name?

[Assaf Raman]

Hi,

Here is a good sample of a scenario where an MD5 Hash can help:
A file viewer (ex: pdf viewer), lets say it is 40 MB in size, with the
current way NaCl works in Chrome - assuming that each site will host
the file locally, you will have to wait the time it takes to download
20 MB on every site (once) before you can view even a very small file.
If the nmf would have included an MD5 Hash, and before downloading a
nexe file Chrome would check the Cache for a file with same name and
MD5 Hash - then it could be reused.
This logic very smiler to the way this is solved with other browser
plugins (ActiveX or Mozilla plug-ins) - there is a string or a GUID
that identifies the plug-in - you can download it from the first site
that requires it for its content, and later it is reused.
Having the MD5 Hash as unique id is best - this way a site that uses
an nexe can be sure the plugin is what the site needs and not an
imposter or a different version with the same GUID.
A digital signature id can be also be an alternative to be used the
same way.

Regards,

Assaf

[Jim]

> > One other note regarding gzip - it has a big cpu toll on the server in
> > many cases, why not have the file zipped in the first place? also -
> > gzip compression is not as good as 7z, and the download time here in
> > this case is significant for the app start time, so why not have the
> > best compression possible?
>
> The drawbacks of implementing 7z in NaCl plugins is the security. It's
> not really a good idea to increase the size of the code you have to
> trust. Probably, it could be addressed if 7z decoder would run inside
> NaCl itself.

As, I understand, any browser including chrome already has support for
gzip. Why not use that same code to decompress .nexe.gz directly
without http server to be involved. It just need a few tweets in
chrome source code, but save lots of http server cpu time.

[Gregg Tavares (wrk)]

On Sat, Aug 27, 2011 at 1:31 PM, Jim <jamesf...@torapp.info> wrote:

> > One other note regarding gzip - it has a big cpu toll on the server in
> > many cases, why not have the file zipped in the first place?

Most servers support already gzipped files. Certainly Apache does. You can put both an ungzipped file and a gzipped file in the same folder with the same name except .gz at the end of the gzipped file (example foo.nexe and foo.nexe.gz. When the browser asks for "foo.nexe" it tells the server whether or not it supports gzip (all browsers do). The server then servers either the foo.nexe or foo.nexe.gz. No heavy work needed by the server.

http://betterexplained.com/articles/how-to-optimize-your-site-with-gzip-compression/

 

also -
> > gzip compression is not as good as 7z, and the download time here in
> > this case is significant for the app start time, so why not have the
> > best compression possible?
>
> The drawbacks of implementing 7z in NaCl plugins is the security. It's
> not really a good idea to increase the size of the code you have to
> trust. Probably, it could be addressed if 7z decoder would run inside
> NaCl itself.

As, I understand, any browser including chrome already has support for
gzip. Why not use that same code to decompress .nexe.gz directly
without http server to be involved. It just need a few tweets in
chrome source code, but save lots of http server cpu time.

[Assaf Raman]

What about the case described in this post (the 20 MB file viewer re-
download for every site)?

[Colt McAnlis]

Hey Assaf,
I think the points you're bringing up are valid; In order to optimize
for the user experience, the delivered NaCl app needs to load fast,
cache easily along with being safe and stable.

A couple thoughts come to mind;
Firstly, a bootstrapper system is a great idea. And because the
architecture of everyone's app is different, I'm not sure there's a
good way to standardize this just yet.

For assets, what I've been recommending is something along these
lines:
1) 5-10mb install footprint that includes basic app & essential
assets.
2) A caching / fetching manager that uses the FileSystem APIs in
pepper. This can be used to stream in assets from a server, and cache/
evict them locally as needed. This will reduce the initial download
footprint, and be a robust solution over time.


For Nexe's, it's a bit of a different story, because we don't support
dynamic nexe linking. I believe what you've proposed is fine, and you
might be able to move it up to the javascript level where;
1) your initial page shows a dancing bear, while the nexe is requested
and cached locally.
2) Once it's downloaded fully, you update the HTML of a div to embed
the nexe and start playing.
I'm certain this would be a smaller footprint than brewing another
NEXE to do the same thing as the javascript could do.


Secondly, Caching as a dev; As you've found, chrome caches pretty
aggressively; There's ways, like you've mentioned to version your
nexe; but have you tried turning off caching in chrome? (open dev-
tools, and click on the gear in the bottom right hand corner). That
disables chrome caching, and forces the page to always grab from
server.


FWIW the OGRE port was fantastic to see, we all are in awe of the
progress and dedication you've put into it.

~Main

[Victor Khimenko]

On Sat, Aug 27, 2011 at 8:14 PM, Colt McAnlis <col...@google.com> wrote:

For Nexe's, it's a bit of a different story, because we don't support
dynamic nexe linking.

We kinda go. GLibC required changes to browser, but last CL landed last week so now it's only matter of changing the SDK. Sadly this is not trivial because we are not just adding DSO to the mix we are replacing newlib with glibc, which is totally different C library! Thus current plan is to ship M15 with newlib-based SDK then later ship another, GLibC-based SDK which will ALSO work with M15. If all goes well then few months later newlib-based SDK will be retired. Note that with GLibC you have a small loader which loads everything else - it should be possible to add LZMA to it if you wish so. But of course there are the catch: currently GLibC build only works under Linux. Mac and Windows use prebuit versions of GLibC.

[Elijah Taylor]

Hi Assaf,

On Sat, Aug 27, 2011 at 4:08 AM, Assaf Raman <assaf...@gmail.com> wrote:

What about the case described in this post (the 20 MB file viewer re-
download for every site)?

This type of thing sounds like a good candidate for a content handling extension using a nacl module.  Check out http://code.google.com/chrome/extensions/beta/manifest.html#nacl_modules

With this you could register mime types your nexe will handle (like OGRE scenes/models, or even full games depending on how your game data is separated from your engine) and then put something like <embed src="file.ogre" type="application/vnd.ogre"> on any page and if the extension is installed your handler will be invoked. I believe you'll need to check the args of your instance creation in order to get the src argument in order to load it.

-Elijah

[Assaf Raman]

Elijah - thanks, this is what I was looking for.

Good to know it exists, I will post about it in the OGRE forum.

Google people - this is a great feature of NaCl - I recommend adding a
nice sample to the NaCl SDK to demonstrate it.

Regards,

Assaf

On Aug 27, 9:31 pm, Elijah Taylor <elijahtay...@google.com> wrote:
> Hi Assaf,
>

> On Sat, Aug 27, 2011 at 4:08 AM, Assaf Raman <assafra...@gmail.com> wrote:
> > What about the case described in this post (the 20 MB file viewer re-
> > download for every site)?
>
> This type of thing sounds like a good candidate for a content handling

> extension using a nacl module.  Check outhttp://code.google.com/chrome/extensions/beta/manifest.html#nacl_modules

[Fernando Carvalho]

In reading this, I can see that NaCl has lot of features unknown by the public. I hope that NaCl developers invest more time in documentation and tutorials, so people could create great web applications.

--
You received this message because you are subscribed to the Google Groups "Native-Client-Discuss" group.
To post to this group, send email to native-cli...@googlegroups.com.
To unsubscribe from this group, send email to native-client-di...@googlegroups.com.
For more options, visit this group at http://groups.google.com/group/native-client-discuss?hl=en.

--
Fernando

[Matthew Ball]

(Just as a brief aside: I hope that MD5 is being used for illustrative purposes only, meaning "some cryptographic hashing function".  We don't want to actually use MD5 itself because it's possible to find MD5 collisions within minutes (or even seconds) on a laptop computer.  In practice, we'd want to use a stronger function, like SHA1 or even SHA256.)

-Matt

[assaf raman]

Yes, MD5 is used just to demonstrate the idea, best to use some kind of strong digital signature. Preferably the kind you can verify with an authentication services company if needed.

[Jason Kim]

I seem to remember asking for this feature (hash signatired on the nexe's) some months back when the .nmf format was being discussed - at the time it was voted down in lieu of a format extension so that we can someday support this feature - is this now that time? :-)

-jason