localStorage Limit

[Nathan]

Just a quick note, primarily to ensure I've said it rather than just
thought it:

The Web Storage spec recommends a 5mb quota per localStorage space -
javascript is UTF-16, thus 16 bytes per char, which means that for all
practical UTF-8/ascii use cases we're only getting 2.5mb of usable
space, enough to store about 260,000 chars. A limit worth remembering.

Best, Nathan

[Bjarni Rúnar Einarsson]

Hmm, I think you misunderstand slightly.

UTF-16 is 2 bytes (16 bits) per character, and UTF-8 is variable
length depending on what you are representing (1-4 bytes, 1 byte for
the common stuff). ASCII is a subset of UTF-8 which is guaranteed to
be 1 byte per character (7 bits).

So unless the Web Storage spec is talking about mega*bits* (which
would be pretty silly), then you are off by a factor of 10 - 2.5
million characters or more is a reasonable expectation. But it's
still a limit, many apps would hit that quite rapidly.

:-)

--
Bjarni R. Einarsson
Founder, lead developer of PageKite.

Make localhost servers visible to the world: http://pagekite.net/

[Nathan]

Bjarni R�nar Einarsson wrote:
> On Tue, Nov 29, 2011 at 11:41 PM, Nathan <nat...@webr3.org> wrote:
>> Just a quick note, primarily to ensure I've said it rather than just thought
>> it:
>>
>> The Web Storage spec recommends a 5mb quota per localStorage space -
>> javascript is UTF-16, thus 16 bytes per char, which means that for all
>> practical UTF-8/ascii use cases we're only getting 2.5mb of usable space,
>> enough to store about 260,000 chars. A limit worth remembering.
>
> Hmm, I think you misunderstand slightly.
>
> UTF-16 is 2 bytes (16 bits) per character, and UTF-8 is variable
> length depending on what you are representing (1-4 bytes, 1 byte for
> the common stuff). ASCII is a subset of UTF-8 which is guaranteed to
> be 1 byte per character (7 bits).
>
> So unless the Web Storage spec is talking about mega*bits* (which
> would be pretty silly), then you are off by a factor of 10 - 2.5
> million characters or more is a reasonable expectation. But it's
> still a limit, many apps would hit that quite rapidly.

Lol yes sorry! 2.6 million chars in chrome, not 260k! Thank for the catch :)

[Michiel de Jong]

unless you're writing the bible, you're unlikely to go over that with user-typed keystrokes (the bible has about 3 million chars apparently)

at the same time, if you use it for photos, one photo can be enough to fill it up.

i was thinking about using localStorage to improve upon app cache, so store part of the application code in it. That might be useful mainly for potentially big applications like Libre Docs (which currently takes 60 seconds to present a login screen). It would give us more control over expulsion strategies, and avoids the warning in firefox.

what we mainly have to think about is paging user data from remoteStorage into localStorage. we currently just load in the whole data set when the user logs in, and  don't have an API for paging data in response to user actions.

thanks for bringing this up!

On Thu, Dec 1, 2011 at 3:57 AM, Nathan <nat...@webr3.org> wrote:

[Bjarni Rúnar Einarsson]

On Thu, Dec 1, 2011 at 2:55 AM, Michiel de Jong <mic...@unhosted.org> wrote:
> unless you're writing the bible, you're unlikely to go over that with
> user-typed keystrokes (the bible has about 3 million chars apparently)

Data point: My personal blog is about 5.5 MB of plain text (stored as
a unix mbox file, actually), including metadata but not counting
comments and images and the like. Admittedly, it does go back to
1999, but hey, I typed it all myself and I'm not even a particularly
verbose writer.

People will hit these limits if the apps are useful. :-)

[Pierre Grimaud]

This is part of the reason i wrote a compression library that does use 15bits for each character produced: http://pieroxy.net/blog/pages/lz-string/index.html

This allows you a little more room ;-)