old snapshots take a *lot* of space

[Jason Grout]

I was just troubleshooting why my sage notebook is slowing way down,
especially in saving and stopping. Michael Abshoff suggested looking at
the snapshots directory. I discovered that I had over 7 gigs in my sage
notebook directory, and most of my worksheets (simple text, maybe an
image or two) had 50-100 megabyte snapshot directories. My oldest
worksheet had over 11,000 snapshots, each about 770 bytes, but each
taking 4K of disk space.

So: can I just delete all of these snapshots? Can we make a sage
function that cleans out these snapshots if we decide we don't need them?

Thanks,

Jason

--
Jason Grout

[William Stein]

2009/4/21 Jason Grout <jason...@creativetrax.com>:

>
> I was just troubleshooting why my sage notebook is slowing way down,
> especially in saving and stopping.  Michael Abshoff suggested looking at
> the snapshots directory.  I discovered that I had over 7 gigs in my sage
> notebook directory, and most of my worksheets (simple text, maybe an
> image or two) had 50-100 megabyte snapshot directories.  My oldest
> worksheet had over 11,000 snapshots, each about 770 bytes, but each
> taking 4K of disk space.
>
> So: can I just delete all of these snapshots?

Yes.

> Can we make a sage
> function that cleans out these snapshots if we decide we don't need them?

When the notebook is about to create a snapshot, it *should* clean up
old snapshots using some simple criterion, e.g., at most one per hour
after the first hour, then at most one per day up to a month, but
after a month just one per month, etc.

-- William

[Rob Beezer]

Jason,

Have a look at:

http://trac.sagemath.org/sage_trac/ticket/5371

http://trac.sagemath.org/sage_trac/ticket/5459

There are few other tickets about that are relevant, as well.

Rob

[mabshoff]

On Apr 21, 11:27 pm, Rob Beezer <goo...@beezer.cotse.net> wrote:
> Jason,
>
> Have a look at:
>
> http://trac.sagemath.org/sage_trac/ticket/5371
>
> http://trac.sagemath.org/sage_trac/ticket/5459
>
> There are few other tickets about that are relevant, as well.

3.4.1 should have most of those (except the default changes for save
interval, etc). But I don't think old snapshots are then dynamically
deleted.

I posted some reviewer patches at #5371 - you might want to review
them.

> Rob

Cheers,

Michael

[kcrisman]

> I was just troubleshooting why my sage notebook is slowing way down,
> especially in saving and stopping.  Michael Abshoff suggested looking at
> the snapshots directory.  I discovered that I had over 7 gigs in my sage

So did deleting do the trick? I have been unable to find out what is
causing this for me (and there have been a bizarre variety, though not
huge number, of posts on this in the last month i.e. not all the same
OS or whatever).

- kcrisman

[Jason Grout]

No, it didn't. I deleted my .sage/sage_notebook folder (well, moved it),
so Sage created a new notebook, and now things are back to being almost
instantaneous.

Thanks,

Jason

[kcrisman]

>
> > So did deleting do the trick?  I have been unable to find out what is
> > causing this for me (and there have been a bizarre variety, though not
> > huge number, of posts on this in the last month i.e. not all the same
> > OS or whatever).
>
> No, it didn't. I deleted my .sage/sage_notebook folder (well, moved it),
> so Sage created a new notebook, and now things are back to being almost
> instantaneous.

Hmm, that is EXACTLY the experience of my sysadmin with our server -
deleting old snapshots did precisely nothing, but starting a new
server with no users and their attendant notebooks seems to be
instantaneous.

Looking at Trac a bit, I see something weird. It seems consistent
with the timing of the downgrade in performance of the notebook on
some machines that # 5291 was merged in 3.3.rc2, while # 5687 (and the
fix at # 5703) does almost exactly the same thing as # 5291 while
deleting # 5291's way of doing it. The efficiency issue of # 5300 has
not been addressed. Tickets # 5371 and # 5459 are also relevant but
would not necessarily be the immediate issue.

So here are some questions.

If someone starts a Sage process during an auto-save in a situation
with not tons of memory and/or CPU, is all hope lost for a quick
evaluation?
Could the additional user created in a server situation exacerbate
this problem by adding connection time by the server?
How long does open(worksheet_txt).read() take, anyway, for a given
text file, and how much processing power?
And why would getting rid of notebook files make a difference if only
the open worksheet(s) is (are) being auto-saved - is it somehow
possible that other worksheets are being at least looked at (though
not actually being updated since nothing has changed)?

I hope at least one of these is non-stupid and helps lead to a
resolution of this, but if not I certainly understand!

- kcrisman

[Rob Beezer]

kcrisman,

You've discovered the tangle of tickets on this I alluded to
above. ;-)

1. You can delete all your old snapshots to reclaim disk space (or
relieve quota limits in a shared environment). I used some
incantation with find, xargs, and rm (carefully) which cleared them
out for every worksheet all at once. Like Jason I had a few gigabytes
worth across about 100 worsheets from about a year's worth of use.

2. I think the performance hit comes from backing up nb.sobj, from
the original in .sage/sage_notebook to a collection of backups
in .sage/backups The default timing on this is very frequent IIRC.
My nb.sobj is about 30 MB, but my home directory is on a portable USB
drive that I shuttle around, so I noticed the delays due to these auto-
saves perhaps sooner than others might. I'm not sure what it takes to
shrink this file. I set the autosave interval to be about 20 minutes
and am keeping fewer backups (two instead of three) and no longer see
the regular slowdowns as I work in the notebook.

3. When I quit a server that has been running the notebook, it takes
about 2 minutes for a clean shutdown, which I assume is the same sort
of action.

The original patch at http://trac.sagemath.org/sage_trac/ticket/5371
has some comments in it about extending the default save times if you
wnt to take immediate action. Some of the names of these autosave
variables are mildly misleading, though this is corrected in some of
the patches (above) addressing the situation.

Rob

[William Stein]

On Wed, Apr 22, 2009 at 9:15 AM, Rob Beezer <goo...@beezer.cotse.net> wrote:
>
> kcrisman,
>
> You've discovered the tangle of tickets on this I alluded to
> above.  ;-)
>
> 1.  You can delete all your old snapshots to reclaim disk space (or
> relieve quota limits in a shared environment).  I used some
> incantation with find, xargs, and rm (carefully) which cleared them
> out for every worksheet all at once.  Like Jason I had a few gigabytes
> worth across about 100 worsheets from about a year's worth of use.
>
> 2.  I think the performance hit comes from backing up  nb.sobj, from
> the original in .sage/sage_notebook to a collection of backups
> in .sage/backups   The default timing on this is very frequent IIRC.
> My nb.sobj is about 30 MB, but my home directory is on a portable USB
> drive that I shuttle around, so I noticed the delays due to these auto-
> saves perhaps sooner than others might.  I'm not sure what it takes to
> shrink this file.  I set the autosave interval to be about 20 minutes
> and am keeping fewer backups (two instead of three) and  no longer see
> the regular slowdowns as I work in the notebook.

I set the autosave interval on sagenb.org to once per *day*.

Note that even with a long autosave, you loose nothing if you properly
shutdown the server before exiting, since it essentially never
crashes.

> 3.  When I quit a server that has been running the notebook, it takes
> about 2 minutes for a clean shutdown, which I assume is the same sort
> of action.

True.

>
> The original patch at http://trac.sagemath.org/sage_trac/ticket/5371
> has some comments in it about extending the default save times if you
> wnt to take immediate action.  Some of the names of these autosave
> variables are mildly misleading, though this is corrected in some of
> the patches (above) addressing the situation.
>
> Rob
>
> >
>

--
William Stein
Associate Professor of Mathematics
University of Washington
http://wstein.org

[kcrisman]

Dear Rob,

>
> > 1.  You can delete all your old snapshots to reclaim disk space (or
> > relieve quota limits in a shared environment).  I used some
> > incantation with find, xargs, and rm (carefully) which cleared them
> > out for every worksheet all at once.  Like Jason I had a few gigabytes
> > worth across about 100 worsheets from about a year's worth of use.

Yes, we had figured that was part of the issue. Running du .sage/
sage_notebook takes forever on my G4, for instance :)

>
> > 2.  I think the performance hit comes from backing up  nb.sobj, from
> > the original in .sage/sage_notebook to a collection of backups
> > in .sage/backups   The default timing on this is very frequent IIRC.
> > My nb.sobj is about 30 MB, but my home directory is on a portable USB
> > drive that I shuttle around, so I noticed the delays due to these auto-
> > saves perhaps sooner than others might.  I'm not sure what it takes to
> > shrink this file.  I set the autosave interval to be about 20 minutes
> > and am keeping fewer backups (two instead of three) and  no longer see
> > the regular slowdowns as I work in the notebook.

I see what is going on. The problem is not the snapshotting, but the
backups to this file. I assumed the discussion of autosave was
referring to worksheet.txt in all these tickets, but it is this
mysterious file. Since it wasn't that big of a file, I thought maybe
it wasn't the problem.

(Out of curiosity, does this file also have any of the permissions for
notebooks? Probably irrelevant.)

>
> I set the autosave interval on sagenb.org to once per *day*.
>

Again, I remember that discussion, but didn't realize there were two
different autosaves going on here.

> > 3.  When I quit a server that has been running the notebook, it takes
> > about 2 minutes for a clean shutdown, which I assume is the same sort
> > of action.
>
> True.
>
>

Yes, it's taking astonishingly long to quit the notebook as well.

I will relay this to sysadmin AND try some of this out on my laptop,
and hopefully that will solve it. I hope the "tangle of tickets" gets
resolved soon, though unfortunately I am not competent to try - just
competent enough to make a well-intentioned mistake, instead!

Thanks,
- kcrisman

[kcrisman]

Followup:

> The original patch athttp://trac.sagemath.org/sage_trac/ticket/5371

> has some comments in it about extending the default save times if you
> wnt to take immediate action.  Some of the names of these autosave
> variables are mildly misleading, though this is corrected in some of
> the patches (above) addressing the situation.

Apropos of that, what is the difference between auto_save_interval,
save_interval, idle_check_interval, idle_timeout? It isn't clear to
me and I might want to mess with several of them.

Thanks,
- kcrisman

[Rob Beezer]

I think auto_save_interval could refer to worksheets, while
save_interval might refer to nb.sobj. This is part of the problem
here - there are variables, keywords and page headings (via "Settings"
in the notebook) that are vague or misleading, and even my attempt to
add a bit of documentation to help isn't sufficient to remind me
enough to give an accurate answer. :-(

Rob

[Rob Beezer]

Here's my $0.02 worth on worksheet management as part of the notebook
interface.

For most new users the notebook is the face of Sage initially. And it
is very impressive. But I worry about users seeing the (very
frustrating) slowdowns that Jason and I experienced after heavy use
(tens or hundreds of worksheets) for a long period (months), and I
really worry about server setups at schools (next fall!) where the
growth of the snapshot directories causes severe headaches for
sysadmins that are not very familiar with the internals of Sage.

There are a good many patches out there to address small aspect of
these problems. In #4886 and #4887 Timothy Clemans has some good
ideas for renaming some variables and making default save periods
configurable from the "Settings" page of the notebook. However, these
patches seem to depend on a variety of other patches, some with
distinctly different purposes, with #4135 being a major obstacle to
freeing up all the dependencies.

It would be great if somebody who understands the notebook, CSS,
templates, and file management were to take the best ideas from these
patches and address the core problems in one, or a few, pointed
patches. Unfortunately, I am not that person, but I'd be ready to
help and review work done in this direction. Done right, this might
obsolete several other patches and then free up others that would also
be welcome enhancements to the notebook.

Down off soap box,
Rob

[William Stein]

I would just forget about all that crap, and write
something so there can be at most say 30 snapshots for any given
worksheet. Full stop. How those 30 are divied out is a matter of
algorithms, but the simplest choice is just delete all but the 30 most
recent.

William

[kcrisman]

> > There are a good many patches out there to address small aspect of
> > these problems.  In #4886 and #4887 Timothy Clemans has some good
> > ideas for renaming some variables and making default save periods
> > configurable from the "Settings" page of the notebook.  However, these
> > patches seem to depend on a variety of other patches, some with
> > distinctly different purposes, with #4135 being a major obstacle to
> > freeing up all the dependencies.
>
> > It would be great if somebody who understands the notebook, CSS,
> > templates, and file management were to take the best ideas from these
> > patches and address the core problems in one, or a few, pointed
> > patches.  Unfortunately, I am not that person, but I'd be ready to
> > help and review work done in this direction.  Done right, this might
> > obsolete several other patches and then free up others that would also
> > be welcome enhancements to the notebook.

Yes, Timothy has done a lot of yeoman's work but the patches were not
reviewed and now they are likely bitrotted - and in any case have far
too long a chain of dependencies for easy review.

I agree with Rob that these would be nice - particularly because it
makes *user config* easy. It looks like setting save_interval to
24*3600 solves a lot of our problems (which indicates to me that
somehow it's not just open worksheets that are saved...) out here,
according to an email I just got - I am very lucky that our admin is
so supportive of Sage! I can imagine other places where that would
not be the case, and a faculty member or a not-particularly-adept low-
level employee was in charge of this, one who is not going to be able
to make it through server_conf.py.

>
> I would just forget about all that crap, and write
> something so there can be at most say 30 snapshots for any given
> worksheet. Full stop.  How those 30 are divied out is a matter of
> algorithms, but the simplest choice is just delete all but the 30 most
> recent.

Something like that for a default would be wonderful (and please
please do the same with whatever controls nb.sobj). Defaults should
be made quite clear and then also clear where they can be changed -
which a lot of the patches discussed in this thread do, I'm not
suggesting that there is all this work which has to be done, more it
has to be organized. But it might as well be done right, after
William's very good ideas on this are implemented for the default.
Yes, I realize that volunteers myself for it... well, not until after
finals, that's for sure.

- kcrisman

[William Stein]

Hi,

Does anybody here ever use snapshots? This is what implements the
"Undo" button.

I never use it.

William

[Jason Grout]

I use it on occasion when something happens with the notebook and I
would otherwise have lost code. I also like the confidence it gives me
to experiment, knowing there is a version history. I would definitely
keep it.

Jason

[William Stein]

Wouldn't the history of the last 500 (say) commands serve roughly the
same purpose? That's the "Log" button at the top of the screen.

William

[Stan Schymanski]

I have used snapshots when I forgot to save a notebook under a new name
before making big changes and then wanted to revert to the original. It
wasn't very convenient, as the plots are not saved, so I could not
easily recognise the right version. However, it saved my day!
I like the idea of just saving the last 30 or so snapshots. I had
numerous computer crashes while working with notebooks, but I never lost
any data. Is this not due to the snapshots?

Cheers,
Stan

[kcrisman]

I definitely agree with those who would keep snapshot. Having a list
of commands is not the same - for one thing, log does not save the
whole worksheet, only commands (i.e. not TinyMCE stuff); for another,
it logs notebook-wide, so it becomes a bit of a jaunt to find stuff.

That said, I also agree that, given log's presence, worksheet
snapshots could be much less frequent - which I don't think anyone has
disagreed with in any case.

- kcrisman

[William Stein]

Hi,

I've written a patch against 3.4.1:

http://trac.sagemath.org/sage_trac/ticket/5880

which simply greatly reduces the number of situations that result in
snapshots. Basically, now you get them when you click "save". There
is no autosave.

Please try/test.

-- William

[Timothy Clemans]

On Thu, Apr 23, 2009 at 4:34 PM, William Stein <wst...@gmail.com> wrote:
>
> Hi,
>
> I've written a patch against 3.4.1:
>
>  http://trac.sagemath.org/sage_trac/ticket/5880
>
> which simply greatly reduces the number of situations that result in
> snapshots. Basically, now you get them when you click "save".   There
> is no autosave.
>

I think this is a really bad idea. It's inconvenient to manually save
changes. Currently you have to scroll to the top. At least in Google
Docs you can use CTRL-S. I much prefer your earlier suggestion of just
keeping the last 30 snapshots.

If we're going to get rid of auto-save, I think we should make it more
convenient to manually save and have reminders to save. I used to use
the notebook for tutoring and would loose work a lot and it was really
annoying. This would just make things worst.

[William Stein]

On Thu, Apr 23, 2009 at 2:13 PM, Timothy Clemans
<timothy...@gmail.com> wrote:
>
> On Thu, Apr 23, 2009 at 4:34 PM, William Stein <wst...@gmail.com> wrote:
>>
>> Hi,
>>
>> I've written a patch against 3.4.1:
>>
>>  http://trac.sagemath.org/sage_trac/ticket/5880
>>
>> which simply greatly reduces the number of situations that result in
>> snapshots. Basically, now you get them when you click "save".   There
>> is no autosave.
>>
>
> I think this is a really bad idea. It's inconvenient to manually save
> changes. Currently you have to scroll to the top. At least in Google
> Docs you can use CTRL-S. I much prefer your earlier suggestion of just
> keeping the last 30 snapshots.
>
> If we're going to get rid of auto-save, I think we should make it more
> convenient to manually save and have reminders to save. I used to use
> the notebook for tutoring and would loose work a lot and it was really
> annoying. This would just make things worst.

So with the current snapshot system you already lost a lot of work? Why?

William

[Timothy Clemans]

The intervals were too big. For awhile users have been able to change
the auto-save interval. I changed mine to 1. That really helped.
> William
>
> >
>

[Rob Beezer]

On Apr 22, 5:36 pm, William Stein <wst...@gmail.com> wrote:
> Does anybody here ever use snapshots?

I have never used a snapshot, that I am aware of. I've lost a cell or
two due to crashes, but I think this was always due to my flaky USB
hard drive setup and not Sage's fault. And it was always just messing
around while writing lecture notes, or something similarly not too
important.

I like the way the proposed patch wipes the slate fairly clean. But I
sort of hope it is temporary and at some point a rational autosave
strategy of some sort is implemented. There was an attempt to not
create a snapshot if there were no changes, but I don't think that
worked as advertised. And the auto-save interval was more like "save
if there is a trigger and it has been *longer* than the interval"

It would be nice if

(a) autosave, or not, was user-configurable from the notebook settings
by user

(b) autosave interval could be set to *any* integer number of seconds/
minutes (maybe 0 means don't) in notebook settings

(c) There really weren't ever identical snapshots created

(d) William's idea of phased snapshots (fewer as they age) was
implemented since it is a good idea.

Is it problematic to spawn a thread that just sleeps for an interval
and then wakes up to consider making a snapshot?

Anyway, this isn't meant to be a proposal, just the suggestion that if
properly designed, then a useful, unobtrusive backup system should be
possible.

I hope to review William's patch soon, but will let this thread run
its course first.

Rob

[Stan Schymanski]

+1 for a 'smart' auto backup system. I don't know whether it was the
snapshot saving or what did it, but I never bothered saving notebooks
while working with them and never lost any data after crashes. It is
such a great peace of mind not to have to remember to save your work
regularly and still not to risk any data loss. This was an experience I
have never had with any other software before.

For some reason, I was thinking that the auto save in sage works
incrementally, like for example in svn. The problem of identical
snapshots suggests otherwise. Am I confusing something here?

Cheers,
Stan

--

[kcrisman]

> > I like the way the proposed patch wipes the slate fairly clean.  But I
> > sort of hope it is temporary and at some point a rational autosave
> > strategy of some sort is implemented.    There was an attempt to not

The only issue is that there is no guarantee that the autosave, in
some "rational" form, will ever come back if this patch is merged,
because it depends on the people who understand that code enough to do
it. This seems problematic. But for now even reducing the default
number autosaved to 100, or 30, or 2, or some integer greater than
zero, would seem to be a good strategy. E.g. for power outages or
battery failures or whatever.

- kcrisman

[William Stein]

Even with the patch we're discussing, the total number of snapshots
produced by people on sagenb.org in the last 20 hours is over 10,000:

sage@sagenb:~/sagenb$ ls -1 */worksheets/*/*/snapshots/*|wc -l
10395

So it really doesn't cut down on them very much. Of course a lot of
people use sagenb.org, e.g., well over 100 distinct uses just
*yesterday*, and over 7200 registered users.

I also like the idea to reduce the total max number of autosaved to
some fixed number, e.g., 30.

-- William

[William Stein]

On Fri, Apr 24, 2009 at 5:51 AM, kcrisman <kcri...@gmail.com> wrote:
>

>> > I like the way the proposed patch wipes the slate fairly clean.  But I
>> > sort of hope it is temporary and at some point a rational autosave
>> > strategy of some sort is implemented.    There was an attempt to not
>
> The only issue is that there is no guarantee that the autosave, in
> some "rational" form, will ever come back if this patch is merged,
> because it depends on the people who understand that code enough to do
> it.  This seems problematic.

You shouldn't worry about that. I've hired Mike Hansen fulltime to
work fulltime
from now until at least the end of the summer, and I'm confident he
will fix the issues as part of a broader project he is working on.
This isn't going to just languish forever.

William

[William Stein]

On Fri, Apr 24, 2009 at 12:35 AM, Stan Schymanski <schy...@gmail.com> wrote:
>
> +1 for a 'smart' auto backup system. I don't know whether it was the
> snapshot saving or what did it, but I never bothered saving notebooks
> while working with them and never lost any data after crashes.

When a snapshot is taken the state of the worksheet is also saved. This will
still happen fairly frequently.

> It is
> such a great peace of mind not to have to remember to save your work
> regularly and still not to risk any data loss. This was an experience I
> have never had with any other software before.

+1 -- I also hate software that causes one to lose work, which is
partly why Sage tries so hard not to.

> For some reason, I was thinking that the auto save in sage works
> incrementally, like for example in svn. The problem of identical
> snapshots suggests otherwise. Am I confusing something here?

It *should* work incrementally, and could be implemented that way.
However that would be more work, and nobody has done that work yet.

[Rob Beezer]

I have made a patch that attempts to limit the number of snapshots
that get saved (per worksheet) to an absolute maximum of 30. There's
a bit of a dilemma about just how to do this, given that at present
some users will have more than 30 snapshots for some worksheets, and
in the future this limit might be configurable. More details at
http://trac.sagemath.org/sage_trac/ticket/5895 - comments invited.

At http://trac.sagemath.org/sage_trac/ticket/5880 Gonzalo Tornaria has
said that since 3.4.1 snapshots that are duplicates are no longer
being created, which I can confirm with limited testing. He has
posted there a bash script that will clean out duplicate snapshots
created by previous versions, should you want to do some more surgical
house-cleaning.

[kcrisman]

Just as a postscript, I was really glad to have the autosave a few
minute ago, where I nearly deleted an entire lecture's worth of notes
and computations which I hadn't saved, by accidentally brushing
against a browser shortcut to a "favorites" page on the wrong window.
(This could happen with "reload" or other shortcuts, too, I suspect.)
When I used the browser "back" button, I had literally nothing, but
the autosave had captured my previous revision!

- kcrisman

[William Stein]

You could have just pressed reload and you would have had your
worksheet back entirely. It was still sitting there on the server in
RAM. Never use the back button with the sage notebook, unless you
immediately press refresh.

william

[Jason Grout]

Tom or some other javascript ninja:

Isn't there some sort of javascript that disables the back button?
That's a common problem in web applications, and I'd be very surprised
if it isn't a solved problem already.

Jason

[kcrisman]

> You could have just pressed reload and you would have had your
> worksheet back entirely.   It was still sitting there on the server in
> RAM.   Never use the back button with the sage notebook, unless you
> immediately press refresh.
>
> william

ok, that seems reasonable! But of course accidents do happen.

- kcrisman

[William Stein]

Not, there is no javascript to do that. Also, disabling the back
button is irrelevant to the above problem, which would only have been
prevented by disabling being able to leave the page by clicking on a
favorites. There is javascript to attempt to prevent users from
leaving pages -- lots of spam sites use it. Of course it doesn't work
well though, and is often extremely annoying.

William

[Franco Saliola]

I am using GMail. I hit reply. I typed this. Now I am going to hit the
browser's back button...

I'm still here. So Google can do it. It presented me with a warning:

Your draft has been modified.

Abandon changes?

See the attached screenshot.

Franco

--

[William Stein]

They did not disable the back button. They are calling a javascript
function on leaving the page, which could be via the back button,
clicking a link, etc. The Sage notebook could also popup a dialog when
you try to leave a worksheet page.

> It presented me with a warning:
>
>    Your draft has been modified.
>
>    Abandon changes?
>
> See the attached screenshot.
>
> Franco
>
> --
>
> >
>

[Franco Saliola]

Franco

--

Right, sorry. I misunderstood and didn't realize that the goal was to
disable the back button.

I think it might be a good idea to pop-up a dialog when one tries to
leave a worksheet if it hasn't been saved.

Franco

--