Learning how to copy over cache.dat files

[Dawn Wolthuis]

We are just starting to learn about working with cache.dat files,
trying to figure out how to replace one with another. We copied over
one that was created in 2009.1 to another box running Cache' 2010.2.1.

We were going to first dismount the database, but decided to see what
happens if you just delete the cache.dat file at the unix level (after
saving it off) and it let us do that. We could then put our other one
(from 2009.1, with no conversions run) in it's place, changing
ownership and permissions accordingly. We are able to ZN to that, but
we cannot LOGTO it from another namespace. This is a DATA database in
the 2010.2.1 system, while it was the whole kit and caboodle for the
namespace in 2009.1.

If we type MV at the cos prompt we get a security error.
[846] You do not have the required database access privileges to logon
to ZIPPEE.

We are able to ZN to the namespace, but then when we type MV, we get
that same error before it goes to the colon prompt

ZIPPEE>MV
[846] You do not have the required database access privileges to logon
to ZIPPEE.

* MultiValue Shell *
* Cache for UNIX (Red Hat Enterprise Linux 5 for x86-32) 2010.2.1
(Build 503_0_9983U) Wed Dec 8 2010 01:16:59 EST*

Also, if we attempt to dismount this database now (after copying it
over or even if we copy the original back) we get an error. I ran an
integrity check on the original cache.dat file after we put it back
and we get things like this in the integ.txt file:

Cache error: <PROTECT>CheckIntegrity+37^%SYS.DATABASE

I've been searching the doc trying to figure out where it will tell me
more about copying overtop of or replacing a cache.dat file with
another, but I'm coming up short. I'm guessing I'm not searching for
the right terms. Any clues? Thanks. --dawn

--
Dawn M. Wolthuis

Take and give some delight today

[Jason Warner]

I could see this corrupting your databases because of the way Cache
works and Unix works.

On an OS level, when Unix "locks" a file and then you replace it, the OS
essentially makes a couple of symlinks. Any programs with locks on the
old file get a link to the old file and anything else gets pointed to
the new file.

As I understand it, for performance reasons, Cache batches up writes in
the WIJ and then makes copies of all these writes to the journal. When
you restarted your namespace, it then replayed the journal corrupting
the new file. However, when you copied the old file back in place, it
was a copy probably missing some writes from the WIJ. This makes it
essentially a corrupt file also (since the journal was replayed already
on your other file).

This is my guess with knowledge of Unix style OSes and my newbie
knowledge of Cache. I think you would want to always unmount a database
before copying or replacing it. This forces the WIJ to write out so your
copy won't be corrupt. It also removes the OS level locks so when you
replace the database it isn't working on a file other than the one you
intended.

Jason

[htg]

It's unbelievable!

> see what happens if you just delete the cache.dat file at the

> unix level and it let us do that.

It's like driving the highway and blowing the wheels at full speed.
You probably won't die fast but slowly and painful.

Suggestion:
This sysadmin better returns his certificate and looks for a job @
McDonald's
and never touches a computer again!

[Dawn Wolthuis]

Yes, just to be clear, we are on a box that is not even live, in a
namespace that has nothing we need in it and we are testing out
everything a person might do so we have an idea of how this all works.
Does that clarify? If we could find doc on what should be done or if
we had found doc where this should not be done, then we would not have
tried to derive it ourselves. Such doc might actually be there, but
not where I could find it. Thanks. --dawn

> --
> You received this message because you are subscribed to the Google Groups "InterSystems: Zen Community" group.
> To post to this group, send email to InterSys...@googlegroups.com
> To unsubscribe from this group, send email to InterSystems-Z...@googlegroups.com
> For more options, visit this group at http://groups.google.com/group/InterSystems-ZEN?hl=en
> Zen Community Terms and Conditions: http://groups.google.com/group/InterSystems-ZEN/web/community-terms-and-conditions

[Dawn Wolthuis]

Oh, and by the way, if you remove a cache.dat file and then put it
back, which remember does not make it usable, then if you no longer
point to that cache database as a database backing any namespace, you
cannot delete the database.

So, after removing this cache.dat file, we could not delete the
database and recreate it with the same names and all, even though we
pointed the namespace to another database.

Because it appeared that the errors we received were mv errors when
trying to recover, I am sending the question here.

Other things we will be testing -- copying a cache.dat file over
another one. I do not know if we need to dismount the source database
before copying the cache.dat file, whether we need to dismount the
destination before copying over it, etc.

One Goal: To go from using version control for deploying new builds of
software to test environments to using some other means (copying
cache.dat files?) for deploying new builds to live environments
without disrupting users unnecessarily when a new build is deployed. I
think I'm just missing the doc on the right way to do this, but
figured I would go through options.

Another Goal: to copy data from one namespace to another. In this
case, the data came from a prior version. We will need to get it moved
to a new box, new version of the OS, of Cache, etc. We are beginning
to test out how to do that too.

So, yeah, figure I'm clueless but motivated to understand just how
these cache.dat files can be handled. Everything we have done to date
with them has been working through version control and utilities. I
could find no utilities for deploying an existing cache.dat data file
or code file to another namespace, but there might well be some. In
the absence of that, we need to develop our own, which means getting
more understanding of what works and what doesn't. We can definitely
cross off the "delete a cache.dat file and put another in it's place"
approach and I'm guessing there are a bunch more that will not work.

cheers! --dawn

[icargill]

Dawn,

> We can definitely cross off the "delete a cache.dat file and put another in it's place"

Not necessarily. We copy CACHE.DATs all the time for various reasons -
sharing test
environments between developers, making copies of user's live system
into a test or
training area, etc.

We don't do it for upgrading live sites (obviously, you'd lose the
data), but it can certainly
be done, and is a useful technique for the right reasons.

The cardinal rule is that the database(s) MUST be dismounted. The
simplest way is simply
to shut down Caché and make the copy. If that is not possible, you can
dismount the individual
databases, either by going to %SYS in the terminal and using ^DATABASE
utility, or by dismounting
from the System Management Portal.

Then copy the CACHE.DAT across and remount the databases. If you are
copying between Caché versions
the once the target database is mounted, you may also need to do and
upgrade ($System.OBJ.Upgrade() ) to
make the CACHE.DAT suitable for the new version.

If you do that, it works just fine. One other thing to look out for
is to NOT copy across any cache.lck lock
file that might be with the cache.data (rare, but it has happened when
someone has copied an entire folder
with the .DAT mounted, then subsequently dismounted it and copied) If
you copy the .lck it can forces
a READ-ONLY mount of the database. (Caché seems to recognise it as a
"foreign" lock.)

There are excellent third-party applications for handling code
deployment (see George James Software's VCm).
Or you can write your own using Caché ObjectScript to export/import
routines and classes, etc.
At the simplest level, you can just define Projects in Studio, include
all the Classes, Routines, CSPs that
you want to distribute, and export/import using Studio.

So lots of possibilities...

--
Ian Cargill

[Dawn Wolthuis]

This is sooooo helpful, Ian! Yes, we use George James vc/m which is
perhaps why we have been otherwise quite ignorant about the cache.dat
files to date. Without your information here we would also be testing
what happens with and without the lck file. Now we can skip that.

I do have a few follow-ups, feel free to ignore, and anyone else
should feel free to jump in.

Does the source cache.dat file need to be dismounted before copying it
as well as the destination?

When you say that obviously you would lose the data if upgrading live
sites, is that the case if you are copying the 'CODE' cache.dat file,
given that your namespace has separate DATA and CODE databases?

I also received some follow-up information offlist suggesting there
are new features in Cache' for minimal downtime when deploying a new
build (this is one of our two goals in fiddling with this now), which
might be why I thought I would be finding information I was not
finding.

We will also definitely be checking with George James, but because the
target namespace in our eventual setup will be on a production box
that would not have vc/m installed in the cache instance (at least
that was our intent), we are trying to learn what we can about these
underlying databases. To a pickie, the cache.dat file is mystery meat.
[I understand that very few humans would enjoy that last sentence, but
here's to those who would. cheers!] --dawn

[jimmy]

Hi Dawn,

Its not 100% neccessary to dismount the "from" database if its not in
use but its certainly best to do so in order to get a clean copy.

Once you have copied your cache.dat and you have performed any
necessary upgrades as per Ian's response also do the following:-

d $SYSTEM.SQL.Purge()
k ^mcq

This will remove any old SQL caches that are likely to cause problems
in you "to" cache.dat.

Regards

Jim

On Feb 26, 1:40 pm, Dawn Wolthuis <dw...@tincat-group.com> wrote:
> This is sooooo helpful, Ian! Yes, we use George James vc/m which is
> perhaps why we have been otherwise quite ignorant about the cache.dat
> files to date. Without your information here we would also be testing
> what happens with and without the lck file. Now we can skip that.
>
> I do have a few follow-ups, feel free to ignore, and anyone else
> should feel free to jump in.
>
> Does the source cache.dat file need to be dismounted before copying it
> as well as the destination?
>
> When you say that obviously you would lose the data if upgrading live
> sites, is that the case if you are copying the 'CODE' cache.dat file,
> given that your namespace has separate DATA and CODE databases?
>
> I also received some follow-up information offlist suggesting there
> are new features in Cache' for minimal downtime when deploying a new
> build (this is one of our two goals in fiddling with this now), which
> might be why I thought I would be finding information I was not
> finding.
>
> We will also definitely be checking with George James, but because the
> target namespace in our eventual setup will be on a production box
> that would not have vc/m installed in the cache instance (at least
> that was our intent), we are trying to learn what we can about these
> underlying databases. To a pickie, the cache.dat file is mystery meat.
> [I understand that very few humans would enjoy that last sentence, but
> here's to those who would. cheers!]  --dawn
>

> > For more options, visit this group athttp://groups.google.com/group/InterSystems-ZEN?hl=en
> > Zen Community Terms and Conditions:http://groups.google.com/group/InterSystems-ZEN/web/community-terms-a...

[Dawn Wolthuis]

Thumbs up. Thanks Jimmy. --dawn

> For more options, visit this group at http://groups.google.com/group/InterSystems-ZEN?hl=en
> Zen Community Terms and Conditions: http://groups.google.com/group/InterSystems-ZEN/web/community-terms-and-conditions

[icargill]

Dawn,

> Does the source cache.dat file need to be dismounted before copying it
> as well as the destination?

Opinions differ, but my opinion is yes, always. Apart from anything
else, it is safer to have a standard methond which will ALWAYS work.

One of my developers once took a copy of a 6Gb CACHE.DAT and spend
three hours backloading to our office for testing, only to find that
he hadn't dismounted it and it was corrupt.
Uou only need to do that once before you standardise on a more
reliable method!! ;-)

> When you say that obviously you would lose the data if upgrading live
> sites, is that the case if you are copying the 'CODE' cache.dat file,
> given that your namespace has separate DATA and CODE databases?

If your namespace does have separate DATA and CODE databases, then no.
But I'm not sure how many poeple work that way. We certainly don't.
Everything, CODE and DATA is in a single CACHE.DAT.

[icargill]

> d $SYSTEM.SQL.Purge()
> k ^mcq

Good suggestion from Jimmy. Please also note that if you are using CSP
and move the CACHE.DAT to a different namespace, you should also run

d $SYSTEM.OBJ.DeletePackage("csp")

This will purge all the compiled CSP page objects. Otherwise, when you
try to run the CSPs in the new namespace, you will conflict errors.

Ian

[Dawn Wolthuis]

This is enlightening in that I thought we were behind the times
working with a single cache.dat database per namespace, while doing
package mapping from code in a second one, so that we had a sort-of
split. This was what was recommended when originally it seemed that
the code and data split worked for mumps but not for pick (per our
original tests in that regard). We now have more information from
InterSystems and I have some confidence we can pull off the code/data
split, even if there are still some mysteries for us.

With our upgrade to 2010.2.1 we asked about how to get rid of the
downtime we have each time we deploy a new build of the software. We
currently move it using version control software which compiles it in
our live-lite alpha namespace, but so that we do not have to move each
item in order, we then compile them in order thereafter. This did not
work well for 24-7 access for a web site. We have a small subset of
our planned software written and what we have already keeps users off
the system for 1/2 hour, in part because the compile of our Zen page
template holds up all pages when it compiles, not just the one it is
compiling. So, to deploy a new build I would get up in the dead of
night or stay up late and hope that none of our alpha customers had
end-customers doing their web stuff at that time.

Yes, if and when we do this in the future, we will dismount the source
and target databases. It seems like it would be good to take down
apache for that too, so that our site is not accessible during that
time. So, this doesn't seem like a good deployment strategy, but it
might be good for moving data (although there might be better
approaches to that too), we shall see. Plenty to learn. Thanks!
--dawn

[icargill]

Dawn,

On Feb 27, 4:51 pm, Dawn Wolthuis <dw...@tincat-group.com> wrote:
> original tests in that regard). We now have more information from
> InterSystems and I have some confidence we can pull off the code/data
> split, even if there are still some mysteries for us.

I didn't mean to imply that you shouldn't split CODE & DATA, just that
we don't, for various reasons.

Apart from other considerations, it is only comparatively recently
that you could map class Packages. Previously you could only do it
with Routines and Globals. So to some extent, we are working with
legacy architecture! ;-)

You situation raises some interesting questions about use of Caché in
24/7 environtments.

.Ian

[jimmy]

Dawn,

I dont think that you are behind the times at all. If you have
different customers then it makes perfectly good sense to have
seperate cache.dat's. I'm sure that there are many other perfectly
valid reasons too.

With regards to your software release downtime:-

If you are compiling Classes then as long as you use the "fv"
parameters or the "Compile In Use" checkbox within Studio, Tools,
Options, Compiler, Flags and Optimization. then you should be able to
release classes whilst users are using your system as any process
using the class whilst you are compiling it will still use the old
version until they have finished with that class. I "believe" that
some people are wary of this feature but its always worked well for
me.

I'm not quite sure if there is a similiar scenario for csp's/zen's/
int's or mac's. Maybe someone could expand on this for me?

Jim

On Feb 27, 4:51 pm, Dawn Wolthuis <dw...@tincat-group.com> wrote:

> > For more options, visit this group athttp://groups.google.com/group/InterSystems-ZEN?hl=en
> > Zen Community Terms and Conditions:http://groups.google.com/group/InterSystems-ZEN/web/community-terms-a...
>
> --
> Dawn M. Wolthuis
>
> Take and give some delight today- Hide quoted text -
>
> - Show quoted text -