Draining a server

[Dennis McEntire]

I was wondering something, we are "retiring" a smaller server and
trying to clear it out, so:

1. I marked the server as "drain".
2. Within a few hours, the files on the box for the most part moved on
to other storage nodes.
3. At this point it appears that the drain is complete; meaning it
stopped removing files.

However, the problem I see is that there is still about 300+MB of
files on the box that was marked to be "drained."

The mogile files go into their own partition, /mogdata, off the main
HD. When I look at the partition, I see hundreds of folders with 3
digit numbers, but inside some of them are random files with
xxxxxx.fid, which are real files, I thought.

One thing to note:

For example, one file is called 0000432388.fid and sits in the
directory /mogdata/0/432/. This file contains data.

BUT, when I query the DB directly:

select * from file where fid = '432388' the results are null.
Actually, the results are NULL for ALL the test queries I made. I
wonder if I am "not allowed" to run a query like that, because the
actual file name will never match the FID field in the file table? Or
does the file name correspond to the fid field in the file table?

Anyway, before shutting down the retired box I wanted to run this by everyone,

Thanks,

Dennis McEntire

--

==============================
Dennis McEntire - gmail account
dmce...@gmail.com
==============================

[sanados]

the name of the file should be exact the fid in the database.
When not in file then this file should not be there imo.
Can you find this fid in file_to_delete* tables.
Or in the file_on table?

Basically when the fid is not in the file table it is not existant.

How this can happen i am not sure about.

lg
Jürgen

[dormando]

> I was wondering something, we are "retiring" a smaller server and
> trying to clear it out, so:
>
> 1. I marked the server as "drain".
> 2. Within a few hours, the files on the box for the most part moved on
> to other storage nodes.
> 3. At this point it appears that the drain is complete; meaning it
> stopped removing files.
>
> However, the problem I see is that there is still about 300+MB of
> files on the box that was marked to be "drained."
>
> The mogile files go into their own partition, /mogdata, off the main
> HD. When I look at the partition, I see hundreds of folders with 3
> digit numbers, but inside some of them are random files with
> xxxxxx.fid, which are real files, I thought.
>
> One thing to note:
>
> For example, one file is called 0000432388.fid and sits in the
> directory /mogdata/0/432/. This file contains data.
>
> BUT, when I query the DB directly:
>
> select * from file where fid = '432388' the results are null.
> Actually, the results are NULL for ALL the test queries I made. I
> wonder if I am "not allowed" to run a query like that, because the
> actual file name will never match the FID field in the file table? Or
> does the file name correspond to the fid field in the file table?

SELECT * FROM file WHERE fid = 432388; is correct. Like sanados said, also
try the file_on, file_to_delete(2), tables, etc. Though I doubt they'll be
in the file_on table, since that's the one drain works against.

If they're not in any of those tables, then the files have been deleted
already, but had failed to be deleted by mogilefs for some reason.

-Dormando

[Jon]

In my experience you will flat out loose those files not moved to
other servers using drain.. and I do not currently know how to correct
the problem.

I attempted to increase replication count for all my classes, and
reset fsck, and run again. But that did not solve my problem. :(

[Kiall Mac Innes]

I've ran into this issue before...

I believe this is caused when a file insert fails fails to send the write close to the tracker - your app should have treated this as a failure and retried.

That may not be the only cause though!

Kiall

On Nov 13, 2009 6:24 a.m., "sanados" <san...@failure.at> wrote:


the name of the file should be exact the fid in the database.
When not in file then this file should not be there imo.
Can you find this fid in file_to_delete* tables.
Or in the file_on table?

Basically when the fid is not in the file table it is not existant.


How this can happen i am not sure about.


lg
Jürgen

Dennis McEntire wrote: > > I was wondering something, we are "retiring" a smaller server and > t...

[Dennis McEntire]

On this drain issue, the table file_on_delete is empty, but
file_to_delete2 has about 21 records in it that don't change. Not sure
if it matters, but file_on_corrupt contains 24 records that also don't
change.

So far, the 300MB+ of files still on the box that was drained still
exist. The FIDs of those files, so far (after checking almost 40+)
don't exist in the "file" tables.

At this point I guess the box will be shut down this week and taken
care of unless the orphan files should be of concern.

Dennis

--

[dormando]

If you log into your database (slave if you have one) and run 'select
count(*) from file_on where devid = ?' where ? == the device you
drained... If that's zero, you're good. If it's nonzero, there're files
that aren't draining for some reason.

> > J?rgen

[Dennis McEntire]

OK, thanks, that's a good idea.

select * from file_on where devid = 4 returns 693 rows, ugh!

I wonder how I can drain those files to ensure we don't lose anything.
Should I go ahead and mark it "down" or "dead" instead to see if it
replicates (moves) those 693 files to another dev?

Thanks,

Dennis

[dormando]

Usually I take a quick hand look at a few of those fids. Resolve them to
domain, keys, then try to fetch with mogtool. If you telnet to a tracker
and run '!watch', then stop drain, start drain on that device you might
also see why it can't drain those.

Odds are those files are gone for some reason. Likely faulty uploads with
no other sources.

If you mark it as 'dead' it will reap all of those rows for sure (which is
something you need to do before you remove the device from the cluster,
either way), but it won't try to replicate from those sources. Drain is
like dead except it's reversable and it can use the draining device as a
read source.

[Dennis McEntire]

Taking a quick look at the box being drained, I found some files in
the /mogdata/dev4/0/000/xxx directories.

I take those FIDs and "select * from file_on where fid = 'xxxx'" and I
get two rows for each one. The file exists on both dev1 and dev4 (the
draining one). I wonder how I can force mogile to replicate those
files on to a different box (what drain is supposed to do)?

If I mark the device dead, I wonder if the results of the above
queries will not show dev4 any more?

Maybe I should mark the device "alive", wait a few minutes, and then
mark it "drain" again ? Is that worth a shot?

Dennis

[dormando]

Give it a shot.... make sure you're watching a tracker with !watch to see
if there's some reason why it's not getting those.

You might also, inbetween marking the device as alive and before marking
it drain again, restart all your replicate workers.
!want 0 replicate
(wait until they're dead)
!want N replicate

just in case you're hitting some particular bug

[Kiall Mac Innes]

If you can get the key's for them...

mogtool extract (key) (key).tmp && mogtool inject (key) (key).tmp

This will replace the file and issue it a new fid...

Kiall

On Nov 16, 2009 9:37 p.m., "Dennis McEntire" <dmce...@gmail.com> wrote:


Taking a quick look at the box being drained, I found some files in
the /mogdata/dev4/0/000/xxx directories.

I take those FIDs and "select * from file_on where fid = 'xxxx'" and I
get two rows for each one. The file exists on both dev1 and dev4 (the
draining one). I wonder how I can force mogile to replicate those
files on to a different box (what drain is supposed to do)?

If I mark the device dead, I wonder if the results of the above
queries will not show dev4 any more?

Maybe I should mark the device "alive", wait a few minutes, and then
mark it "drain" again ? Is that worth a shot?

Dennis

On Mon, Nov 16, 2009 at 1:23 PM, dormando <dorm...@rydia.net> wrote: > > Usually I take a quick ...

--

============================== Dennis McEntire - gmail account dmce...@gmail.com ================...

[Kiall Mac Innes]

Don't forget to give the files the right class with the inject BTW

On Nov 16, 2009 9:40 p.m., "Kiall Mac Innes" <ki...@managedit.ie> wrote:

If you can get the key's for them...

mogtool extract (key) (key).tmp && mogtool inject (key) (key).tmp

This will replace the file and issue it a new fid...

Kiall

> > On Nov 16, 2009 9:37 p.m., "Dennis McEntire" <dmce...@gmail.com> wrote: > > > Taking a quick ...

[Dennis McEntire]

OK, so I marked the device alive, waited a min., then marked it drain
again. The telnet session shows (!watching the tracker) some errors
like this:

:: [delete(3183)] Error: unlink failure:
http://192.168.0.26:7500/dev4/0/000/244/000024489
8.fid: HTTP code 400

IP 0.26 is dev4, the box we are trying to drain.

Does this offer anything?

Dennis

--

==============================

[Kiall Mac Innes]

Http 400 is access denied...what happens when you try access the file direct from a browser?

A chmod -T 777 /var/mogdata might let the drain complete (normally this is a very bad idea - but since the server is being decomissioned...)

On Nov 16, 2009 10:26 p.m., "Dennis McEntire" <dmce...@gmail.com> wrote:


OK, so I marked the device alive, waited a min., then marked it drain
again. The telnet session shows (!watching the tracker) some errors
like this:

:: [delete(3183)] Error: unlink failure:
http://192.168.0.26:7500/dev4/0/000/244/000024489
8.fid: HTTP code 400

IP 0.26 is dev4, the box we are trying to drain.

Does this offer anything?

Dennis

On Mon, Nov 16, 2009 at 1:42 PM, Kiall Mac Innes <ki...@managedit.ie> wrote: > Don't forget to giv...

[Kiall Mac Innes]

Grr at phone auto correction ... chmod -R...

On Nov 16, 2009 10:26 p.m., "Dennis McEntire" <dmce...@gmail.com> wrote:


OK, so I marked the device alive, waited a min., then marked it drain
again. The telnet session shows (!watching the tracker) some errors
like this:

:: [delete(3183)] Error: unlink failure:
http://192.168.0.26:7500/dev4/0/000/244/000024489
8.fid: HTTP code 400

IP 0.26 is dev4, the box we are trying to drain.

Does this offer anything?

Dennis

On Mon, Nov 16, 2009 at 1:42 PM, Kiall Mac Innes <ki...@managedit.ie> wrote: > Don't forget to giv...

[Dennis McEntire]

Oh boy! I think I have some hardware problems. I did the chmod 777 to
the mogdata directory and there are quite a few "Input/Output Error"
messages appearing. Some other errors that show up are "Stale NFS file
handle" whatever that means.

Well, actually, there are 600+ files on the system and I only got 21
errors like the above.

Dennis

[Kiall Mac Innes]

Oh boy is right :)

What happened with the remainder of the files? (The ones that didn't error)

Have they drained successfully? Are the 400 errors only on those 21?

A mogtool extract / inject will fix those 21 assuming there stored elsewhere on the system... marking the device as dead should also get them replicated to new devices (again - assuming there safe elsewhere).

Thanks,
Kiall

2009/11/17 Dennis McEntire <dmce...@gmail.com>

[dormando]

How is this set up? You're mounting something over NFS? Did you tell me
this before and I already forgot? :P

I think kiall is about right. You can either just mark it dead (Which will
force mogilefs to re-replicate from elsewhere and give up on the old
references), or you can reinject those dead files.

On Mon, 16 Nov 2009, Dennis McEntire wrote:

[Dennis McEntire]

Nope, no NFS mounts actually, so I am not sure where the NFS error comes from.

I also believe that there is at least ONE copy of each file on dev1,
my random DB queries of FIDs on dev4 (draining) confirms this. I could
be wrong, but so far there is at least one copy of each file on
another live box. Note that this is still a completely isolated test
environment and does not actually host any live files, so data loss is
not an issue (at this time) since we don't care about the files. But
if this were a live situation this might be a big problem if we can't
drain a server completely for replacing.

I will let the boxes sit as they are for the night and try a "mark
dead" on the device tomorrow to see if it replicates files that were
on the dev4 box.

Thanks,

Dennis

[Kiall Mac Innes]

Yea - it sounds like a hardware issue rather than a mfs issue.. but mfs should be handling the error IMO..

Wondering if a file system fsck will help?

On Nov 17, 2009 2:15 a.m., "Dennis McEntire" <dmce...@gmail.com> wrote:

Nope, no NFS mounts actually, so I am not sure where the NFS error comes from.

I also believe that there is at least ONE copy of each file on dev1,
my random DB queries of FIDs on dev4 (draining) confirms this. I could
be wrong, but so far there is at least one copy of each file on
another live box. Note that this is still a completely isolated test
environment and does not actually host any live files, so data loss is
not an issue (at this time) since we don't care about the files. But
if this were a live situation this might be a big problem if we can't
drain a server completely for replacing.

I will let the boxes sit as they are for the night and try a "mark
dead" on the device tomorrow to see if it replicates files that were
on the dev4 box.

Thanks,

Dennis

On Mon, Nov 16, 2009 at 5:32 PM, dormando <dorm...@rydia.net> wrote: > How is this set up? You'r...

--

============================== Dennis McEntire - gmail account dmce...@gmail.com ================...

[dormando]

mogile err's on the side of leaving shit in place if it does something
like that... Then you can look at the error, go "oh fux", and mark the
device as "dead" anyway.

fsck may or may not fix it.

[a_mogilefs_user]

I have seen that error on multiple Fedora machines related to the rpm
db.
Nothing to do with NFS, not on a partition that had ever been shared
via NFS.
It is a filesystem corruption error that causes it and I was able to
fix it by forcing fsck, IIRC.

Given the IO errors, I would dd or ddrescue (there are 2 different dd
rescues) the partition to a different drive before fscking.

On Nov 16, 5:32 pm, dormando <dorma...@rydia.net> wrote:
> How is this set up? You're mounting something over NFS? Did you tell me
> this before and I already forgot? :P
>
> I think kiall is about right. You can either just mark it dead (Which will
> force mogilefs to re-replicate from elsewhere and give up on the old
> references), or you can reinject those dead files.
>
> On Mon, 16 Nov 2009, Dennis McEntire wrote:
> > Oh boy! I think I have some hardware problems. I did the chmod 777 to
> > the mogdata directory and there are quite a few "Input/Output Error"
> > messages appearing. Some other errors that show up are "Stale NFS file
> > handle" whatever that means.
>
> > Well, actually, there are 600+ files on the system and I only got 21
> > errors like the above.
>
> > Dennis
>
> > On Mon, Nov 16, 2009 at 3:46 PM, Kiall Mac Innes <ki...@managedit.ie> wrote:
> > > Http 400 is access denied...what happens when you try access the file direct
> > > from a browser?
>
> > > A chmod -T 777 /var/mogdata might let the drain complete (normally this is a
> > > very bad idea - but since the server is being decomissioned...)
>

> > > On Nov 16, 2009 10:26 p.m., "Dennis McEntire" <dmcent...@gmail.com> wrote:
>
> > > OK, so I marked the device alive, waited a min., then marked it drain
> > > again. The telnet session shows (!watching the tracker) some errors
> > > like this:
>
> > > :: [delete(3183)] Error: unlink failure:
> > >http://192.168.0.26:7500/dev4/0/000/244/000024489
> > > 8.fid: HTTP code 400
>
> > > IP 0.26 is dev4, the box we are trying to drain.
>
> > > Does this offer anything?
>
> > > Dennis
>
> > > On Mon, Nov 16, 2009 at 1:42 PM, Kiall Mac Innes <ki...@managedit.ie> wrote:
> > >> Don't forget to giv...
>
> > > --
>
> > > ==============================
>

> > > Dennis McEntire - gmail account dmcent...@gmail.com

>
> > > ==============================
>
> > --
>
> > ==============================
> > Dennis McEntire - gmail account

> > dmcent...@gmail.com
> > ==============================

[Kiall Mac Innes]

"ddrescue" - Something tell's me this version would have been helpful to me over the last few days ... ;)

Thanks,
Kiall

2009/11/19 a_mogilefs_user <rob...@gmail.com>