post command for gnt-backup

[tschend]

Hi,

we are exporting our instance with gnt-backup export and are using the
snf-image os defs.

In our setup we run flashcache to get better disk perfromance.
The problem is that snf-image exports with dd and the sequential IO
detection from flashcache cannot detect this because the other
machines on the host are busy at the same time and create IO.

We can blacklist the dd pid. Currently we run a script which looks for
dd in the process list and then blacklists it.

Has anyone a idea how we can solve this smarter?

We though about a wrapper but it is not possible to start an process
with a specific pid.

Best Regards
Thomas

[Thomas Thrainer]

Hi,

For OS definition specific questions it is usually better to ask directly at a mailing list of the provider of the definitions.

But for your case, wouldn't it be possible to replace the dd command on your nodes with a small wrapper script?

You could use dpkg-divert [0] (if you are using Debian, or something similar for your distribution) to rename the original dd to dd_orig for example. Then you create a small shell script called dd, which essentially calls the original dd in background, captures the PID and blacklists it, and then waits for the PID. Something like the following should work (untested):

#!/bin/bash

/bin/dd_orig $@

DD_PID=$!

# do whatever is required to blacklist $DD_PID

wait $DD_PID

exit $?

Cheers,

Thomas

[0] http://manpages.debian.net/cgi-bin/man.cgi?query=dpkg-divert

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

Google Germany GmbH

Dienerstr. 12

80331 München

Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg
Geschäftsführer: Graham Law, Katherine Stephens

[tschend]

Hello Thomas,

I do not think this is os definition specific but i will post on the
mailinglist.
thanks for the hint. We are using debian squeeze.

On 23 Mai, 15:17, Thomas Thrainer <thoma...@google.com> wrote:
> Hi,
>
> For OS definition specific questions it is usually better to ask directly
> at a mailing list of the provider of the definitions.
>
> But for your case, wouldn't it be possible to replace the dd command on
> your nodes with a small wrapper script?
> You could use dpkg-divert [0] (if you are using Debian, or something
> similar for your distribution) to rename the original dd to dd_orig for
> example. Then you create a small shell script called dd, which essentially
> calls the original dd in background, captures the PID and blacklists it,
> and then waits for the PID. Something like the following should work
> (untested):
>
> #!/bin/bash
>
> /bin/dd_orig $@
> DD_PID=$!
> # do whatever is required to blacklist $DD_PID
> wait $DD_PID
> exit $?
>

Looks good and i tough about execalty this but will the script not
wait till the dd job is done and will blacklist the pid after this?
Running the job in the background with & will not work.

Have you any ideas?

> Cheers,
> Thomas
>

Regards

Thomas

> [0]http://manpages.debian.net/cgi-bin/man.cgi?query=dpkg-divert
>
>
>
>
>
>
>
>
>
> On Thu, May 23, 2013 at 11:48 AM, tschend <thomas.sch...@gmail.com> wrote:
> > Hi,
>
> > we are exporting our instance with gnt-backup export and are using the
> > snf-image os defs.
>
> > In our setup we run flashcache to get better disk perfromance.
> > The problem is that snf-image exports with dd and the sequential IO
> > detection from flashcache cannot detect this because the other
> > machines on the host are busy at the same time and create IO.
>
> > We can blacklist the dd pid. Currently we run a script which looks for
> > dd in the process list and then blacklists it.
>
> > Has anyone a idea how we can solve this smarter?
>
> > We though about a wrapper but it is not possible to start an process
> > with a specific pid.
>
> > Best Regards
> > Thomas
>
> --

> Thomas Thrainer | Software Engineer | thoma...@google.com |

[Thomas Thrainer]

Sorry, I actually forgot the &...

Here's the script (I called it dd_test and didn't rename the original dd):

#!/bin/bash

/bin/dd $@ &

DD_PID=$!

# do whatever is required to blacklist $DD_PID

echo $DD_PID

trap "kill $DD_PID" SIGINT

wait $DD_PID

exit $?

This starts dd in the background, then prints the PID of the dd process, and then waits for the dd process to finish. I also installed a SIGINT handler, so you can quit the script with ^C. You might want to add more signal handlers if you require them. Doesn't that work for you?

--

Thomas Thrainer | Software Engineer | thom...@google.com | 

[tschend]

Hi Thomas,

ok i will try that and report back if it works!
Thanks!

Thomas

On 23 Mai, 16:50, Thomas Thrainer <thoma...@google.com> wrote:

[tschend]

Hi,

works fine on the console but not with the export.

Fri May 24 14:37:04 2013 - WARNING: export 'export-
disk0-2013-05-24_14_37_03-HNaRFt' on sgntko21 failed: Exited with
status 1
Fri May 24 14:37:04 2013 snapshot/0 failed to send data: Exited with
status 1 (recent output: dd: 0 bytes (0 B) copied, 8.404e-06 s, 0.0
kB /s\n/bin/dd.real: writing `standard output': Broken
pipe\n1+0 records in\n0+0 records out\n0 bytes (0 B) copied, 0.0566731
s, 0.0 kB/s)
Fri May 24 14:37:04 2013 Removing snapshot of disk/0 on node
sgntko21.iaas.cgm.ag
Fri May 24 14:37:05 2013 Finalizing export on sgntko21.iaas.cgm.ag
Failure: command execution error:
Export failed, errors in disk export: disk(s) 0

in the export script is the dd command only specified with

dd if="$blockdev" bs=4M

Maybe someone has an idea.

Regards
Thomas

[Thomas Thrainer]

Hi,

The problem is that the script expects dd to write to stdout. As dd runs in the background, no output is generated...

You can try to not run dd in the background, but to first gather the pid of the script you write, blacklist this pid, and then replace the process with a dd instance.

Here's how you would do that:

#!/bin/bash

DD_PID=$$

echo $DD_PID

# do whatever you want to blacklist $DD_PID

exec /bin/dd $@

As the original dd process replaces the current one, all file handles (including stdin/stdout/stderr) are inherited, and the export script should work.

Cheers,

Thomas

Thomas Thrainer | Software Engineer | thom...@google.com | 

[tschend]

Hi Thomas,

This did the trick! Very nice.

I need to look if it makes a differnce invoking bash when exporting
but i do not think so.

Thanks a lot.

Regards
Thomas

On 24 Mai, 15:00, Thomas Thrainer <thoma...@google.com> wrote:
> Hi,
>