SyncIQ Reporting

[Donald King]

Has anyone come up with a method to automatically generate reporting for SyncIQ jobs?  I can log into node 1 and run the command "isi sync reports list --reports-per-policy 1" and it gives me a nice little snapshot of the previous night's run.  But how can I get the cluster to do this automagically and email it to me?  Is cron an option?  I am not an experienced scripter - any help is appreciated!

Thanks,

-Don

[Chris Pepper]

Donald,

You can put cluster cron jobs into /etc/mcp/override/crontab.smbtime. The files format includes the username to execute the command before the command itself. Note that *every* node runs this crontab, so you’ll want to avoid having every single node generate and run the same report. If it’s a lightweight job use isi_ropc to run only on the lowest-numbered (effective ‘head’) node. If you have a bunch of heavy-duty jobs you might want to assign them to individual nodes inside the job scripts…

Chris

> solisi-1# isi_ropc --help
> Usage: isi_ropc [OPTION]... [COMMAND]
>
> Runs the passed command once per cluster no matter how many nodes
> it's run on. Useful for things like the newsyslog-on-ifs crontab
> entry, which wants to be in each nodes' crontab but should only be
> run once per cluster. By default, only run the node on the lowest
> devid.
>
> Options:
> -h, --help Print this message and exit
> -s, --hash Hash the command and use it as a lookup into the
> device list, effectively running the command on a
> random node, rather than running on the lowest devid.

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Richard Kunert]

Or you can just have one node run them by putting the command into /etc/local/crontab.local on that node. Not as robust in case of node failure, but I’m not terribly concerned about my little scripts running if I actually have a node down.

# send isi status to sysadmins every day at 9 AM
0 9 * * * root /usr/bin/isi stat | mail -s "Isilon status" sysa...@yourcompany.com

But first I would try:
echo Test | mail -s Test your...@yourcompany.com
...at the command line to make sure mail is working, depending on your particular email and dns setup you might need to modify the Isilon sendmail configuration to get a routable sending address. Which is another topic entirely.

Richard

[Donald King]

Thanks guys this is exactly what I needed.  Sendmail of course does not work by default, I am off to research that.

-Don

[Jason Davis]

I've been working with yet another open source dashboard building tool: Cyclotron. 

https://github.com/ExpediaInceCommercePlatform/cyclotron

The nice thing about this is that you can have it consume just about anything without having a backing TSDB (Graphite/InfluxDB). To that end I got an example dashboard built that is a SyncIQ report consuming PAPI.

​

The running job here has a weird last success date but this is due to a blank date-time stamp being passed through. 

[Richard Kunert]

Check /var/log/maillog for the error. It’s likely sending mail, but probably your mailer is rejecting it because it’s not sending from a legal email address. You fix that by going to /etc/mail, doing “make”, which will generate hostname.mc and hostname.submit.mc, then adding some combination of SMART_HOST, MASQUERADE_AS and MASQUERADE_DOMAIN in the hostname.submit.mc file, using the strange syntax with the “dnl” entries. Then do a "make all install restart" in /etc/mail. Or, depending on your situation you might have access to a mailer that you can configure to accept the mail as is. I really don’t like sendmail. I don’t think Isilon actually uses it, it’s just part of the FreeBSD distribution OneFS is based on.

This would need to be done on every node that needs to send mail. In my case I just did node 1, where my scripts run.

Looks like I’ve answered this before: https://groups.google.com/forum/#!topic/isilon-user-group/yPf439SZEiw

Richard

[Donald King]

Hi Richard, here is the logging from the node:

from /var/log/messages

2016-03-22T09:25:09-04:00 <2.2> ishtar-2(id17) sendmail[81680]: u2MDP8G3081680: SYSERR(root): null destination for [IPC] mailer

From /var/log/maillog

2016-03-22T09:25:09-04:00 <2.6> ishtar-2(id17) sendmail[81680]: u2MDP8G3081680: to=MAILER-DAEMON, delay=00:00:00, mailer=local, pri=31068, dsn=5.1.1, stat=User unknown

This looks like a sendmail config issue to me. We allow non-legal email address senders (the Isilon GUI sends alert emails from an invalid address).  

[Richard Kunert]

Don,

Is your recipient valid? I can produce those exact errors by trying to send to a local user (just username, no @ sign) that doesn’t exist on the Isilon.

Try setting up the SMART_HOST. If there isn’t a smart host normally a mail server will look at the mx records to see what the mx is for the recipient, perhaps in your environment that fails. Try doing “dig -t mx yourdomain.com” on the Isilon to see if it has access to that record.

I’m hardly a sendmail expert, the Isilon is the only system we have that runs it. Usually we use Postfix, which is easier to configure.

[Dan Pritts]

I just use:

/usr/sbin/sendmail -f $FROM -t

the -t means to pick up the to: address from the headers.

So this shell snippet works:

cat <<EOF | /usr/sbin/sendmail -f isi...@example.com -t
From: isi...@example.com
To: foo...@example.com
Subject: nightly isilon report

This is a report
EOF

Of course, the cluster has to be able to reach the SMTP server for the domain in the To: line.  If that's not the case, you will have to modify the sendmail config to define a smarthost, or similar.

danno

Donald King

March 22, 2016 at 9:28 AM

Hi Richard, here is the logging from the node:

from /var/log/messages

2016-03-22T09:25:09-04:00 <2.2> ishtar-2(id17) sendmail[81680]: u2MDP8G3081680: SYSERR(root): null destination for [IPC] mailer

From /var/log/maillog

2016-03-22T09:25:09-04:00 <2.6> ishtar-2(id17) sendmail[81680]: u2MDP8G3081680: to=MAILER-DAEMON, delay=00:00:00, mailer=local, pri=31068, dsn=5.1.1, stat=User unknown

This looks like a sendmail config issue to me. We allow non-legal email address senders (the Isilon GUI sends alert emails from an invalid address).  

On Monday, March 21, 2016 at 4:35:55 PM UTC-4, Richard wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Richard Kunert

March 21, 2016 at 4:35 PM

Check /var/log/maillog for the error. It’s likely sending mail, but probably your mailer is rejecting it because it’s not sending from a legal email address. You fix that by going to /etc/mail, doing “make”, which will generate hostname.mc and hostname.submit.mc, then adding some combination of SMART_HOST, MASQUERADE_AS and MASQUERADE_DOMAIN in the hostname.submit.mc file, using the strange syntax with the “dnl” entries. Then do a "make all install restart" in /etc/mail. Or, depending on your situation you might have access to a mailer that you can configure to accept the mail as is. I really don’t like sendmail. I don’t think Isilon actually uses it, it’s just part of the FreeBSD distribution OneFS is based on.

This would need to be done on every node that needs to send mail. In my case I just did node 1, where my scripts run.

Looks like I’ve answered this before: https://groups.google.com/forum/#!topic/isilon-user-group/yPf439SZEiw

Richard

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Donald King

March 21, 2016 at 11:19 AM

Thanks guys this is exactly what I needed.  Sendmail of course does not work by default, I am off to research that.

-Don

On Monday, March 21, 2016 at 10:12:20 AM UTC-4, Richard wrote:

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Richard Kunert

March 21, 2016 at 10:12 AM

Or you can just have one node run them by putting the command into /etc/local/crontab.local on that node. Not as robust in case of node failure, but I’m not terribly concerned about my little scripts running if I actually have a node down.

# send isi status to sysadmins every day at 9 AM
0 9 * * * root /usr/bin/isi stat | mail -s "Isilon status" sysa...@yourcompany.com

But first I would try:
echo Test | mail -s Test your...@yourcompany.com
...at the command line to make sure mail is working, depending on your particular email and dns setup you might need to modify the Isilon sendmail configuration to get a routable sending address. Which is another topic entirely.

Richard

Chris Pepper

March 21, 2016 at 9:35 AM

Donald,

	You can put cluster cron jobs into /etc/mcp/override/crontab.smbtime. The files format includes the username to execute the command before the command itself. Note that *every* node runs this crontab, so you’ll want to avoid having every single node generate and run the same report. If it’s a lightweight job use isi_ropc to run only on the lowest-numbered (effective ‘head’) node. If you have a bunch of heavy-duty jobs you might want to assign them to individual nodes inside the job scripts…

Chris

solisi-1# isi_ropc --help
Usage: isi_ropc [OPTION]... [COMMAND]

Runs the passed command once per cluster no matter how many nodes
it's run on. Useful for things like the newsyslog-on-ifs crontab
entry, which wants to be in each nodes' crontab but should only be
run once per cluster. By default, only run the node on the lowest
devid.

Options:
   -h, --help           Print this message and exit
   -s, --hash           Hash the command and use it as a lookup into the
                        device list, effectively running the command on a
                        random node, rather than running on the lowest devid.

On Mar 21, 2016, at 9:25 AM, Donald King <donald...@gmail.com> wrote:

Has anyone come up with a method to automatically generate reporting for SyncIQ jobs?  I can log into node 1 and run the command "isi sync reports list --reports-per-policy 1" and it gives me a nice little snapshot of the previous night's run.  But how can I get the cluster to do this automagically and email it to me?  Is cron an option?  I am not an experienced scripter - any help is appreciated!

Thanks,
-Don

-- 
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
Dan Pritts
ICPSR Computing & Network Services
University of Michigan
+1 (734) 615-9529

[Erik Weiman]

Something to keep in mind is that if / when you move to OneFS 8.0 we are based on FreeBSD 10 so you may need to re-invent some of these little tweaks as the way things work may have changed. 

For example sendmail / mailx / mailwrapper doesn't exist in 8.0. 

This should send an email using the Isilon mail system rather than relying on configuring another mail applicaiton on the cluster. There is a python script you can use to do this. 

v8000-1# sendml.py -f root -t us...@company.com -s "Sending Mail Test" -b "Status of cluster" -a /var/log/isi_migrate.log

I copied this to "/usr/sbin/sendml.py" on my virtual node and made it executable, as I don't have mail configured on my test cluster I can't show any example output but this should work on 7.x and 8.0. 
It would appear that using the user of 'root' isn't valid based on /var/log/messages so you may want to use a valid email address. 

2016-03-22T12:55:32-05:00 <1.4> v8000-1 python: Sender email address "root" is invalid. Using "ro...@v8000-1.isilon.com".

As I'm not expecting attachments to work in this case here is a cut & paste of the script. You'd have to place it on any node that you want to be able to send custom emails from the cluster. 

I take no credit for this script and I'm not sure who made it either. 
It should be 64 lines long according to xcode.

********************************************************

#!/usr/bin/env python

#

# Helper script to send mail using the Isilon libraries

#

import sys

from optparse import OptionParser

import socket

from isi.app.lib.emailer import Emailer, EmailAttachmentFromFile

# Emailer.send_email(to_addresses(list), message(string), from_address=None(string),  subject=None(string),

#                    attachments=None(list), headers=None(list), charset="us-ascii"(string))

def main():

    usage = '%prog: [-f sender] -t recipient [ -t recipient ... ] [-s subject] [-b body] [-a attachment]'

    argparser = OptionParser(usage = usage, description = 'Send email from a cluser node')

    argparser.add_option('-f', '--from', '--sender', dest='sender',

help="email sender (From:)")

    argparser.add_option('-t', '--to', '--recipients', dest='recipients',

action = 'append', help="email recipient (To:)")

    argparser.add_option('-s', '--subject', dest='subject',

help="email subject (Subject:)")

    argparser.add_option('-b', '--body', dest='body',

help="email body (default stdin)")

    argparser.add_option('-a', '--attachment', '--file', dest='attfiles',

action = 'append', help="attachment filename")

    (options, args) = argparser.parse_args()

    if options.sender is None:

fqdn = socket.getfqdn()

sender = "root@%s" % fqdn

    else:

       sender = options.sender

    if options.recipients is None:

argparser.error("Unable to send mail without at least one recipient");

sys.exit(1);

    else:

recipients = options.recipients

    if options.subject is None:

    subject = 'No subject specified'

    else:

subject = options.subject

    if options.body is None:

lines = sys.stdin.readlines()

body = ''.join(lines)

    else:

body = options.body

    if options.attfiles is None:

atts = None

    else:

atts = []

for attfile in options.attfiles:

   att = EmailAttachmentFromFile(attfile)

   atts.append(att)

    try:

        Emailer.send_email(recipients, body, sender, subject, attachments = atts)

    except:

        print "Error sending email."

        sys.exit(1)

    sys.exit(0)

if __name__ == "__main__":

    main()

********************************************************

--

Erik Weiman

[Richard Kunert]

Nice. The script is set up to read the body from standard input, so piping works the same as with mailx. So this works (I called it sendmail.py).

# send isi status to sysadmins every day at 9 AM

0 9 * * * root /usr/bin/isi stat | sendmail.py -s "Isilon status" -t sysa...@mycompany.com

Sender defaults to "root@%s" % fqdn”, which works in my case. Logging goes to /var/log/messages. No dealing with sendmail configuration is a plus for me.

I’ve gotten very accustomed to “the way things work” changing, with all of the (sometimes arbitrary seeming) changes in the parameters for the isi commands. I sometimes wish the old commands could be kept for backwards compatibility, breaking scripts could be avoided. Documenting the changes wouldn’t hurt either.

Richard

> --
> Dan Pritts
> ICPSR Computing & Network Services
> University of Michigan
> +1 (734) 615-9529
>
>
> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
>

[Andrew MacKay]

Our product offers RPO reporting for SyncIQ with automated reports and graphing and trending.  I attached sample report measuring SyncIQ target RPO values vs last 30 days and last 24 hours.   If anyone wants a demo let me know.   We sell this through EMC exclusively.  http://site.superna.net/products/eyeglass-isilon/feature-matrix

Regards
Andrew

[Jeff]

Jason, 

I'm very interested in what you've done with Cyclotron and want to try that in my shop.  Care to share the pitfalls and triumphs of what you've done.  Can it work with data from SmartQuotas as well?