Help with advisory quotas

[Michael Coffman]

Hello,

I am a fairly new to using the isilon's and this is my first message
to this list.

I am trying to understand the advisory quotas and notification and am
having some problems. I have set up hard quotas and they seem to work
fine in that they show available space as I would expect and when full
don't allow writes.

I want to be able to set up some advisory quotas for notification purposes only.

I used the following command...
isi quota create \
--directory \
--path=/ifs/projects/testing2 \
--accounting \
--advisory-threshold=1G \
--notify-for-advisory-threshold \
--notify-on-threshold-exceeded-once-per=1h \
--notify-email-to=my-email-address

And it seems to create it OK.

# isi quota ls -v --path=/ifs/projects/testing2
Type Path Policy Snap Usage
-------------------- ------------------------------- ----------- ----- --------
directory /ifs/projects/testing2 accounting no 42B
[advisory-threshold] ( 1.0G)
[notify-on-threshold-exceeded-once-per]
(1 h, email <michael...@avagotech.com>)
[advisory-threshold-exceeded] (no)
[usage-with-no-overhead] ( 42B)
[usage-with-overhead] ( 5.5K)
[usage-inode-count] (3)

But when I go over the limit, nothing seems to happen. It doesn't
even register in the quota
listing as being over. It shows the usage as over, but not the
threshold as exceeded.

# isi quota ls -v --path=/ifs/projects/testing2
Type Path Policy Snap Usage
-------------------- ------------------------------- ----------- ----- --------
directory /ifs/projects/testing2 accounting no 2.0G
[advisory-threshold] ( 1.0G)
[notify-on-threshold-exceeded-once-per]
(1 h, email <michael...@avagotech.com>)
[advisory-threshold-exceeded] (no)
[usage-with-no-overhead] ( 2.0G)
[usage-with-overhead] ( 2.7G)
[usage-inode-count] (4)

What am I missing?

Thanks.
--
-MichaelC

[Jerry Uanino]

Hm. Have you tried going through the GUI and setting the global default notifications.  Then leave off --notify-email-to and see if it makes a difference?  Just a thought.  We've only ever used the global notification so I've never actually set the per quota email address.

--
-MichaelC

--
You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

[Jason Davis]

Agreed, I do global notifications using a semi fancy HTML message body and when a quota notification is triggered its fairly instantaneous.

I Wonder if you are seeing any errors in logs centered around the email notification?

First guess would be to look in /var/log/message but I bet there might be a more specific log... Not in front of a computer right now to verify.

[Matthew Dey]

Not sure what version you are running.  After we upgraded to 7.0 something happened during the upgrade and isi_quota_notify_d was unable to start backup properly.  Look in /var/log/messages for isi_quota_notify_d.  If you have the same problem I had your log will have lots of messages about it not being able to start up.  Without that process you will never get any notifications.

It also seems odd that it's not showing the advisory threshold as being over.  That seems like a bug.

Matt Dey
203-428-4166

[Peter Serocka]

Michael,

in your example the policy was set to accounting.

>
> # isi quota ls -v --path=/ifs/projects/testing2
> Type Path Policy Snap Usage
> -------------------- ------------------------------- ----------- ----- --------
> directory /ifs/projects/testing2 accounting no 2.0G
>

> [advisory-threshold] ( 1.0G)

> [advisory-threshold-exceeded] (no)


As Matt pointed out, the adv threshold should be already reported
as exceeded=yes --

Can you set the policy to enforcing (isi quota mod ... --enforcement)
to see wether this affects the registration of exceeded adv thresholds?

Peter

> --
> You received this message because you are subscribed to the Google Groups "Isilon Technical User Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to isilon-user-gr...@googlegroups.com.
> For more options, visit https://groups.google.com/groups/opt_out.
>
>
>

Peter Serocka
CAS-MPG Partner Institute for Computational Biology (PICB)
Shanghai Institutes for Biological Sciences (SIBS)
Chinese Academy of Sciences (CAS)
320 Yue Yang Rd, Shanghai 200031, China
pser...@picb.ac.cn

[Michael Coffman]

Thanks for all the input. I have tried falling back to a global
notification. I also converted from accounting to enforcement when
then immediately showed that the quota was being violated. I also
tried just doing some testing with a hard quota and the global
notification options. I recieved one notification for a hard overage
and one for an
advisory one that I wasn't even testing.

When looking for quota stuff in the /var/log/messages file, I see
stuff like the following:

/var/log/messages:2013-05-09T13:46:49-06:00 <3.4> isilon-3(id3)
isi_quota_notify_d[99237]: Exiting on error: Failed notification ack
for hard-exceeded event.: Invalid argument from
process_domain_notify_event
(/build/mnt/src/isilon/bin/isi_quota_d/notify/qnotify_events.c:162):
Error processing events.
/var/log/messages:2013-05-09T13:46:49-06:00 <3.5> isilon-3(id3)
isi_quota_notify_d[99237]: Exiting.

This doesn't seem good. I have a call open with EMC and they are
looking into this. I still want to spend more time on trying to
understand the notification commands as there are lots of options and
it's not clear to me how to configure things :(

BTW: we are running 6.5.5.12

Thanks for the input.

--
-MichaelC

[Jerry Uanino]

fwiw we just export the isi quota list output to a script every 5 minutes and wrote our own tools for it for alerting. we let the hard quotas get enforced by onefs, but for alerting we just invented our own.  I believe we export it as XML but in later vesions it's json output.

[Michael Coffman]

Thanks for the tip.

[Peter Serocka]

We are about to write our own notification system, too.

Mainly because

- advisories should be more comprehensive, like stating
advisory AND hard limits together with current usage;
which you can't do with the builtin template macros.

- the shares' paths should be given in the form the users
actually see them on the clients, not /ifs/data/.../sharename

- add more detailed information for groups' shares,
such as usage per user as well as history tracking.

Sharing of further experience on how to write /effective/
notifications to users are welcome...


Peter

[Saker Klippsten]

We just take the output of 'isi quota list' via a cron job running on another machine (Linux VM) that logs in via ssh (setup ssh keys) , parse that via python and then email out accordingly based on the criteria we are monitoring for ( rules for each directory) we only deal with directory based quotas and then email to the "managers" of those project folders which we pull from our project management database. ( could use flat files or pickle file or even pull from say AD or ldap )

We also take that quota data convert to xml and push that into fusion charts to monitor and track usage over time. We get the size of the entire cluster via 'isi quota -q -d' this outputs the pool sizes and then we can subtract all the quotas to represent the free space. We specify certain folders as nearline (apply advisory quotas so it's easy for us to calculate what lives on nearline vs performance.

We shove all thia into a nice HTML
That shows the quotas usage over time with a nice graph...

I have requested a feature from Isilon that's specifies a breakdown for a quota where it shows how much lives on specific pools. That and and a 'du' type command .







Saker Klippsten | CTO | Zoic Studios

[SCR512]

Bumping this old thread... So is anyone sharing these tools publicly or is this internal secret sauce? At present, I'm working on some Python bits to send quota data to Graphite for visualization.

Ideally I'd like to just query the quota info via SNMP, parse and then send on over to Graphite or something else to visualize.

[SCR512]

Welp, here is my attempt at writing a script in Python to spirit away User Quota statistics and shove them into Graphite:

https://gist.github.com/scr512/6178204

Eventual plan is to expose these visualizations to end users.

[steve]

Bumping this to see if you guys have had any success with creating your own isi quota notifications?

The default notifications are nearly useless for our users.

[Michael Coffman]

I created my own system which I would be happy to share.   I have a very simple environment so this helped me out.   Pre-req's

- No user level quotas

- Only use hard quota limit

- Quota checker runs from a system with ssh access to isilon

I created a ruby script and a config file to define my own notification rules and whom to notify for each file system.    If you're interested in the ruby script, let me know.     You will need to customize it for your own use.    It is very basic  but, perfect for my environment.

[Jason Davis]

Here is an example that leverages the Graphite work I've done:

From: isilon@your_company.com

Subject: <ISI_QUOTA_PATH> Quota Exceeded.

<html>

<body>

The disk quota on directory <ISI_QUOTA_PATH> owned by <ISI_QUOTA_OWNER> was exceeded.

The <ISI_QUOTA_TYPE> quota limit is <ISI_QUOTA_THRESHOLD>, and <ISI_QUOTA_USAGE> is in use.

Please free some disk space by deleting unnecessary files.

<p><b>1 Hour</b>

<img src=http://192.168.0.2/render?target=legendValue(alias(isilon_user_quotas.isilon.<ISI_QUOTA_OWNER>.*.used,"Used"),"last")&areaMode=first&areaAlpha=0.7&target=legendValue(alias(isilon_user_quotas.isilon.<ISI_QUOTA_OWNER>.*.hard,"Treshold"),"last")&from=-1hours&lineWidth=2&title=Quota%20usage%20-%20<ISI_QUOTA_OWNER>%20-%20<ISI_QUOTA_PATH>&colorList=1BC6E0,red&yMin=0&height=300&width=600&format=png</img>

</p>

<p><b>12 Hours</b>

<img src=http://192.168.0.2/render?target=legendValue(alias(isilon_user_quotas.isilon.<ISI_QUOTA_OWNER>.*.used,"Used"),"last")&areaMode=first&areaAlpha=0.7&target=legendValue(alias(isilon_user_quotas.isilon.<ISI_QUOTA_OWNER>.*.hard,"Treshold"),"last")&from=-12hours&lineWidth=2&title=Quota%20usage%20-%20<ISI_QUOTA_OWNER>%20-%20<ISI_QUOTA_PATH>&colorList=1BC6E0,red&yMin=0&height=300&width=600&format=png</img>

</p>

</body>

</html>

This nets out something like this: