Quota enforcement in 2014.01-r11

[Andrew Lewis]

Hi

We're running a 3 node cluster in Centos 6 on an XFS file system with quota enforcement support enabled as per this page: http://www.fhgfs.com/wiki/EnableQuota

However the quotas do not seem to be enforced, even with updateinterval set fairly low.  The UID\GID range variable is a bit ambiguous, does this mean above or below the value? (we've tried both, no luck).

The XFS file system is mounted on all storage nodes as follows:

/dev/mapper/vg_srvcntqut002-LogVol02 on /data type xfs (rw,uqnoenforce,gqnoenforce)

 

The getquota command works:

fhgfs-ctl --getquota --uid --all

  user/group    ||                 size              ||         chunk files

 name     |  id  || used   |    soft   |  hard    ||  used   |  soft   |  hard

--------------|------||------------|------------|------------||---------|---------|---------

   user1|   510 ||390.63 MB  |      0 Byte| 100.00 MB  ||   4|  0|   5

   user2|  2000||  11.44 GB  |      0 Byte| 500.00 MB  ||  11|  0|  3

   user3|  3000||    7.63 GB  |      0 Byte| 500.00 MB  ||   8|  0|   3

And clearly all users are in violation of quota but can still write data to the file system.  We've run fhgfs-fsck --enablequota  successfully but this also has no result.

What are we doing wrong?

regards

Andrew

[Arnvid Lau Karstad]

Hi

I saw something similar - but I had forgotten to set the parameter to true in fhgfs-client.conf.

Br

Arnvid

[Andrew Lewis]

The quotaEnabled = true is set in the fhgfs-client.conf file.  

Is there a preferred logging level for debugging?  Changing it to 7 does not seem to increase the verbosity.  The only reference I can see to the word "quota" appears in the fhgfs-storage.log:

"fhgfs-storage.log:(3) Dec24 09:17:12 HBeatMgr [HBeatMgr] >> Downloading ID list of exceeded quota..."

We've tried setting quota on the uid or gid, each time the command indicates that this was done but the user can still write data to the fhgfs file system beyond the set quota.

regards

Andrew

[Frank Kautz]

Hello,

I can not reproduce the issue.

All changes in the configuration file of a daemon/client need a restart
of the daemon/client. The maximum log level is 5. This is documented in
the description part of the configuration file.

> The UID\GID range variable is a bit ambiguous, does this mean above or below the value? (we've tried both, no luck).

Did you have a look into the help of this command? A range needs a start
and a end value.

$fhgfs-ctl --setquota --help
GENERAL USAGE:
$ fhgfs-ctl --<modename> [mode_arguments] [client_arguments]

CLIENT ARGUMENTS:
Optional:
--mount=<path> Use config settings from this FhGFS mountpoint
(instead of "--cfgFile").
--cfgFile=<path> Path to FhGFS client config file.
(Default: /etc/fhgfs/fhgfs-client.conf)
--logEnabled Enable detailed logging.
--<key>=<value> Any setting from the client config file to
override
the config file values (e.g. "--logLevel=5").

MODE ARGUMENTS:
Mandatory:
One of these arguments is mandatory:
--uid Set quota limits for users.
--gid Set quota limits for groups.

One of these arguments is mandatory:
<ID> Set quota limits for this single user or group ID.
--all Set quota limits for all user or group IDs on
localhost.
(System users/groups with UID/GID less than 100 or
shell set to "nologin" or "false" are filtered.)
--file=<path> Read the quota limits from a file and set the
quota limits.
A csv file is requires each line is a user or
a group.
Line format: ID/name,size limit,inode limit
--list <list> Use a comma separated list of user/group IDs.
--range <start> <end> Use a range of user/group IDs.

Mandatory if not imported via --file:
--sizelimit The diskspace limit for the users/groups.
--inodelimit The inodes (chunk files) limit for the
users/groups.

USAGE:
This mode sets the size and inodes (chunk files) quota limits for users and
groups.

Example: Set quota size limit to 50 MiB and 50 chunk files for user ID
1000.
$ fhgfs-ctl --setquota --sizelimit=50M --inodelimit=50 --uid 1000

Example: Set group quota limits given in file /tmp/quota_limits.txt.
$ fhgfs-ctl --setquota --gid --file=/tmp/quota_limits.txt

Example: Set quota size limit to 20 MiB and 500 chunk files for user
IDs 100
and 110.
$ fhgfs-ctl --setquota --uid --sizelimit=20M --inodelimit=500 --list
100,110

Example: Set quota to unlimited diskspace and 5000 chunk files for
group IDs
in range 1000 to 1500.
$ fhgfs-ctl --setquota --sizelimit=0 --inodelimit=5000 --gid --range
1000 1500


kind regards,
Frank Kautz
Fraunhofer

> --
> You received this message because you are subscribed to the Google
> Groups "fhgfs-user" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to fhgfs-user+...@googlegroups.com
> <mailto:fhgfs-user+...@googlegroups.com>.
> For more options, visit https://groups.google.com/d/optout.

[Andrew Lewis]

Hi Frank

 

> The UID\GID range variable is a bit ambiguous, does this mean above or below the value? (we've tried both, no luck). 

Did you have a look into the help of this command? A range needs a start
and a end value.

What I meant by this is the documentation on the following page:  http://www.fhgfs.com/wiki/EnableQuota

If you look at the section on FhGFS management daemon

quotaQueryType                 = range
quotaQueryUIDRange             = 2000
quotaQueryGIDRange             = 20000

It's the above I'm asking about. Does this mean that everything from 0-range is queried or everything from range and above?  I'm assuming the former.

I can quite happily set the quota and I can then see the result from getquota, however the quota is not enforced.  

Also wanted to confirm that the /etc/fstab for XFS is meant to have uqnoenforce,gqnoenforce and not uquota,gquota?

I've restarted the mgmtd with log level 5 and can see the following in the log:

(3) Dec30 13:34:34 Main [QuotaManager] >> Loaded quota user limits: 3501 from file /data/fhgfs/fhgfs_mgmtd/quota/quotaUserLimits.store

(3) Dec30 13:34:34 Main [QuotaManager] >> Loaded quota group limits: 3501 from file /data/fhgfs/fhgfs_mgmtd/quota/quotaGroupLimits.store

(3) Dec30 13:34:34 Main [QuotaManager] >> Loaded quota data for users of 0 targets from file /data/fhgfs/fhgfs_mgmtd/quota/quotaDataUser.store

(3) Dec30 13:34:34 Main [QuotaManager] >> Loaded quota data for groups of 3 targets from file /data/fhgfs/fhgfs_mgmtd/quota/quotaDataGroup.store

(3) Dec30 13:34:39 QuotaMgr [QuotaManager] >> Saved quota data for users of 0 targets to file quota/quotaDataUser.store

(3) Dec30 13:34:39 QuotaMgr [QuotaManager] >> Saved quota data for groups of 3 targets to file quota/quotaDataGroup.store

regards

Andrew

 

[Frank Kautz]

Hi Andrew,

there is an mistake in the wiki page, thank you for reporting this. It
is fixed, now.

This also explains why the the quota enforcment doesn't work. The ranges
requires two numeric values separated by a comma. This is also
documented in the configuration file of the management daemon. When the
packages are updated the new configuration file with the additional
options and additional descriptions is stored as fhgfs-mgmtd.conf.rpmnew.

kind regards,
Frank

Am 12/30/2014 um 12:38 PM schrieb Andrew Lewis:
> Hi Frank
>
>
> > The UID\GID range variable is a bit ambiguous, does this mean
> above or below the value? (we've tried both, no luck).
>
> Did you have a look into the help of this command? A range needs a
> start
> and a end value.
>
> What I meant by this is the documentation on the following page:
> http://www.fhgfs.com/wiki/EnableQuota
>

> If you look at the section on *FhGFS management daemon*

[tee...@gmail.com]

Hi Frank,

I've quickly setup a small FhGFS virtual machine with all the FhGFS components installed. I've enabled the enforcement and setup a XFS disk mounted with the noenforce options for user / group. All works well. I've set the quotaUpdateIntervalMin to 1 instead of the detault 10 but looking at the log from fhgfs-storage.log it seems to be updating every 10 minutes. I know that its updating every 1 minute because of the updates I am doing to the filesystem within the 1 minute period.

(3) Dec30 16:15:21 QuotaMgr [QuotaManager] >> Saved quota data for users of 1 targets to file quota/quotaDataUser.store
(3) Dec30 16:15:21 QuotaMgr [QuotaManager] >> Saved quota data for groups of 1 targets to file quota/quotaDataGroup.store
(3) Dec30 16:25:21 QuotaMgr [QuotaManager] >> Saved quota data for users of 1 targets to file quota/quotaDataUser.store
(3) Dec30 16:25:21 QuotaMgr [QuotaManager] >> Saved quota data for groups of 1 targets to file quota/quotaDataGroup.store

Thanks
Timothy

[Frank Kautz]

Hi Timothy,

this is only the message when the quota data are stored on the hard-drives.

kind regards,
Frank