Quota enforcement

[Jon Tegner]

Hi!

I have tried to follow the instructions in 

/www.beegfs.com/wiki/EnableQuota

to set up quota on our system. 

It seems to be partly working at the moment, in that e.g., the command:

beegfs-ctl --getquota --uid --all

indicates how much is used as well as the hard limits for each user.

However, this information does not seem to be updated, and an usage exceeding the hard limit does not seem to be enforced.

As I said, I have tried to follow the instructions in the link above. We are using beegfs-client-6.7-el7 etc (which I hope should be newer than 2015.03-r19 which is required to have quota enforcement). All running on CentOS-7.3.

I realize now that although the users exist on the clients, they don't on the servers - could that be the reason for the issues I see?

Thanks!

/jon

[Harry Mangalam]

You didn't mention what filesystem you were using (XFS or ZFS) (and whether it was a new system or an existing system), tho the beegfs command should have worked identically on both (altho  I think that the chunk quotas don't work yet with ZFS..?)

Did you wait until the  beegfs-fsck  command completed?  We had that problem when we started quotas on a large existing beegfs system. It is now working admirably.

as an example, on an XFS-based system, which has mounted multiple beegfs FSs (have to specify the correct client conf to direct the command).

$ beegfs-ctl --cfgFile=/etc/beegfs/dfs1.d/beegfs-client.conf --getquota --uid hmangala                                                                  
      user/group    ||           size          ||    chunk files     
    name     |  id  ||    used    |    hard    ||  used   |  hard    
--------------|------||------------|------------||---------|---------
     hmangala|   785||   1.63 TB  |   3.00 TB  ||      972|     5000

because of my inability to remember the syntax of the command, we've written up a trivial script to query the quota info for a user across multiple beegfs filesystems, iterating thru their groups (since we have a lot of sometimes confusingly overlapping groups).  The output looks like this (mod some spacing artefacts)

1046 $ dfsquotas hmangala dfs1                                                           

===== [USER quota: hmangala on dfs1]                                                                                                                         

      user/group    ||           size          ||    chunk files                                                                                            
    name     |  id  ||    used    |    hard    ||  used   |  hard                                                                                           
--------------|------||------------|------------||---------|---------                                                                                        
     hmangala|   785||   1.63 TB  |   3.00 TB  ||      972|     5000                                                                                        
                                                                                                                                                            
                                                                                                                                                            
==== [Group Quotas on dfs1]                                                                                                                                  
                                                                                                                                                            
==== Group quota [staff]                                                                                                                                     
      user/group    ||           size          ||    chunk files                                                                                            
    name     |  id  ||    used    |    hard    ||  used   |  hard                                                                                           
--------------|------||------------|------------||---------|---------                                                                                        
        staff|   200||   1.63 TB  |      0 Byte||     1542|        0                                                                                        
                                                                                                                                                            
==== Group quota [apache]                                                                                                                                    
No data as USER [hmangala], group [apache] on [dfs1]                                                                                                         

==== Group quota [stata]   
No data as USER [hmangala], group [stata] on [dfs1]

==== Group quota [som]   
      user/group    ||           size          ||    chunk files     
    name     |  id  ||    used    |    hard    ||  used   |  hard    
--------------|------||------------|------------||---------|---------
          som|   140||  87.23 TB  | 100.00 TB  ||  4024726| 13000000

==== Group quota [biolinux]   
No data as USER [hmangala], group [biolinux] on [dfs1]

==== Group quota [gene]   
      user/group    ||           size          ||    chunk files     
    name     |  id  ||    used    |    hard    ||  used   |  hard    
--------------|------||------------|------------||---------|---------
         gene|   148||  88.74 TB  | 250.00 TB  ||  2579991| 13000000

Anyone is welcome to use it as a starting point for their own systems:  <http://moo.nac.uci.edu/~hjm/dfsquotas>

(tho it's hardcoded to use our 'dfs' nomenclature for different FSs)

hjm

[Jon Tegner]

Thanks for info!

Forgot the file system part. We are using xfs on storage nodes and ext4 on the meta. The quota was also added to an existing system.

Regarding the completion of the

beegfs-fsck --enablequota

command, I think this was completed before I put  

quotaEnableEnforcement = true

in beegfs-mgmtd.conf and restarted the beegfs-mgmtd service (but that was all that was restarted at that stage).

Regarding putting the users on the mgmtd/meta and storage servers, is this something I should do?

Thanks again!

/jon

[harry mangalam]

On Wednesday, 14 June 2017 06:14:41 PDT Jon Tegner wrote:

> Thanks for info!

>

> Forgot the file system part. We are using xfs on storage nodes and ext4 on

> the meta. The quota was also added to an existing system.

>

> Regarding the completion of the

> beegfs-fsck --enablequota

>

> command, I think this was completed before I put

> quotaEnableEnforcement = true

>

> in beegfs-mgmtd.conf and restarted the beegfs-mgmtd service (but that was

> all that was restarted at that stage).

 

All the above is essentially identical to our setup: MD=ext4 / storage=XFS, enable quotas, but the one thing I didn't see was that you initiated the beegfs-fsck to initialize the ownership data. Did that complete?

 

>

> Regarding putting the users on the mgmtd/meta and storage servers, is this

> something I should do?

 

No, we didn't do that and I don't think I have ever seen that recommended and in fact there's good reason NOT to.

 

I would say that I would rec NOT running the admon monitoring app long-term - I don't think there's a memory leak (it's Java), but it takes up a honking amount of memory (it's Java) and we recently started seeing allocation failures on our 128GB MD server (I assume) due to its 36GB footprint.

 

It's /extremely/ useful in debugging misbehaving clients and especially users but only when you need to.

 

hjm

--

Harry Mangalam - Research Computing, OIT, Rm 225 MSTB, UC Irvine

[m/c 2225] / 92697 Google Voice Multiplexer: (949) 478-4487

415 South Circle View Dr, Irvine, CA, 92697 [shipping]

XSEDE 'Campus Champion' - ask me about your research computing needs.

Map to MSTB| Map to Data Center Gate