Web UI show Capacity doesn't work

[Chanh Le]

bin/alluxio fs du  /

/ is 38601484482 bytes

38601484482 bytes -> 38.6014845 GB

Is that a bug? Because just in case check

I am using Alluxio  1.2.0-RC2

I've delete a lot of file to see the change of capacity but It seems not work.

[Gene Pang]

Hi,

I am currently not aware of bugs in the web UI accounting.

Are you saying that the command-line tool is correct, and the web UI is not correct? Is the command-line tool connecting to the same master?

If you add or remove a file in Alluxio, does it show up with the "./bin/alluxio fs"? Does it show up in the web ui?

Thanks,

Gene

[Chanh Le]

Hi Gene,

Are you saying that the command-line tool is correct, and the web UI is not correct? Is the command-line tool connecting to the same master?

I don’t know but It needs to be similar right? Yes I do the command at master.

If you add or remove a file in Alluxio, does it show up with the "./bin/alluxio fs"? Does it show up in the web ui?

I remove by ./bin/alluxio fs rm -R /folder_name. No after I removed It doesn’t show up in the UI.

Regards,

Chanh

--
You received this message because you are subscribed to a topic in the Google Groups "Alluxio Users" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/alluxio-users/LC8TERIj12k/unsubscribe.
To unsubscribe from this group and all its topics, send an email to alluxio-user...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Chanh Le]

Hi Gene,

I tested in 1.2.0 already It has the same issue as well.

Please help fix this.

[Gene Pang]

Hi,

Could you describe a simple test case to reproduce this? From a fresh installation, what are the steps to take to reach this state?

I just tried this with Alluxio 1.2.0, and the web UI matches "alluxio fs du".

Thanks,

Gene

[Chanh Le]

Hi Gene,

Here some step to reproduce:

1. Install a cluster with 5 nodes.

2. Push data into cluster until it nearly full of cluster capacity about 90 - 99%.

3. Delete some of them then delete about half of data.

4. Monitor the WebUI

I think it sort of slowly update to UI because after an hour I check again in version 1.2.0 it shows nearly in the command line.

[Gene Pang]

Hi,

How are you monitoring the web ui? By default, the usage will not update automatically, so you will have to refresh the page to see the new changes.

Also, deletion of the files happens immediately, but the workers are notified asynchronously. This means the files will be gone when you delete, but the workers will not physically remove the files until it is notified via heartbeat messages. There will be a short delay, but I do not expect it to be delayed by an hour.

If you said it ends up being accurate, could you see if you can refresh periodically to determine the delay?

Thanks,

Gene

[Chanh Le]

Hi Gene,

If you said it ends up being accurate, could you see if you can refresh periodically to determine the delay?

I had checked very carefully by refresh the page a lot of time by F5 also clear browser cache before I reported. 

Also, deletion of the files happens immediately, but the workers are notified asynchronously. This means the files will be gone when you delete, but the workers will not physically remove the files until it is notified via heartbeat messages. There will be a short delay, but I do not expect it to be delayed by an hour.

 

but why du command so correct but UI is not?

[Gene Pang]

Hi,

Were you able to determine the "delay"?

I think "du" command is different, because it lists the files and computes the sizes manually. https://github.com/Alluxio/alluxio/blob/master/shell/src/main/java/alluxio/shell/command/DuCommand.java#L62 Since the workers remove the blocks asynchronously, the actual blocks are removed at a later time.

Thanks,

Gene

[Chanh Le]

On Aug 2, 2016, at 8:48 PM, Gene Pang <gene...@gmail.com> wrote:

Hi,

Were you able to determine the "delay”?

I tested today from 11 AM and until now 11 PM it still not match or close with du command.

[root@master2:/home/spark/alluxio-1.2.0]# ./bin/alluxio fs du /

/ is 60740660683 bytes ~ 60.740660683 GB

I think "du" command is different, because it lists the files and computes the sizes manually. https://github.com/Alluxio/alluxio/blob/master/shell/src/main/java/alluxio/shell/command/DuCommand.java#L62

But du command is about the size of folder/file right?

Since the workers remove the blocks asynchronously, the actual blocks are removed at a later time.

But why it so long 5 hours under road?

[Gene Pang]

Hi,

It should not take that long. What do the worker logs say?

Thanks,

Gene

[Chanh Le]

Sorry, I forgot to backup the logs.

Right now I applied HDFS as a UnderFS I will continue monitor the issue and get back to you with more details.

Or it may cause because I didn’t use the UnderFS.

[Calvin Jia]

Hi,

The web UI and du do not represent the same information. 

The web UI displays how much Alluxio storage is used to store hot data. This number includes replicas, meaning even if you have only a 1 GB file, it could be stored on 5 workers, therefore taking up 5 GB of space. 

The du command displays how much total space is used, counting each file once whether or not it is stored in Alluxio storage.

Hope this helps,

Calvin

[Chanh Le]

Thank you Calvin,

This information help me a lot now I fully understand.

Regards,

Chanh

[Gene Pang]

Thanks for the confirmation!

-Gene