HDFS Space Utilization keeps on increasing

[Shashi Vishwakarma]

Hi,

I have  DataTorrent 3.x installed on my cluster.Even thought there is no data torrent application is running , still my hdfs space utilization goes on increasing. Below is hdfs path that has occupied most of the space.

/user/dtadmin/datatorrent/apps

Why this is happening? Am I missing something here?

Thanks

Shashi

[David Yan]

Hi Shashi,

That directory is where Apex stores application information, like application jar files, checkpoints, container information, etc.  

Please run this command to see which directory is taking the most space.

$ hdfs dfs -du /user/dtadmin/datatorrent/apps

Then open dtcli and use the get-app-info command look at the information of that application.  For example:

dt> get-app-info application_1439598948299_0557

The field "state" will tell you whether the application is running or not. 

If you don't care about the application, you can safely kill it if it's running and delete the HDFS directory by doing hdfs dfs -rm -r /user/dtadmin/datatorrent/apps/application_xxx_yyy (replace xxx and yyy with appropriate values).  Note that doing so will wipe all stored information about that application.

David

--
You received this message because you are subscribed to the Google Groups "apex-dev" group.
To unsubscribe from this group and stop receiving emails from it, send an email to apex-dev+u...@googlegroups.com.
To post to this group, send email to apex...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/8754d662-4948-4920-96f3-cb58f70d5f39%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Amol Kekre]

Adding d...@apex.incubator.apache.org

Thks,

Amol

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAMqituP83nSGd4Ln6phTe0okyojwsE%3DGq22unu%3D-yDgyf0Y8tA%40mail.gmail.com.

[Shashi Vishwakarma]

Thanks David for detailed explanation. I checked apps directory in HDFS,there are around 12858 application in that folder each of having 6.2 M size. It will be a time consuming process to find status of each application by running get-app-info in dtcli. So logged in to web interface of datatorrent(port 9090) but there is no application running at this moment.

Still HDFS space utilization  is increasing,any pointers on this?

Thanks and Regards, 

Shashi

[David Yan]

That's a lot of applications.  I suspect there is something that keeps starting the application, which causes the folder to keep increasing in size. Can you just run get-app-info on dtcli on just one application and see what is being spawned up?

David

[Tushar Gosavi]

You can also check yarn resource manager ui and logs to verify which applications are getting restarted continuously.

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAMqituMeKHC84rJpFAHKbcFi-psC-zDqrOTRwQXhq75CbSQcBQ%40mail.gmail.com.

For more options, visit https://groups.google.com/d/optout.

--

“I'd have blown my top, because I want to beat this damn thing,
 as long as I've gone this far. I can't just leave it after I've found
 out so much about it. I have to keep going to find out ultimately
what is the matter with it in the end."
                Richard P. Feynman

[Chetan Narsude]

Can you check: yarn.resourcemanager.am.max-attempts setting for YARN (yarn-site.xml or yarn-default.xml whichever you are using)?

Also can you look at the application master logs for one of the app instances you did not start to see why it was shutdown?


--

Chetan
 

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CAHYazdeHVNqPgn8ABwic92HEkSrEoWU%3D_cXDw%2Brb5Li4GoDpww%40mail.gmail.com.

[Shashi Vishwakarma]

Hi All,

Thanks for your reply. I believe you guys are right. There is data torrent application which keeps on restarting. I observed resource manager UI, I always see one application running even no one running app from my team.

Chetan,

yarn.resourcemanager.am.max-attempts property is currently set to 2. I checked a log for that application,there are some AlreadybeingCreatedException is coming.Attaching log along this mail.Can some one help me on this?

Thanks and Regards,

Shashi

[Gaurav Gupta]

Shashi,

I see what is happening. For now, please stop gateway, clear /user/dtadmin/datatorrent/audit/ folder and start gateway again. This should resolve the issue for now.

Thanks

-Gaurav

To view this discussion on the web visit https://groups.google.com/d/msgid/apex-dev/CA%2BaZ0XP872qRKPFxtqrP9aCJAHaLOZe6BBw2iMMA3rtVKmvYyA%40mail.gmail.com.

[Gaurav Gupta]

Shashi,

Are you running multiple instances of gateway with same license?

Thanks

- Gaurav