Disk Az

0 views
Skip to first unread message

Agnella Caamano

unread,
Jun 30, 2024, 10:52:57 AM6/30/24
to rehiphaso

As most with a PA-220 have experienced, regardless version running (currently latest 10.2.3-h2) root partition fills up all the time and have to run the disk-usage cleanup commands manually and tried enabling aggressive-cleaning as well, it just continuously hits level/ But it just fills up all the effin time:-)

With the root disk issues seemingly persistent (since at least the high 9 versions) and with the PA-220 only containing a measly 32GB, apart from the disk issues we are perfectly happy with the PA-220 and have a handful across our network, so does anyone know if there is a storage upgrade option? just want to be able to have a bigger eMMC storage in it if possible, either by buying and installing or any Palo support routes that would enable a swap of storage?

Yes thanks, but I've already read those links and others several times as I'm sure everyone else has who is frustrated at the root space issue, as I said in the post we have aggressive cleaning enabled and it hits the 95% threshold several times a day, as I also said above we have ran the disk-usage cleanup command several times with threshold of 90 but it can never make more than 93% free.

Coming with such a small eMMC is so disappointing, I read somewhere about others logging support issues and being able to get device swapped out with one with bigger 64GB, so information on if this is an option would be great, as otherwise may have to look at getting slightly bigger model and migrating configs across but that would be a pain in the ***.

But for the life of me, I cannot figure out how to add another disk (virtual hard drive) to my BIG-IQ VE. I don't want to expand the current disk... which is what I am finding all of the articles pertain to (and which I already know how to do). My desire is to add another HD (via HD2) to the VE.

So, what this means for deduplicated jobs is that an original job that protected all of the application data on a client was written to disk. Then subsequent jobs do not write this data again to the disk, which is the principle of deduplication only write data once.

So, even though that original job may have qualified for aging, because subsequent jobs refer to this original data, it is still retained. Removing this data would invalidate all the jobs that ran later.

Hi @Neil Cooper I have a few questions. Are these media agents physical or virtual? Would you be able to get one VM using the stable vmtools version (please make sure you are not using version 11269 issue reported here ( -Tools/11.0/rn/VMware-Tools-1105-Release-Notes.html)

Hi Neil,

Have you checked that Automount is disabled and the SAN policy is OfflineShared on the affected Media Agents?
Is there any AV on the Media Agents that could be scanning the attached disks or interfering with the CV Processes?

As I navigate down the internet researching this issue I sure see some other very similar cases. So what I`m about to share it may (not) be totally connected to your issue, so take all of this with some grain of salt.

So again, Im not an expert in Dell Compellent, but after reading some of these docs I do wonder if you have the software configured with the VSS on your media agents. Would you be able to review the links and confirm here?

This is an old document from Dell on their - CommVault Simpana 10 Best Practices for the Dell Compellent Storage Center and though the articles in my previous post do mention the requirement Compellent Replay Manager Service on Hyper-V - it raises the question as to whether or not you have Dell Compellent Software Replay or any other software that may be interfering with the way Intellisnap works when it takes the snaps and mounts on your Media Agent during an snapmount.

In any case, if you do have it installed, check out the version as some older versions may not support integrate well (or integrate at all) with Commvault. From what I could find, it seems version 8.0.1 is the latest.

If you do have replay manager, make sure to be using the latest version as it fixes issues with the VSS Provider, if you have the latest version and still does not work, try removing it completely and retry the operation.

Hi, to me the fact that you see multiple LUNs and dead paths has a lot to do with the MPIO software and the Dell Compellent Software I mentioned above. Check it out and let us know when you have a chance. Enjoy your weekend.

I`m quite unsure as to what your thought process is here or even at what your questions are. Previously shared some links that points to best practices as well as a software for the Media Agents, did you have a chance to look at that?

I'm new to AW. Installed it as a VM on my Dell PE T610 server, which has 4 other Windows VMs running. I allocated 8GB RAM, but only 40GB disk space when I created my AW VM. It has already consumed the entire 40GB! Yet I only have 10 devices (1 AP) and 12 clients. I can't fathom how so much space could get used up so quickly in the 60 days sent I installed AW. Are there tricks to reduce disk usage? Maybe I can change the default polling interval that SNMP devices get polled? Are there any stats as to how much data a single poll of 1 SNMP device consumes?

You can see the disk usage in System -> Performance -> (scroll down) -> Disk Space. Which does not tell where the data goes in, but it is likely to be in the database, in upgrade files, and in backups. You can clean up some of those via the AMPCLI Delete File option.

Thanks for the info. I'm really surprised that the database gets so big for such a small amount of devices & clients. I have Kaseya and Labtech servers accumulating data, including Windows event logs, for hundreds of computers and the databases are very small in comparison to AW.

Reduce the retention times down. If I remember correctly reducing the 'client data retention interval' will have the biggest effect. You should see this after the overnight maintenance, or you could change that to run in 10 mins time to get a more immediate cleanup.

If you go to System -> Performance, there's a widget on there (about 3/4 of the way down under Database Statistics) that shows the largest db tables. Most likely the AppRF (PEF) and device config tables are usually among the heavy db space consuming tables.

Hmmm, maybe I don't have a disk space issue, as here is what AW's System->Performance->Disk Space shows about my 40GB. I don't pretend to understand Linux (CentOS), but I'm guessing the graph tells me my disk usage is 45.9%?

Looks like it's not your disk space that's full, it's the RAM that's being fully utilized. OOMkiller comes into play when there's a process that's starved for memory. Do you have a capture of the top portion of the System -> Performance page? I'm interest in seeing how the RAM was setup on install. You may have selected 8GB RAM, but it's not helpful if somehow the OS decided to put 4GB of it allocated to swap.

Also, since it's hitting OOMkiller - opening a support case to have someone take a closer look may help. I'd start by looking at the low_level_service_watcher, service_watcher, kernel, and messages logs.

If you use'd the ISO file, then you should have had an option at the OS install splash to enter 'manual' to allow for customizing the disk partitioning and RAM allocation. If you ran 'install' it will default to a split calculated based on allocated disk space and RAM. The default will usually take 2/3 for RAM, and then 1/3 into swap.

So in my case, since I allocated 8GB RAM in VMware for this AW vm, for some reason it allocated a 50/50 memory/swap. Is there any way to change it? If not, I might try a fresh install and do the manual setup. I'm a bit puzzled as to why the default settings wouldn't handle 10 devices (with 1 AP having 12 clients). I do have to say, the GUI is slow as mollasses :(

But as I think back, I believe I initially only allocated 4 or 6GB RAM to the vm. And initially I installed it in a datastore that resided on a NAS unit rather than my Dell PE server. Once I saw that AW looked promising, I used the VMware vCenter Converted Standalone utility to copy the AW vm on to my Dell server's datastore. I think I noticed that the copied AW vm no longer reported the os to be Linux - instead it simply says "Other (32-bit)".

To save what you've done so far, grab a backup from System -> Backups. If you've made changes since the last nightly maintenance time, you could force a backup from the AMPCLI menu - then SCP the file to a remote server (any linux server running sshd, or you can install bitvise sshd onto your win laptop, if on a Mac, you might have to update the ciphers to make scp work).

This is interesting: I powered down the AW guest vm, then edited the VMware settings for the os - changing it to Linux, CentOS 4/5/6 (64-bit). It was previously set at Other - unknown. I then powered up the AW vm, but I'm unable to see any console - just black screen. I figured that meant the os never booted. Just for kicks I tried to log into the AW GUI - and got in! It also seems a good bit faster on most of the pages.

If you hit System -> Performance, it should show you the kernel info. Here's an example from one of my lab AMPs. The whole kernel section is extracted into a pretty format from parsing the unix 'uname' cmd output. x86_64 translates into 64-bit architecture.

I guess AMP needed to rest overnight, as now the CLI is working. But I still see the out of memory errors. There's no way to partition memory and swap space differently post install, right? Is there an English translation as to what those OOM messages mean? It seems to be "killing" processes that always mention "httpd" - what are those processes? And when I look at the rest of the OOM messages, it looks to be reporting the amounts of memory used by various things:

I'm guessing the first one is total memory used by vm (the AW guest vm under ESXi?). That's showing about 3.5GB. Add in the other two amounts and it's under 4GB. And I guess I only have 4GB to play with, as the other 4GB is dedicated to swap space and does not count?

d3342ee215
Reply all
Reply to author
Forward
0 new messages