Re: [fedora-tech] Fedora 6 storage space issue

[Peter Winckles]

Nikolay,

OCFL and Fedora 6 produce a lot of small files. What is your filesystem's block size? A large block size means that there'd be a lot of wasted space. I thought there was a wiki page about this and the inode issue, but I'm struggling to find it.

Peter

-------- Original Message --------
On 9/26/24 10:31 AM, Nikolay Gannev wrote:

Hi Everyone,

We recently migrated from fcrepo 4 to 6 and noticed something that we would need help from the forum.

There were 70000 records migrated from fcrepo 4 - fcrepo 6 on July 22nd.

After the migration, we ingested 500 new records with documents size totaling 350 MB on July 30th.

data\ocfl-root reduced the available space of the drive by 21 GB.

Anyone knows why it did consume 21 GB of space for ingesting 350MB of binaries?  

Thanks, 

Nikolay.

--
You received this message because you are subscribed to the Google Groups "Fedora Tech" group.
To unsubscribe from this group and stop receiving emails from it, send an email to fedora-tech...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/72c6a6d1-df49-4df8-91f9-0a43ab2c9c15n%40googlegroups.com.

[Bhavna Gehlot]

Peter,

The filesystem block size is 

Bytes Per Sector  :                512
Bytes Per Physical Sector :        4096

What would be your suggestion ?

Bhavna

[Peter Winckles]

Bhavna,

That seems normal.

Did you note the disk usage after your initial migration? Just want to make sure that the bloat was part of the subsequent ingest and not the migration process.

Peter

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/8ba9d8e7-5829-45f0-a0ef-117fd1c7d6bbn%40googlegroups.com.

[Dan Field]

There is a brief mention of the issue on https://wiki.lyrasis.org/display/FEDORA6x/Migrate+to+Fedora+6#MigratetoFedora6-Prerequisites Maybe that's what you were thinking of Peter?

---

Dan Field

Developer, Fedora Program

LYRASIS

 

---

From: 'Peter Winckles' via Fedora Tech <fedor...@googlegroups.com>
Sent: 27 September 2024 12:18
To: fedor...@googlegroups.com <fedor...@googlegroups.com>
Subject: Re: [fedora-tech] Fedora 6 storage space issue

 

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/93f3NmDdGhIKdZrfQT3CC3nO2OdPZ3wUinonkMCNq_pXAg2rEY5r431_X9yB0cygwoULA2cef3ZLc0tQK65LxCUeDTdvX-eIwRkQkc1WsWE%3D%40pm.me.

[Peter Winckles]

Yeah, that's what I was thinking of.

The other thing I'd be curious about is how many versions your ocfl objects have and how large the inventory files are. If there's a lot of versions, and you're using the default sha512, the files can get quite large, unfortunately.

To view this discussion on the web visit https://groups.google.com/d/msgid/fedora-tech/SJ2PR22MB5093E62730B621973339891BE67E2%40SJ2PR22MB5093.namprd22.prod.outlook.com.

[Bhavna Gehlot]

Peter,

Did you note the disk usage after your initial migration?    

Yes, we did notice the disk usage was more with fcrepo 6 during initial migration but that was not concerning, and it was not a big difference.  

But the subsequent ingest disk usage is concerning. 

The other thing I'd be curious about is how many versions your ocfl objects have and how large the inventory files are. If there's a lot of versions, and you're using the default sha512, the files can get quite large, unfortunately.

We have fcrepo.autoversioning.enabled=false added.  So, there is only one version.  The size of inventory files is super small (2kb)

Bhavna