Moving from diskpacked to blobpacked

[Tamás Gulácsi]

Hi,

I'm using diskpacked storage engine, as I don't want to pay the price of lot of small files.
Blobpacked came, and I've measured that is quite good - and with the new "union" storage, I can even move its localdisk files into a diskpacked store.

But blobpacked packs blobs of the same file, so the blobs should be copied into the blobpacked store in "natural" order, not lexicographical, as sync does.
That's what BlobStreamer interface is for. But that interface does not have a HTTP endpoint, as BlobEnumerator (sync) has.

I've tried to implement it, but it is a big undertaking, and I only want to shovel my blobs from my diskpacked into my blobpacked.

The I thought maybe camput could get a "rawdpstream" subcommand, and "cat diskpacked/pack-*.blobs | camput rawdpstream",
but camput is too intelligent, and I cannot specify the destination exactly - it only accepts the server as the server prefix,
contrary to camtool sync, which can.

Shall I try to add a "stream-diskpacked" subcommand to camtool?

Or any other idea?

Thanks,
Tamás Gulácsi

[Mathieu Lonjaret]

Hi,

iirc, you're facing a problem similar to
https://github.com/camlistore/camlistore/issues/681 , aren't you?

> --
> You received this message because you are subscribed to the Google Groups
> "Camlistore" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to camlistore+...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.

[Gulácsi Tamás]

Seems yes, exactly.

But I can't find a workaround, sorry.

You received this message because you are subscribed to a topic in the Google Groups "Camlistore" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/camlistore/tw3WizMybE8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to camlistore+...@googlegroups.com.

[Brad Fitzpatrick]

I think your email's subject line is the closest thing to a problem
statement in your email.

Is the problem that you want an efficient way to move from diskpacked
to blobpacked?

[Gulácsi Tamás]

Yes, an efficient one.

Or an idea, how to push the file blobs from a diskpacked storage into a bloblacped storage "file-level sync".

All other blobs could move with "camtool sync".

You received this message because you are subscribed to a topic in the Google Groups "Camlistore" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/camlistore/tw3WizMybE8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to camlistore+...@googlegroups.com.