Dedicated hardware suggestion

[Albert Zeyer]

I would like to run some kind of dedicated hardware/server just for Perkeep, which both acts as a backup, but also as a data provider to e.g. access my photos or other things.

Are there suggestions what to run/use? I guess Linux? What file system? And a hardware RAID? Or software RAID? Or is RAID kind of obsolete with Perkeep, and I should do the duplication on the Perkeep level?

I would also like to have this in my home, so it should be quiet (preferably only passive cooling) and consume low energy.

Best regards,

Albert

[Tamás Gulácsi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

AFAIK the disks can be slow, but plenty (for a home server) of RAM is needed,
as the metadata index is kept in memory.

perkeepd consumes about 7GB of RAM for

Jan 02 15:27:48 tequila perkeepd[7308]: 2019/01/02 15:27:48 index/corpus: stats: 3354.762 MiB mem: 8111182 blobs (71.815 GiB) (7239923 schema (76206 permanode, 54526 file (53213 image), ...)
Jan 02 15:27:48 tequila perkeepd[7308]: 2019/01/02 15:27:48 index/corpus: scanning CPU usage: 3m13.711292s

Tamás
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> --
> You received this message because you are subscribed to the Google Groups "Perkeep" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to perkeep+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail
Comment: https://protonmail.com

wsBcBAEBCAAGBQJcLMrAAAoJELj70Bsqr19tA8QH/RsW8VNXyei8ewXi0lOZ
mZLNpTMCvXhgGiwVHQlnidDEmassFIkroV4Sqpd7er3cpNnJCexEcjuU0rwt
zr2NwKE+T0AgCF13QE2CcwWsF8TAoIAF9D0ud6sDvu3+mUN2pYw+p1lpf5Bm
j0cF5g/4lxvS5BPKEOQoGK4+MK4DSLAsvYQjWpsflSA/iYAkOA2NwIQJgdj9
XOxQ5d8357TZfN+LqTfRwEi1cpnFxXJo2eHUG5UodMAqYvI+2jtWuT6qMLXa
3iIF8NwsH/BtbgMYl+Lt5c4J/a7bDkOK/avDxfMBLI88nxS9lUs3rYyDxskJ
+YHVbi+Fa6LdhCGVajiaB8o=
=p1cU
-----END PGP SIGNATURE-----

[Albert Zeyer]

I want to store several TB of data in the database.

So given that you need 7GB of RAM for about 70GB of blobs, does that mean I would have to assume maybe 100GB RAM usage for 1TB of blobs?

It seems to me like this does not seem to scale. Is this a fundamental problem of Perkeep?

Best,

Albert

[Tamás Gulácsi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

IMHO it is a fundamental problem - it should use LevelDB for the index, not memory.

But this index is "just" for the UI and searching - I'm using perkeep in production as
a deduplicating storage with severalt TB of small files without problem,
with a few MBs of RAM usage - but without indexing.

Tamás


‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> --
> You received this message because you are subscribed to the Google Groups "Perkeep" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to perkeep+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail
Comment: https://protonmail.com

wsBcBAEBCAAGBQJcLN5KAAoJELj70Bsqr19tqFUH/is0+bUJynLinYFzZfyr
OKGeR14UFJ6rxsYvnQqnuZrh9FKEnWx2jmFkCLHCBq8/Me7SQ1HdVossVSd8
Q122cFyVErDDAFYY8Ob2gnQQKX1IT3RD7Ybzeb7oaSLbQWUXYY9QWoe9gcb9
Jo3lgYiR5vvaodAVi64dJirUK0vqU/mcfV7oubKuWS91R1qv3MOD9kcbdZ+M
SrOD/yKF7iMaFS20ah32M28oL2MsC6MtnYbjjCBUZc+r+ooSORbqG9o1MpA0
O98yo+MCezwauxbUO1bAqtxxWOXvc0ZyaugH868HXCXowd4bL6XzeKmTyRG+
Ggs0KfHfGhdPCeo53Jc/sUg=
=d100
-----END PGP SIGNATURE-----

[Albert Zeyer]

Is it possible to only use the indexing on parts of my storage? I.e. let's say I have a copy of my whole Linux FS, but I only want to index the documents + photos?

Otherwise, I maybe could run two independent Perkeep stores, one for the whole system backup, and another one for all documents (e.g. just everything in /home).

But my other question (https://groups.google.com/forum/#!topic/perkeep/Z_utKIPe2o0) basically is more related to this.

But it sounds like in any case, a reasonable amount of RAM makes sense for such hardware.

You received this message because you are subscribed to a topic in the Google Groups "Perkeep" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/perkeep/IuUE3f12rhw/unsubscribe.
To unsubscribe from this group and all its topics, send an email to perkeep+u...@googlegroups.com.

[Tamás Gulácsi]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

To answer the file system question: the file blobserver stores lots of files (each blob in a separate file) which may impose some stress on the filesystem.
But the diskpacked storage does not.

If you may delete files, use the blobpacked blobserver;
if you not, then use diskpacked, or blobpacked with diskpacked for the large blobs and the local blobserver for the cache.

This may or may not be configured with the high level server-config.json.
If not, then configure everything you can with the high level config, then dump the low-level config with "pk dumpconfig", tweak it, and then use the low level config.

With the low level config, the cond, overlay and union blobservers allow creating very complex topologies, too.


Tamás
‐‐‐‐‐‐‐ Original Message ‐‐‐‐‐‐‐

> --
> You received this message because you are subscribed to the Google Groups "Perkeep" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to perkeep+u...@googlegroups.com.
> For more options, visit https://groups.google.com/d/optout.
-----BEGIN PGP SIGNATURE-----
Version: ProtonMail
Comment: https://protonmail.com

wsBcBAEBCAAGBQJcLOB1AAoJELj70Bsqr19tzeQH/R+eKOHCUSapSiZzU9Q5
GisRcaoBtQseMdvonBxy5iyWm4i9zxxuX0CqmN7DAbvX6NI8jYHi5ZkcR8Ng
S4sGhqxY7PmDp9Yed+vJXDXy1sXETAtwLhn64w4Dp3gQos8Mk3VmWZtyvPHF
K9iRRKOtqiiFKtAB4Js0YDATC/BExVFwQprf6Jl0cpiK/dp1LQWIanJHQmb0
hfXjB90tm/1LkoJa3P3eqI38EtX0Dj0Q7utcawtRlqF51WT9ITjYvDQbSdow
IdaBmiIxOWQ1g11ZkdtjrTgnLA4At0x44W5rqrjqkXNZtWUjDpm321xsSgkT
tW0xSqRZ6kBlhymp1bCj5bg=
=fr7S
-----END PGP SIGNATURE-----