full ha destributed replica cluster possible?

[Vlad CopyLove]

It is not clear from documentation if and how if possible.

Is it possible to create full HA including metadata server?

Also if it is possible to mount it as FS locally if meta chunk and client are on same node (for read speed)?

How is read replica selected? Is it possible to direct read to specific nodes?

Thank you,

[Michael Ovsiannikov]

Vlad,

Starting from QFS release 2.0 full HA, including meta server is supported.

QFS fuse can work with QFS fs as long as client / fuse node has reasonably adequate network connectivity with respective chunk and meta servers. This includes the case where all meta server components deployed on the same node. QFS test, including QFS fuse test, runs in the configuration where all QFS components deployed on the same node / OS host.

Performance tests with QFS fuse will, of course, incur fuse overhead: at least two extra switches, and possibly copies between kernel and user space, and possibly overhead of kernel FS page/buffer cache depending on the kernel / fuse implementation and parameters.

For read and write benchmarking cpfromqfs and cptoqfs tools can be used, as these have minimal overhead, especially in the case when /dev/zero and/or /dev/null or equivalents are used.

With sufficiently large files, default chunk server IO settings [multiple concurrent non sequential IOs per device], spinning media, and all components running the single node single read with reasonable buffering / pipelining is expected to be disk IO bounded, assuming adequate CPU and memory bandwidth.

For read meta server does random shuffle of chunk servers hosting chunk replicas, such that the servers with smaller “load" go before the servers with larger load. Chunk server load calculated based on the chunk server current number of IOs and telemetry data: IO queue wait time and such. “Local” (same node as client) replicas are “preferred” by the client read logic, if no “local” replica available, then client attempts to read from replicas in the order returned by the meta server unit read succeeds.

In the most common and storage efficient case of RS files with replication 1 (striped files with Reed Solomon encoding), there is obviously only one replica to choose from for each stripe.

— Mike.

--
You received this message because you are subscribed to the Google Groups "QFS Development" group.
To unsubscribe from this group and stop receiving emails from it, send an email to qfs-devel+...@googlegroups.com.
To post to this group, send email to qfs-...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/b4dc1ec6-f855-4e19-b08a-fff75deed71b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Vlad Kopylov]

Thank you for reply Mike.

Not bothered much with fuse overhead as I am trying to use it on
distributed servers with latency. Most likely those 1-2 ms between
nodes will be a main concern. Plus file load are small files.
Somewhat replacing DRBD.
I see meta fail-over needs dns to return all meta server IPs to chuck
and client so I went to test replication with one meta for now.

trying 3 replica simple config with one meta from the wiki on Centos 7
with qfs 2.1.1
maybe there is another place with example and including meta replicated settings
here is what I get:

MetaServer.prp :
metaServer.clientPort = 20000
metaServer.chunkServerPort = 30000
metaServer.rackPrefixes = 192.168.1.221 1 192.168.1.222 2 192.168.1.223 3
metaServer.createEmptyFs = 1
metaServer.logDir = /home/admin/qfs/transaction_logs
metaServer.cpDir = /home/admin/qfs/checkpoint
metaServer.clusterKey = kuhsw8usnlisj89w656

and chunk server with ChunkServer.prp :
chunkServer.metaServer.hostname = 192.168.1.222
chunkServer.metaServer.port = 30000
chunkServer.clientPort = 22000
chunkServer.chunkDir = /home/admin/qfs/data0
chunkServer.clusterKey = kuhsw8usnlisj89w656

mounting fuse:
qfs_fuse localhost:20000 /home/admin/g01 -o rrw,allow_other

when I copy folder with 2500 small files ~50k

meta giving me repeated: WARN - (LayoutManager.cc:10946) can not find
replication destination for: <121,65653> replicas: 1 extra: 2

chunks repeat: INFO - (DiskIo.cc:3099) 0xf14110 0xfcd190fs space
available (0 0) status: -2 get fs available No such file or directory
2

and mount repeats:
INFO - (Writer.cc:1633) PW 1,957,1902,1854.extension scheduling retry:
1 of 6 in 5 sec. op: allocate: fid: 957 offset: 0 access: s: c:
valid for: 0 lease duration: -1
11-23-2018 21:21:13.492 ERROR - (Writer.cc:1522) PW
1,957,1902,1854.extension operation failure, seq: 333130974822407189
status: -28 msg: no space available op: allocate: fid: 957 offset: 0
access: s: c: valid for: 0 lease duration: -1 current chunk server:
-1 chunkserver: no data sent
Request:
ALLOCATE
c:49f84db49f81c15
p:74
w:1e848
H:vm2
N:/_data/cur2/1854.extension
P:3bd
S:0

> To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/8A361401-6CEB-4801-B4F2-908D573E8919%40quantcast.com.

[Vlad Kopylov]

plus
files copied are around 200Mb total

chunk folder go to 5Gb in no time

-rw-------. 1 root root 0 Nov 23 22:05 0-fsid-5982866580013357880
-rw-------. 1 root root 17K Nov 23 22:12 100.65633.1
-rw-------. 1 root root 42K Nov 23 22:12 102.65635.1
-rw-------. 1 root root 44K Nov 23 22:12 103.65636.1
-rw-------. 1 root root 222K Nov 23 22:12 104.65637.1
-rw-------. 1 root root 33K Nov 23 22:12 105.65638.1
-rw-------. 1 root root 72K Nov 23 22:06 10.65543.1
-rw-------. 1 root root 30K Nov 23 22:12 106.65639.1
-rw-------. 1 root root 50K Nov 23 22:12 107.65640.1
-rw-------. 1 root root 33K Nov 23 22:12 108.65641.1
-rw-------. 1 root root 1021K Nov 23 22:12 109.65642.1
-rw-------. 1 root root 26K Nov 23 22:12 110.65643.1
-rw-------. 1 root root 31K Nov 23 22:12 111.65644.1
-rw-------. 1 root root 44K Nov 23 22:12 112.65645.1
-rw-------. 1 root root 27K Nov 23 22:12 113.65646.1
-rw-------. 1 root root 47K Nov 23 22:12 114.65647.1
-rw-------. 1 root root 48K Nov 23 22:12 115.65648.1
-rw-------. 1 root root 437K Nov 23 22:06 11.65544.1
-rw-------. 1 root root 39K Nov 23 22:12 116.65649.1
-rw-------. 1 root root 128K Nov 23 22:12 117.65650.1
-rw-------. 1 root root 32K Nov 23 22:12 118.65651.1
-rw-------. 1 root root 28K Nov 23 22:12 119.65652.1
-rw-------. 1 root root 241K Nov 23 22:12 120.65653.1
-rw-------. 1 root root 244K Nov 23 22:12 121.65654.1
-rw-------. 1 root root 29K Nov 23 22:20 122.65655.1
-rw-------. 1 root root 65M Nov 23 22:20 124.65657.1
-rw-------. 1 root root 147K Nov 23 22:06 12.65545.1
-rw-------. 1 root root 26K Nov 23 22:18 128.65661.1
-rw-------. 1 root root 961K Nov 23 22:06 13.65546.1
-rw-------. 1 root root 27K Nov 23 22:12 140.65673.1
-rw-------. 1 root root 42K Nov 23 22:12 142.65675.1
-rw-------. 1 root root 137K Nov 23 22:20 143.65676.1
-rw-------. 1 root root 67K Nov 23 22:12 144.65679.1
-rw-------. 1 root root 51K Nov 23 22:12 145.65680.1
-rw-------. 1 root root 80K Nov 23 22:06 14.65547.1
-rw-------. 1 root root 53K Nov 23 22:06 146.65681.1
-rw-------. 1 root root 22K Nov 23 22:06 147.65682.1
-rw-------. 1 root root 2.5M Nov 23 22:06 148.65683.1
-rw-------. 1 root root 27K Nov 23 22:06 149.65684.1
-rw-------. 1 root root 55K Nov 23 22:06 150.65685.1
-rw-------. 1 root root 26K Nov 23 22:06 151.65686.1
-rw-------. 1 root root 20K Nov 23 22:06 152.65687.1
-rw-------. 1 root root 276K Nov 23 22:06 153.65688.1
-rw-------. 1 root root 62K Nov 23 22:06 154.65689.1
-rw-------. 1 root root 58K Nov 23 22:06 155.65690.1
-rw-------. 1 root root 300K Nov 23 22:06 15.65548.1
-rw-------. 1 root root 141K Nov 23 22:06 156.65691.1
-rw-------. 1 root root 43K Nov 23 22:06 157.65692.1
-rw-------. 1 root root 32K Nov 23 22:06 158.65693.1
-rw-------. 1 root root 29K Nov 23 22:06 159.65694.1
-rw-------. 1 root root 27K Nov 23 22:06 160.65695.1
-rw-------. 1 root root 62K Nov 23 22:06 161.65696.1
-rw-------. 1 root root 91K Nov 23 22:06 162.65697.1
-rw-------. 1 root root 38K Nov 23 22:06 163.65698.1
-rw-------. 1 root root 80K Nov 23 22:06 164.65699.1
-rw-------. 1 root root 44K Nov 23 22:06 165.65700.1
-rw-------. 1 root root 56K Nov 23 22:06 16.65549.1
-rw-------. 1 root root 28K Nov 23 22:06 166.65701.1
-rw-------. 1 root root 782K Nov 23 22:06 167.65702.1
-rw-------. 1 root root 34K Nov 23 22:06 168.65703.1
-rw-------. 1 root root 782K Nov 23 22:06 169.65704.1
-rw-------. 1 root root 33K Nov 23 22:06 170.65705.1
-rw-------. 1 root root 49K Nov 23 22:06 171.65706.1
-rw-------. 1 root root 35K Nov 23 22:06 172.65707.1
-rw-------. 1 root root 29K Nov 23 22:06 173.65708.1
-rw-------. 1 root root 29K Nov 23 22:06 174.65709.1
-rw-------. 1 root root 94K Nov 23 22:06 175.65710.1
-rw-------. 1 root root 55K Nov 23 22:06 17.65550.1
-rw-------. 1 root root 63K Nov 23 22:06 176.65711.1
-rw-------. 1 root root 36K Nov 23 22:06 177.65712.1
-rw-------. 1 root root 67K Nov 23 22:06 178.65713.1
-rw-------. 1 root root 55K Nov 23 22:06 179.65714.1
-rw-------. 1 root root 434K Nov 23 22:06 180.65715.1
-rw-------. 1 root root 37K Nov 23 22:06 181.65716.1
-rw-------. 1 root root 79K Nov 23 22:06 182.65717.1
-rw-------. 1 root root 54K Nov 23 22:06 183.65718.1
-rw-------. 1 root root 42K Nov 23 22:06 184.65719.1
-rw-------. 1 root root 25K Nov 23 22:06 185.65720.1
-rw-------. 1 root root 529K Nov 23 22:06 18.65551.1
-rw-------. 1 root root 100K Nov 23 22:06 186.65721.1
-rw-------. 1 root root 28K Nov 23 22:06 187.65722.1
-rw-------. 1 root root 42K Nov 23 22:06 188.65723.1
-rw-------. 1 root root 27K Nov 23 22:06 189.65724.1
-rw-------. 1 root root 65M Nov 23 22:19 190.65725.1
-rw-------. 1 root root 65M Nov 23 22:20 193.65728.1
-rw-------. 1 root root 65M Nov 23 22:21 194.65729.1
-rw-------. 1 root root 65M Nov 23 22:21 195.65730.1
-rw-------. 1 root root 536K Nov 23 22:06 19.65552.1
-rw-------. 1 root root 65M Nov 23 22:21 196.65731.1
-rw-------. 1 root root 65M Nov 23 22:21 197.65732.1
-rw-------. 1 root root 65M Nov 23 22:21 198.65733.1
-rw-------. 1 root root 65M Nov 23 22:21 199.65734.1
-rw-------. 1 root root 65M Nov 23 22:21 200.65735.1
-rw-------. 1 root root 35K Nov 23 22:19 201.65736.1
-rw-------. 1 root root 65M Nov 23 22:21 202.65737.1
-rw-------. 1 root root 121K Nov 23 22:19 203.65738.1
-rw-------. 1 root root 65M Nov 23 22:21 204.65739.1
-rw-------. 1 root root 407K Nov 23 22:19 205.65740.1
-rw-------. 1 root root 272K Nov 23 22:06 20.65553.1
-rw-------. 1 root root 65M Nov 23 22:21 206.65741.1
-rw-------. 1 root root 65M Nov 23 22:21 207.65742.1
-rw-------. 1 root root 65M Nov 23 22:21 208.65743.1
-rw-------. 1 root root 27K Nov 23 22:06 21.65554.1
-rw-------. 1 root root 84K Nov 23 22:12 217.65752.1
-rw-------. 1 root root 28K Nov 23 22:12 220.65755.1
-rw-------. 1 root root 30K Nov 23 22:06 22.65555.1
-rw-------. 1 root root 52K Nov 23 22:06 23.65556.1
-rw-------. 1 root root 221K Nov 23 22:20 236.65771.1
-rw-------. 1 root root 112K Nov 23 22:20 240.65775.1
-rw-------. 1 root root 97K Nov 23 22:12 242.65777.1
-rw-------. 1 root root 31K Nov 23 22:20 243.65778.1
-rw-------. 1 root root 32K Nov 23 22:06 24.65557.1
-rw-------. 1 root root 109K Nov 23 22:20 247.65782.1
-rw-------. 1 root root 35K Nov 23 22:20 248.65783.1
-rw-------. 1 root root 41K Nov 23 22:20 249.65784.1
-rw-------. 1 root root 99K Nov 23 22:18 252.65787.1
-rw-------. 1 root root 106K Nov 23 22:18 253.65788.1
-rw-------. 1 root root 37K Nov 23 22:06 25.65558.1
-rw-------. 1 root root 65M Nov 23 22:20 256.65791.1
-rw-------. 1 root root 28K Nov 23 22:12 263.65798.1
-rw-------. 1 root root 65M Nov 23 22:19 264.65799.1
-rw-------. 1 root root 60K Nov 23 22:18 265.65800.1
-rw-------. 1 root root 67K Nov 23 22:06 26.65559.1
-rw-------. 1 root root 54K Nov 23 22:18 269.65804.1
-rw-------. 1 root root 47K Nov 23 22:12 270.65805.1
-rw-------. 1 root root 45K Nov 23 22:12 271.65806.1
-rw-------. 1 root root 47K Nov 23 22:18 272.65807.1
-rw-------. 1 root root 371K Nov 23 22:18 273.65808.1
-rw-------. 1 root root 110K Nov 23 22:18 274.65809.1
-rw-------. 1 root root 1.6M Nov 23 22:19 275.65810.1
-rw-------. 1 root root 62K Nov 23 22:06 27.65560.1
-rw-------. 1 root root 65M Nov 23 22:19 276.65811.1
-rw-------. 1 root root 428K Nov 23 22:12 282.65819.1
-rw-------. 1 root root 39K Nov 23 22:12 284.65821.1
-rw-------. 1 root root 111K Nov 23 22:12 285.65822.1
-rw-------. 1 root root 29K Nov 23 22:06 28.65561.1
-rw-------. 1 root root 46K Nov 23 22:12 288.65825.1
-rw-------. 1 root root 34K Nov 23 22:12 289.65826.1
-rw-------. 1 root root 97K Nov 23 22:12 290.65827.1
-rw-------. 1 root root 27K Nov 23 22:12 292.65829.1
-rw-------. 1 root root 65M Nov 23 22:19 293.65830.1
-rw-------. 1 root root 57K Nov 23 22:12 294.65831.1
-rw-------. 1 root root 29K Nov 23 22:12 295.65832.1
-rw-------. 1 root root 146K Nov 23 22:06 29.65562.1
-rw-------. 1 root root 53K Nov 23 22:12 296.65833.1
-rw-------. 1 root root 65M Nov 23 22:19 297.65834.1
-rw-------. 1 root root 93K Nov 23 22:19 298.65835.1
-rw-------. 1 root root 25K Nov 23 22:19 300.65837.1
-rw-------. 1 root root 102K Nov 23 22:18 301.65838.1
-rw-------. 1 root root 88K Nov 23 22:12 302.65839.1
-rw-------. 1 root root 72K Nov 23 22:12 303.65840.1
-rw-------. 1 root root 66K Nov 23 22:12 304.65841.1
-rw-------. 1 root root 58K Nov 23 22:12 305.65842.1
-rw-------. 1 root root 14M Nov 23 22:06 30.65563.1
-rw-------. 1 root root 52K Nov 23 22:12 306.65843.1
-rw-------. 1 root root 55K Nov 23 22:12 307.65844.1
-rw-------. 1 root root 36K Nov 23 22:12 308.65845.1
-rw-------. 1 root root 114K Nov 23 22:12 309.65846.1
-rw-------. 1 root root 94K Nov 23 22:12 310.65847.1
-rw-------. 1 root root 118K Nov 23 22:12 311.65848.1
-rw-------. 1 root root 83K Nov 23 22:12 312.65849.1
-rw-------. 1 root root 103K Nov 23 22:12 313.65850.1
-rw-------. 1 root root 104K Nov 23 22:12 314.65851.1
-rw-------. 1 root root 42K Nov 23 22:12 315.65852.1
-rw-------. 1 root root 61K Nov 23 22:06 31.65564.1
-rw-------. 1 root root 44K Nov 23 22:12 316.65853.1
-rw-------. 1 root root 68K Nov 23 22:12 317.65854.1
-rw-------. 1 root root 96K Nov 23 22:12 318.65855.1
-rw-------. 1 root root 1.6M Nov 23 22:12 319.65856.1
-rw-------. 1 root root 38K Nov 23 22:12 320.65857.1
-rw-------. 1 root root 73K Nov 23 22:12 321.65858.1
-rw-------. 1 root root 91K Nov 23 22:12 322.65859.1
-rw-------. 1 root root 74K Nov 23 22:12 323.65860.1
-rw-------. 1 root root 44K Nov 23 22:12 324.65861.1
-rw-------. 1 root root 88K Nov 23 22:12 325.65862.1
-rw-------. 1 root root 77K Nov 23 22:06 32.65565.1
-rw-------. 1 root root 93K Nov 23 22:12 326.65863.1
-rw-------. 1 root root 36K Nov 23 22:18 329.65866.1
-rw-------. 1 root root 46K Nov 23 22:18 330.65867.1
-rw-------. 1 root root 341K Nov 23 22:18 331.65868.1
-rw-------. 1 root root 18K Nov 23 22:18 332.65869.1
-rw-------. 1 root root 28K Nov 23 22:18 333.65870.1
-rw-------. 1 root root 83K Nov 23 22:18 334.65871.1
-rw-------. 1 root root 52K Nov 23 22:18 335.65872.1
-rw-------. 1 root root 56K Nov 23 22:06 33.65566.1
-rw-------. 1 root root 29K Nov 23 22:18 336.65873.1
-rw-------. 1 root root 501K Nov 23 22:18 337.65874.1
-rw-------. 1 root root 44K Nov 23 22:18 338.65875.1
-rw-------. 1 root root 67K Nov 23 22:18 339.65876.1
-rw-------. 1 root root 460K Nov 23 22:19 340.65877.1
-rw-------. 1 root root 122K Nov 23 22:19 341.65878.1
-rw-------. 1 root root 38K Nov 23 22:19 342.65879.1
-rw-------. 1 root root 26K Nov 23 22:19 343.65880.1
-rw-------. 1 root root 65M Nov 23 22:21 344.65881.1
-rw-------. 1 root root 28K Nov 23 22:19 345.65882.1
-rw-------. 1 root root 51K Nov 23 22:06 34.65567.1
-rw-------. 1 root root 202K Nov 23 22:19 346.65883.1
-rw-------. 1 root root 36K Nov 23 22:19 347.65884.1
-rw-------. 1 root root 43K Nov 23 22:19 348.65885.1
-rw-------. 1 root root 42K Nov 23 22:19 349.65886.1
-rw-------. 1 root root 27K Nov 23 22:19 350.65887.1
-rw-------. 1 root root 26K Nov 23 22:19 351.65888.1
-rw-------. 1 root root 124K Nov 23 22:19 352.65889.1
-rw-------. 1 root root 81K Nov 23 22:19 353.65890.1
-rw-------. 1 root root 100K Nov 23 22:19 354.65891.1
-rw-------. 1 root root 101K Nov 23 22:20 355.65892.1
-rw-------. 1 root root 415K Nov 23 22:06 35.65568.1
-rw-------. 1 root root 22K Nov 23 22:20 356.65893.1
-rw-------. 1 root root 65M Nov 23 22:19 357.65894.1
-rw-------. 1 root root 65M Nov 23 22:19 358.65895.1
-rw-------. 1 root root 65M Nov 23 22:19 359.65896.1
-rw-------. 1 root root 65M Nov 23 22:19 360.65897.1
-rw-------. 1 root root 65M Nov 23 22:19 361.65898.1
-rw-------. 1 root root 65M Nov 23 22:19 362.65899.1
-rw-------. 1 root root 65M Nov 23 22:19 363.65900.1
-rw-------. 1 root root 65M Nov 23 22:19 364.65901.1
-rw-------. 1 root root 65M Nov 23 22:19 365.65902.1
-rw-------. 1 root root 395K Nov 23 22:06 36.65569.1
-rw-------. 1 root root 65M Nov 23 22:19 366.65903.1
-rw-------. 1 root root 65M Nov 23 22:19 367.65904.1
-rw-------. 1 root root 56K Nov 23 22:12 368.65905.1
-rw-------. 1 root root 283K Nov 23 22:12 369.65906.1
-rw-------. 1 root root 65M Nov 23 22:19 370.65907.1
-rw-------. 1 root root 65M Nov 23 22:19 371.65908.1
-rw-------. 1 root root 379K Nov 23 22:19 372.65909.1
-rw-------. 1 root root 97K Nov 23 22:06 37.65570.1
-rw-------. 1 root root 65M Nov 23 22:19 377.65914.1
-rw-------. 1 root root 65M Nov 23 22:19 378.65915.1
-rw-------. 1 root root 65M Nov 23 22:19 379.65916.1
-rw-------. 1 root root 65M Nov 23 22:19 380.65917.1
-rw-------. 1 root root 65M Nov 23 22:19 381.65918.1
-rw-------. 1 root root 65M Nov 23 22:19 382.65919.1
-rw-------. 1 root root 65M Nov 23 22:19 383.65920.1
-rw-------. 1 root root 65M Nov 23 22:20 384.65921.1
-rw-------. 1 root root 65M Nov 23 22:20 385.65922.1
-rw-------. 1 root root 93K Nov 23 22:06 38.65571.1
-rw-------. 1 root root 65M Nov 23 22:20 386.65923.1
-rw-------. 1 root root 65M Nov 23 22:20 387.65924.1
-rw-------. 1 root root 22K Nov 23 22:18 388.65925.1
-rw-------. 1 root root 166K Nov 23 22:18 389.65926.1
-rw-------. 1 root root 97K Nov 23 22:18 390.65927.1
-rw-------. 1 root root 65M Nov 23 22:20 391.65928.1
-rw-------. 1 root root 65M Nov 23 22:20 392.65929.1
-rw-------. 1 root root 65M Nov 23 22:20 393.65930.1
-rw-------. 1 root root 65M Nov 23 22:20 394.65931.1
-rw-------. 1 root root 65M Nov 23 22:20 395.65932.1
-rw-------. 1 root root 88K Nov 23 22:06 39.65572.1
-rw-------. 1 root root 65M Nov 23 22:20 396.65933.1
-rw-------. 1 root root 65M Nov 23 22:20 397.65934.1
-rw-------. 1 root root 65M Nov 23 22:20 398.65935.1
-rw-------. 1 root root 21K Nov 23 22:18 399.65936.1
-rw-------. 1 root root 24K Nov 23 22:18 400.65937.1
-rw-------. 1 root root 65K Nov 23 22:12 401.65938.1
-rw-------. 1 root root 65M Nov 23 22:20 402.65939.1
-rw-------. 1 root root 65M Nov 23 22:20 403.65940.1
-rw-------. 1 root root 80K Nov 23 22:18 405.65942.1
-rw-------. 1 root root 82K Nov 23 22:06 40.65573.1
-rw-------. 1 root root 65M Nov 23 22:20 406.65943.1
-rw-------. 1 root root 65M Nov 23 22:20 407.65944.1
-rw-------. 1 root root 65M Nov 23 22:20 408.65945.1
-rw-------. 1 root root 73K Nov 23 22:19 409.65946.1
-rw-------. 1 root root 65M Nov 23 22:20 410.65947.1
-rw-------. 1 root root 65M Nov 23 22:20 411.65948.1
-rw-------. 1 root root 65M Nov 23 22:20 412.65949.1
-rw-------. 1 root root 65M Nov 23 22:21 413.65950.1
-rw-------. 1 root root 100K Nov 23 22:12 414.65951.1
-rw-------. 1 root root 35K Nov 23 22:06 41.65574.1
-rw-------. 1 root root 65M Nov 23 22:21 416.65953.1
-rw-------. 1 root root 65K Nov 23 22:19 417.65954.1
-rw-------. 1 root root 65M Nov 23 22:21 418.65955.1
-rw-------. 1 root root 65K Nov 23 22:19 419.65956.1
-rw-------. 1 root root 65M Nov 23 22:21 420.65958.1
-rw-------. 1 root root 32K Nov 23 22:12 421.65959.1
-rw-------. 1 root root 79K Nov 23 22:12 422.65960.1
-rw-------. 1 root root 65M Nov 23 22:21 423.65961.1
-rw-------. 1 root root 33K Nov 23 22:12 424.65962.1
-rw-------. 1 root root 83K Nov 23 22:06 42.65575.1
-rw-------. 1 root root 25K Nov 23 22:12 426.65964.1
-rw-------. 1 root root 20K Nov 23 22:12 427.65965.1
-rw-------. 1 root root 65M Nov 23 22:20 428.65966.1
-rw-------. 1 root root 79K Nov 23 22:06 43.65576.1
-rw-------. 1 root root 65M Nov 23 22:19 445.65986.1
-rw-------. 1 root root 49K Nov 23 22:06 44.65577.1
-rw-------. 1 root root 65M Nov 23 22:20 446.65987.1
-rw-------. 1 root root 65M Nov 23 22:20 447.65988.1
-rw-------. 1 root root 22K Nov 23 22:19 455.65996.1
-rw-------. 1 root root 48K Nov 23 22:06 45.65578.1
-rw-------. 1 root root 30K Nov 23 22:19 456.65997.1
-rw-------. 1 root root 37K Nov 23 22:19 457.65998.1
-rw-------. 1 root root 27K Nov 23 22:19 458.65999.1
-rw-------. 1 root root 130K Nov 23 22:19 459.66000.1
-rw-------. 1 root root 197K Nov 23 22:18 460.66001.1
-rw-------. 1 root root 65M Nov 23 22:20 461.66002.1
-rw-------. 1 root root 23K Nov 23 22:06 46.65579.1
-rw-------. 1 root root 25K Nov 23 22:19 472.66013.1
-rw-------. 1 root root 739K Nov 23 22:18 475.66016.1
-rw-------. 1 root root 41K Nov 23 22:06 47.65580.1
-rw-------. 1 root root 65M Nov 23 22:21 481.66022.1
-rw-------. 1 root root 65M Nov 23 22:21 482.66023.1
-rw-------. 1 root root 65M Nov 23 22:19 485.66026.1
-rw-------. 1 root root 33K Nov 23 22:06 48.65581.1
-rw-------. 1 root root 65M Nov 23 22:19 486.66027.1
-rw-------. 1 root root 29K Nov 23 22:06 49.65582.1
-rw-------. 1 root root 58K Nov 23 22:20 496.66037.1
-rw-------. 1 root root 65M Nov 23 22:20 497.66038.1
-rw-------. 1 root root 65M Nov 23 22:20 498.66039.1
-rw-------. 1 root root 4.6M Nov 23 22:06 50.65583.1
-rw-------. 1 root root 31K Nov 23 22:18 508.66051.1
-rw-------. 1 root root 29K Nov 23 22:06 51.65584.1
-rw-------. 1 root root 28K Nov 23 22:06 52.65585.1
-rw-------. 1 root root 38K Nov 23 22:06 53.65586.1
-rw-------. 1 root root 56K Nov 23 22:06 54.65587.1
-rw-------. 1 root root 52K Nov 23 22:06 55.65588.1
-rw-------. 1 root root 49K Nov 23 22:06 56.65589.1
-rw-------. 1 root root 41K Nov 23 22:06 57.65590.1
-rw-------. 1 root root 55K Nov 23 22:06 58.65591.1
-rw-------. 1 root root 63K Nov 23 22:06 59.65592.1
-rw-------. 1 root root 26K Nov 23 22:06 60.65593.1
-rw-------. 1 root root 90K Nov 23 22:06 61.65594.1
-rw-------. 1 root root 44K Nov 23 22:06 62.65595.1
-rw-------. 1 root root 34K Nov 23 22:06 63.65596.1
-rw-------. 1 root root 50K Nov 23 22:06 64.65597.1
-rw-------. 1 root root 24K Nov 23 22:06 65.65598.1
-rw-------. 1 root root 38K Nov 23 22:06 6.65539.1
-rw-------. 1 root root 62K Nov 23 22:06 66.65599.1
-rw-------. 1 root root 38K Nov 23 22:06 67.65600.1
-rw-------. 1 root root 51K Nov 23 22:06 68.65601.1
-rw-------. 1 root root 51K Nov 23 22:06 69.65602.1
-rw-------. 1 root root 33K Nov 23 22:06 70.65603.1
-rw-------. 1 root root 39K Nov 23 22:06 71.65604.1
-rw-------. 1 root root 87K Nov 23 22:06 72.65605.1
-rw-------. 1 root root 26K Nov 23 22:06 73.65606.1
-rw-------. 1 root root 171K Nov 23 22:06 74.65607.1
-rw-------. 1 root root 37K Nov 23 22:06 75.65608.1
-rw-------. 1 root root 32K Nov 23 22:06 7.65540.1
-rw-------. 1 root root 51K Nov 23 22:06 76.65609.1
-rw-------. 1 root root 67K Nov 23 22:06 77.65610.1
-rw-------. 1 root root 65M Nov 23 22:19 80.65613.1
-rw-------. 1 root root 37K Nov 23 22:12 81.65614.1
-rw-------. 1 root root 26K Nov 23 22:12 82.65615.1
-rw-------. 1 root root 42K Nov 23 22:06 8.65541.1
-rw-------. 1 root root 35K Nov 23 22:18 86.65619.1
-rw-------. 1 root root 65M Nov 23 22:19 88.65621.1
-rw-------. 1 root root 125K Nov 23 22:12 92.65625.1
-rw-------. 1 root root 65M Nov 23 22:20 93.65626.1
-rw-------. 1 root root 52K Nov 23 22:12 94.65627.1
-rw-------. 1 root root 17K Nov 23 22:12 95.65628.1
-rw-------. 1 root root 47K Nov 23 22:06 9.65542.1
-rw-------. 1 root root 729K Nov 23 22:12 96.65629.1
-rw-------. 1 root root 36K Nov 23 22:12 97.65630.1
-rw-------. 1 root root 36K Nov 23 22:12 98.65631.1
-rw-------. 1 root root 45K Nov 23 22:12 99.65632.1
drwxr-xr-x. 2 root root 6 Nov 23 22:21 dirty
-rw-r--r--. 1 root root 0 Nov 23 22:04 lock
drwxr-xr-x. 2 root root 6 Nov 23 22:04 lost+found

On Fri, Nov 23, 2018 at 3:02 PM Michael Ovsiannikov
<movsia...@quantcast.com> wrote:
>

> To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/8A361401-6CEB-4801-B4F2-908D573E8919%40quantcast.com.

[Michael Ovsiannikov]

Each chunk file has 16KB header. Copy has no special sparse file handling, i.e. disk utilization after copy with be no less than logical file size. It might be simpler to understand space implication by copying one file, and inspecting corresponding chunk file sizes.

> To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/CACeDKtccmq%3DH%2B1R34coA%2BjvGZCavTABmo_uq1tHU60PXD8bNKg%40mail.gmail.com.

[Vlad Kopylov]

With one file everything seems fine. Seems to clutter around file 200
in 2500 file list.

With one files only error I see form chunks is:
11-25-2018 21:03:50.939 INFO - (DiskIo.cc:3099) 0x25ade40 0x25e8680fs

space available (0 0) status: -2 get fs available No such file or
directory 2

Sizes seems toe be fine, though 47483 on the storage folder is higher
then 31099 of the file:

[admin@vm2 ~]$ cp cur2/1000.extension g01/_data/cur/
[admin@vm2 ~]$ ls -la cur2/1000.extension g01/_data/cur/
-rw-r--r--. 1 admin admin 31099 Nov 18 12:47 cur2/1000.extension

g01/_data/cur/:
total 2
drwxr-xr-x. 0 root root 0 Nov 25 20:54 .
drwxr-xr-x. 0 root root 0 Nov 25 20:54 ..
-rw-r--r--. 0 root root 31099 Nov 25 20:54 1000.extension

[admin@vm2 ~]$ ls -la qfs/data0/
total 48
drwxrwxr-x. 4 admin admin 99 Nov 25 20:58 .
drwxrwxr-x. 6 admin admin 174 Nov 25 20:51 ..
-rw-------. 1 root root 0 Nov 25 20:51 0-fsid-907879414745623023
-rw-------. 1 root root 47483 Nov 25 20:54 7.65539.1
drwxr-xr-x. 2 root root 6 Nov 25 20:54 dirty
-rw-r--r--. 1 root root 0 Nov 25 20:51 lock
drwxr-xr-x. 2 root root 6 Nov 25 20:51 lost+found
On Sat, Nov 24, 2018 at 8:03 PM Michael Ovsiannikov

> To view this discussion on the web visit https://groups.google.com/d/msgid/qfs-devel/C86728AB-2116-4307-888A-CD80856BA9F8%40quantcast.com.