nodetool refresh throws error (link failed, file exists)

77 views
Skip to first unread message

Michael Wohlwend

<micha-1@fantasymail.de>
unread,
Jul 10, 2020, 11:20:41 AM7/10/20
to scylladb-users@googlegroups.com
Hi,

after copying snapshots from some nodes to a new node into the upload folder
and doing a refresh, I get this error, it's the last refresh. The upload
folder gets cleared after a refresh. The first 7 refreshs worked, the last one
not. The snapshot files a take on our 3.9 cluster, the new is 4.0.1
The table was empty on the new cluster, there were no files in the table folder
before the refresh.

Any help on this?

Cheers
Michael

nodetool: Scylla API server HTTP POST to URL '/storage_service/sstables/
spielplatz' failed: Failed to load new sstables:
std::filesystem::__cxx11::filesystem_error (error system:17, filesystem error:
link failed: File exists [/data/scylla-data-dirs/scylla-data/spielplatz/
testtable-76b4ed50bba811eabe38000000000006/mc-3241-big-Summary.db] [/data/
scylla-data-dirs/scylla-data/spielplatz/
testtable-76b4ed50bba811eabe38000000000006/mc-3271-big-Summary.db])
See 'nodetool help' or 'nodetool help <command>'.

2020/07/10 14:56:45 FATAL command failed: &{/usr/bin/nodetool [nodetool
refresh spielplatz testtable] [] <nil> %!s(*os.File=&{0xc0000921e0}) %!
s(*os.File=&{0xc0000921e0}) [] %!s(*syscall.SysProcAttr=<nil>) %!
s(*os.Process=&{548 0 1 {{0 0} 0 0 0 0}}) exit status 1 <nil> <nil> %!
s(bool=true) [%!s(*os.File=&{0xc000093680}) %!s(*os.File=&{0xc0000921e0}) %!
s(*os.File=&{0xc0000921e0})] [%!s(*os.File=&{0xc000093680})] [] [] %!s(chan
error=0xc0001747e0) %!s(chan struct {}=<nil>)} with err exit status 1


Raphael S. Carvalho

<raphaelsc@scylladb.com>
unread,
Jul 10, 2020, 11:58:25 AM7/10/20
to ScyllaDB users
On Fri, Jul 10, 2020 at 12:20 PM Michael Wohlwend
<mic...@fantasymail.de> wrote:
>
> Hi,
>
> after copying snapshots from some nodes to a new node into the upload folder
> and doing a refresh, I get this error, it's the last refresh. The upload
> folder gets cleared after a refresh. The first 7 refreshs worked, the last one
> not. The snapshot files a take on our 3.9 cluster, the new is 4.0.1
> The table was empty on the new cluster, there were no files in the table folder
> before the refresh.
>
> Any help on this?

Please open a github issue. I will investigate this. If refresh
failed, the upload dir shouldn't have been cleared. Any chance you
moved the upload dir files to the parent (table) directory during the
operation? The table directory was really empty during refresh, right?

Thanks.

>
> Cheers
> Michael
>
> nodetool: Scylla API server HTTP POST to URL '/storage_service/sstables/
> spielplatz' failed: Failed to load new sstables:
> std::filesystem::__cxx11::filesystem_error (error system:17, filesystem error:
> link failed: File exists [/data/scylla-data-dirs/scylla-data/spielplatz/
> testtable-76b4ed50bba811eabe38000000000006/mc-3241-big-Summary.db] [/data/
> scylla-data-dirs/scylla-data/spielplatz/
> testtable-76b4ed50bba811eabe38000000000006/mc-3271-big-Summary.db])
> See 'nodetool help' or 'nodetool help <command>'.
>
> 2020/07/10 14:56:45 FATAL command failed: &{/usr/bin/nodetool [nodetool
> refresh spielplatz testtable] [] <nil> %!s(*os.File=&{0xc0000921e0}) %!
> s(*os.File=&{0xc0000921e0}) [] %!s(*syscall.SysProcAttr=<nil>) %!
> s(*os.Process=&{548 0 1 {{0 0} 0 0 0 0}}) exit status 1 <nil> <nil> %!
> s(bool=true) [%!s(*os.File=&{0xc000093680}) %!s(*os.File=&{0xc0000921e0}) %!
> s(*os.File=&{0xc0000921e0})] [%!s(*os.File=&{0xc000093680})] [] [] %!s(chan
> error=0xc0001747e0) %!s(chan struct {}=<nil>)} with err exit status 1
>
>
> --
> You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/scylladb-users/6537564.IHxFhHYzDf%40pc21a.

Raphael S. Carvalho

<raphaelsc@scylladb.com>
unread,
Jul 10, 2020, 11:59:52 AM7/10/20
to ScyllaDB users
On Fri, Jul 10, 2020 at 12:58 PM Raphael S. Carvalho
<raph...@scylladb.com> wrote:
>
> On Fri, Jul 10, 2020 at 12:20 PM Michael Wohlwend
> <mic...@fantasymail.de> wrote:
> >
> > Hi,
> >
> > after copying snapshots from some nodes to a new node into the upload folder
> > and doing a refresh, I get this error, it's the last refresh. The upload
> > folder gets cleared after a refresh. The first 7 refreshs worked, the last one
> > not. The snapshot files a take on our 3.9 cluster, the new is 4.0.1
> > The table was empty on the new cluster, there were no files in the table folder
> > before the refresh.
> >
> > Any help on this?
>
> Please open a github issue. I will investigate this. If refresh
> failed, the upload dir shouldn't have been cleared. Any chance you
> moved the upload dir files to the parent (table) directory during the
> operation? The table directory was really empty during refresh, right?

In the github issue, please share the full logs.

Michael Wohlwend

<micha-1@fantasymail.de>
unread,
Jul 13, 2020, 8:59:29 AM7/13/20
to scylladb-users@googlegroups.com, Raphael S. Carvalho
Hi,

I wanted to be sure, that there is no error on my side.

I tried again, a few times now.
I made a fresh snapshot on the source cluster.

The dest table dir was empty, the upload dir also.

If I copy from each src snapshot to the upload dir and make a refresh, there
are "compaction manager- resharding failed:
sstables::malformed_sstable_exception" in the scylla log with ...mc-21-big-
TOC.txt not found

This happens more than one, with other toc files mentioned.
In the end it aborts with loading failed, filesystem error link failed, file
exists.


If I don't copy the toc*.txt files into the upload dir nothing happens (they
are needed, if they are copied they cause an error.

Is scylla 4.0.1 not compatible to 3.9 for refreshing?


Cheers
Michael

Avi Kivity

<avi@scylladb.com>
unread,
Jul 13, 2020, 9:20:01 AM7/13/20
to scylladb-users@googlegroups.com, Michael Wohlwend, Raphael S. Carvalho
It should work. Can you share logs?

Michael Wohlwend

<micha-1@fantasymail.de>
unread,
Jul 13, 2020, 10:58:27 AM7/13/20
to scylladb-users@googlegroups.com, Avi Kivity, Raphael S. Carvalho
Does nodetool refresh return before the refresh is completed?

I tried it manually and it worked. I did it with my go script and got errors.
I have the impression, that there is a timing isssue, so I added a sleep 20s
after the refresh and this *works*

So it seems nodetool refresh returns, but the refresh isn't completed.
My go programm copies data from the next node at once after the previous
"nodetool refresh" returns and starts the next refresh. That causes the error.

I think, nodetool refresh should return after the refresh ended, not slightly
before.


Cheers,
Michael

Avi Kivity

<avi@scylladb.com>
unread,
Jul 13, 2020, 11:16:39 AM7/13/20
to micha-1@fantasymail.de, scylladb-users@googlegroups.com, Raphael S. Carvalho

On 13/07/2020 17.58, Michael Wohlwend wrote:
> Does nodetool refresh return before the refresh is completed?


No. Or at least, it is written not to.


> I tried it manually and it worked. I did it with my go script and got errors.
> I have the impression, that there is a timing isssue, so I added a sleep 20s
> after the refresh and this *works*
>
> So it seems nodetool refresh returns, but the refresh isn't completed.
> My go programm copies data from the next node at once after the previous
> "nodetool refresh" returns and starts the next refresh. That causes the error.
>
> I think, nodetool refresh should return after the refresh ended, not slightly
> before.


It should, but maybe there's a bug.


"notetool refresh" operation is locked, and should show in the logs
between the lines


    Loading new SSTables for


and


   Done loading new SSTables for


Can you show where the errors you encountered are relative to those lines?

Raphael S. Carvalho

<raphaelsc@scylladb.com>
unread,
Jul 13, 2020, 11:21:40 AM7/13/20
to Avi Kivity, micha-1@fantasymail.de, ScyllaDB users
On Mon, Jul 13, 2020 at 12:16 PM Avi Kivity <a...@scylladb.com> wrote:
>
>
> On 13/07/2020 17.58, Michael Wohlwend wrote:
> > Does nodetool refresh return before the refresh is completed?
>
>
> No. Or at least, it is written not to.
>
>
> > I tried it manually and it worked. I did it with my go script and got errors.
> > I have the impression, that there is a timing isssue, so I added a sleep 20s
> > after the refresh and this *works*
> >
> > So it seems nodetool refresh returns, but the refresh isn't completed.
> > My go programm copies data from the next node at once after the previous
> > "nodetool refresh" returns and starts the next refresh. That causes the error.
> >
> > I think, nodetool refresh should return after the refresh ended, not slightly
> > before.
>
>
> It should, but maybe there's a bug.
>
>
> "notetool refresh" operation is locked, and should show in the logs
> between the lines
>
>
> Loading new SSTables for
>
>
> and
>
>
> Done loading new SSTables for
>
>
> Can you show where the errors you encountered are relative to those lines?

Perhaps it's a directory synchronization issue in the upload dir, so
the fix would be about issuing a sync on it. Reading the code, it
looks like we don't do any async work which could continue after
refresh returns.

Dan Yasny

<dyasny@scylladb.com>
unread,
Jul 13, 2020, 11:30:58 AM7/13/20
to ScyllaDB users, Avi Kivity, micha-1@fantasymail.de
On Mon, Jul 13, 2020 at 11:21 AM Raphael S. Carvalho <raph...@scylladb.com> wrote:
On Mon, Jul 13, 2020 at 12:16 PM Avi Kivity <a...@scylladb.com> wrote:
>
>
> On 13/07/2020 17.58, Michael Wohlwend wrote:
> > Does nodetool refresh return before the refresh is completed?
>
>
> No. Or at least, it is written not to.
>
>
> > I tried it manually and it worked. I did it with my go script and got errors.
> > I have the impression, that there is a timing isssue, so I added a sleep 20s
> > after the refresh and this *works*
> >
> > So it seems nodetool refresh returns, but the refresh isn't completed.
> > My go programm copies data from the next node at once after the previous
> > "nodetool refresh" returns and starts the next refresh. That causes the error.
> >
> > I think, nodetool refresh should return after the refresh ended, not slightly
> > before.
>
>
> It should, but maybe there's a bug.
>
>
> "notetool refresh" operation is locked, and should show in the logs
> between the lines
>
>
>      Loading new SSTables for
>
>
> and
>
>
>     Done loading new SSTables for
>
>
> Can you show where the errors you encountered are relative to those lines?

Perhaps it's a directory synchronization issue in the upload dir, so
the fix would be about issuing a sync on it. Reading the code, it
looks like we don't do any async work which could continue after
refresh returns.

In my work on unirestore, I had to take care to always ensure I have copied the full 9-file sets of sstables into the upload directory, otherwise similar errors were thrown. Maybe it would make sense to make things more resilient and warn on incomplete file sets, but ignore them and move on to the full sets

 

>
>
>
> >
> > Cheers,
> >       Michael
> >
> >
> >
> > Am Montag, 13. Juli 2020, 15:19:57 CEST schrieb Avi Kivity:
> >> It should work. Can you share logs?
> >>
> >> On 13/07/2020 15.59, Michael Wohlwend wrote:
> >>> Hi,
> >>>
> >>> I wanted to be sure, that there is no error on my side.
> >>>
> >>> I tried again, a few times now.
> >>> I made a fresh snapshot on the source cluster.
> >>>
> >>> The dest table dir was empty, the upload dir also.
> >>>
> >>> If I copy from each src snapshot to the upload dir and make a refresh,
> >>> there are "compaction manager- resharding failed:
> >>> sstables::malformed_sstable_exception" in the scylla log with
> >>> ...mc-21-big-
> >>> TOC.txt not found
> >>>
> >
> >

--
You received this message because you are subscribed to the Google Groups "ScyllaDB users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to scylladb-user...@googlegroups.com.

Avi Kivity

<avi@scylladb.com>
unread,
Jul 13, 2020, 11:38:43 AM7/13/20
to Raphael S. Carvalho, micha-1@fantasymail.de, ScyllaDB users

On 13/07/2020 18.21, Raphael S. Carvalho wrote:
> On Mon, Jul 13, 2020 at 12:16 PM Avi Kivity <a...@scylladb.com> wrote:
>>
>> On 13/07/2020 17.58, Michael Wohlwend wrote:
>>> Does nodetool refresh return before the refresh is completed?
>>
>> No. Or at least, it is written not to.
>>
>>
>>> I tried it manually and it worked. I did it with my go script and got errors.
>>> I have the impression, that there is a timing isssue, so I added a sleep 20s
>>> after the refresh and this *works*
>>>
>>> So it seems nodetool refresh returns, but the refresh isn't completed.
>>> My go programm copies data from the next node at once after the previous
>>> "nodetool refresh" returns and starts the next refresh. That causes the error.
>>>
>>> I think, nodetool refresh should return after the refresh ended, not slightly
>>> before.
>>
>> It should, but maybe there's a bug.
>>
>>
>> "notetool refresh" operation is locked, and should show in the logs
>> between the lines
>>
>>
>> Loading new SSTables for
>>
>>
>> and
>>
>>
>> Done loading new SSTables for
>>
>>
>> Can you show where the errors you encountered are relative to those lines?
> Perhaps it's a directory synchronization issue in the upload dir, so
> the fix would be about issuing a sync on it. Reading the code, it
> looks like we don't do any async work which could continue after
> refresh returns.
>

sync is only needed to protect from host kernel crashes or power failures.

Michael Wohlwend

<micha-1@fantasymail.de>
unread,
Jul 14, 2020, 10:06:23 AM7/14/20
to scylladb-users@googlegroups.com, Avi Kivity, Raphael S. Carvalho
Am Montag, 13. Juli 2020, 17:16:35 CEST schrieb Avi Kivity:
> On 13/07/2020 17.58, Michael Wohlwend wrote:
>
> "notetool refresh" operation is locked, and should show in the logs
> between the lines
>
>
> Loading new SSTables for
>
>
> and
>
>
> Done loading new SSTables for
>
>
> Can you show where the errors you encountered are relative to those lines?
>


Here is part of the log.
Around line 190 is the first link error,

The errors are between "Loading new.." and "Done loading...", but there line
saying "resharding" after the refresh ended. Does the resharding belong to the
refresh?




Cheers
Michael


---------------------------- snippet of log ------------------------------





Jul 10 14:43:40 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...


Jul 10 14:43:40 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:43:40 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 2835 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 1642 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 1590 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 1597 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 1596 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 1596 usec
Jul 10 14:43:40 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 1597 usec
Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards
Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2768-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2874-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2867-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2896-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2911-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2872-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2857-big-Data.db:level=0, ]
Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)
Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...
Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 2471 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 1204 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 2473 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 2048 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 2474 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 2479 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 4013 usec







Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards






Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2768-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2874-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2867-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2896-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2911-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2872-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2857-big-Data.db:level=0, ]
Jul 10 14:43:41 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)




Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...



Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:43:42 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 2471 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 1204 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 2473 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 2048 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 2474 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 2479 usec
Jul 10 14:43:42 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 4013 usec
Jul 10 14:43:43 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: f23de17f9cab659488452062f1eb03af648f4a09:11240848580-
a56nm1-la flows (1864187 bytes)
Jul 10 14:43:44 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 148374a72654a5522b04006de2450026e403aadc:
11502140640-08932d-m1 flows (1696579 bytes)
Jul 10 14:43:45 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-
y1vu1z-la flows (1710805 bytes)
Jul 10 14:43:46 mw-scylla01 scylla: [shard 0] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2865-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2878-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2850-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2900-big-Data.db:level=0, ]
Jul 10 14:43:46 mw-scylla01 scylla: [shard 6] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2864-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2905-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2885-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2879-big-Data.db:level=0, ]




Jul 10 14:43:46 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards







Jul 10 14:43:47 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...



Jul 10 14:43:48 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 2060b2d2e02b71d7e41186125d401d7fa1e02809:11395747440-
n6gemh-la flows (1589541 bytes)
Jul 10 14:43:48 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 49392a52d4662df024c6ac1ee96a2bad4ba026ed:11502073492-
i03alc-m1 flows (2319919 bytes)
Jul 10 14:43:48 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 8cec8be3888546e1a26e75d546071925c8eef5ba:11377040389-
rr81q2-la flows (1706113 bytes)
Jul 10 14:43:51 mw-scylla01 scylla: [shard 6] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2932-big-Data.db:level=0, ].
10242901 bytes to 8458825 (~82% of original) in 5083ms = 1.59MB/s. ~16512
total partitions merged to 13552.
Jul 10 14:43:52 mw-scylla01 scylla: [shard 0] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2933-big-Data.db:level=0, ].
9242995 bytes to 7724397 (~83% of original) in 6353ms = 1.16MB/s. ~16256 total
partitions merged to 13421.
Jul 10 14:43:52 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:43:53 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 5489963 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 5489982 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 5489976 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 5490024 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 5490025 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 5490024 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 5491509 usec
Jul 10 14:43:53 mw-scylla01 scylla: [shard 0] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2871-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2886-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2775-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2881-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2903-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2918-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2912-big-Data.db:level=0, ].
27954263 bytes to 27949143 (~99% of original) in 12603ms = 2.11MB/s. ~48384
total partitions merged to 47986.
Jul 10 14:43:54 mw-scylla01 scylla: [shard 0] compaction_manager - resharding
failed: sstables::malformed_sstable_exception (/data/scylla-data-dirs/scylla-
data/playground/stuff_data-76b4ed50bba811eabe38000000000006/mc-2912-big-
TOC.txt: file not found)
Jul 10 14:43:54 mw-scylla01 scylla: [shard 1] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2874-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2867-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2782-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2895-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2926-big-Data.db:level=0, ]
Jul 10 14:43:54 mw-scylla01 scylla: [shard 0] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2893-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2917-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2888-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2910-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2892-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2919-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2768-big-Data.db:level=0, ]
Jul 10 14:43:55 mw-scylla01 scylla: [shard 1] seastar_memory - oversized
allocation: 1867776 bytes. This is non-fatal, but could lead to latency and/or
fragmentation issues. Please report: at 0x33286dd#012 0x2e42c24#012
0x2e474ab#012 0x12c39d7#012 0x12d6812#012 0x12d94df#012 0x12da258#012
0x130b23e#012 0x12bacf9#012 0x130c7de#012 0x10c68c6#012 0x10c8558#012
0x10c8b68#012 0x1346d27#012 0x1348328#012 0x13489a7#012 0x314817c
Jul 10 14:43:55 mw-scylla01 scylla: [shard 1] large_data - Writing large cell
playground/stuff_data: f23de17f9cab659488452062f1eb03af648f4a09:11240848580-
a56nm1-la flows (1864187 bytes)
Jul 10 14:44:00 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-
y1vu1z-la flows (1710805 bytes)
Jul 10 14:44:01 mw-scylla01 scylla: [shard 1] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2964-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2860-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2761-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2945-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2934-big-Data.db:level=0, ]
Jul 10 14:44:01 mw-scylla01 scylla: [shard 6] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2949-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2918-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2962-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2932-big-Data.db:level=0, ]




Jul 10 14:44:01 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards



Jul 10 14:44:02 mw-scylla01 scylla: [shard 1] compaction - Resharded 5
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2948-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2956-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2950-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2951-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2959-big-Data.db:level=0, ].
9223599 bytes to 9201953 (~99% of original) in 7968ms = 1.10MB/s. ~16512 total
partitions merged to 16135.
Jul 10 14:44:02 mw-scylla01 scylla: [shard 4] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2954-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2925-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2951-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2881-big-Data.db:level=0, ]




Jul 10 14:44:02 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...


| Jul 10 14:44:03 mw-scylla01 scylla: [shard 4] large_data - Writing large
cell playground/stuff_data: 148374a72654a5522b04006de2450026e403aadc:
11502140640-08932d-m1 flows (1696579 bytes)
| Jul 10 14:44:04 mw-scylla01 scylla: [shard 4] large_data - Writing large
cell playground/stuff_data:
244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-y1vu1z-la flows (1710805
bytes)
| Jul 10 14:44:04 mw-scylla01 scylla: [shard 6] large_data - Writing large
cell playground/stuff_data: 49392a52d4662df024c6ac1ee96a2bad4ba026ed:
11502073492-i03alc-m1 flows (2319919 bytes)
| Jul 10 14:44:05 mw-scylla01 scylla: [shard 6] large_data - Writing large
cell playground/stuff_data: 8cec8be3888546e1a26e75d546071925c8eef5ba:
11377040389-rr81q2-la flows (1706113 bytes)
| Jul 10 14:44:05 mw-scylla01 scylla: [shard 6] large_data - Writing large
cell playground/stuff_data:
d836f229742291aba45d4ce61930d979bc4f1d00:11395818293-iga3nk-la flows (1402701
bytes)
| Jul 10 14:44:06 mw-scylla01 scylla: [shard 1] compaction - Compacted 5
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2969-big-Data.db:level=0, ].
5103676 bytes to 5076630 (~99% of original) in 4401ms = 1.10MB/s. ~9728 total
partitions merged to 9205.
| Jul 10 14:44:06 mw-scylla01 scylla: [shard 4] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2979-big-Data.db:level=0, ].
6247055 bytes to 6145066 (~98% of original) in 4828ms = 1.21MB/s. ~9600 total
partitions merged to 9184.
| Jul 10 14:44:07 mw-scylla01 scylla: [shard 6] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2974-big-Data.db:level=0, ].
13933465 bytes to 11535190 (~82% of original) in 5900ms = 1.86MB/s. ~22656
total partitions merged to 18199.
| Jul 10 14:44:07 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables
in /data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 6107880 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 6107865 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 6107898 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 6107511 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 6107944 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 6107945 usec
| Jul 10 14:44:08 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 6109516 usec
| Jul 10 14:44:09 mw-scylla01 scylla: [shard 0] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2947-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2955-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2963-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2957-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2965-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2966-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2960-big-Data.db:level=0, ].
30814216 bytes to 29666021 (~96% of original) in 15161ms = 1.87MB/s. ~55808
total partitions merged to 52771.
| Jul 10 14:44:09 mw-scylla01 scylla: [shard 3] compaction - Compacting [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2952-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2950-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2957-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2775-big-Data.db:level=0, ]
| Jul 10 14:44:09 mw-scylla01 scylla: [shard 0] compaction_manager -
resharding failed: sstables::malformed_sstable_exception (/data/scylla-data-
dirs/scylla-data/playground/stuff_data-76b4ed50bba811eabe38000000000006/
mc-2947-big-TOC.txt: file not found)
| Jul 10 14:44:09 mw-scylla01 scylla: [shard 1] compaction - Resharding [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2940-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2942-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2972-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2944-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2958-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2943-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2946-big-Data.db:level=0, ]
| Jul 10 14:44:09 mw-scylla01 scylla: [shard 0] compaction - Resharding [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2893-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2917-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2888-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2768-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2973-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2967-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2953-big-Data.db:level=0, ]
| Jul 10 14:44:19 mw-scylla01 scylla: [shard 3] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2992-big-Data.db:level=0, ].
12223528 bytes to 8216897 (~67% of original) in 10049ms = 0.78MB/s. ~23168
total partitions merged to 15376.
| Jul 10 14:44:20 mw-scylla01 scylla: [shard 0] compaction - Compacting [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3019-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2999-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2936-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2984-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2941-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2990-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2933-big-Data.db:level=0, ]
| Jul 10 14:44:20 mw-scylla01 scylla: [shard 2] compaction - Compacting [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2889-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2986-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2935-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2956-big-Data.db:level=0, ]
| Jul 10 14:44:20 mw-scylla01 scylla: [shard 6] compaction - Compacting [/
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3011-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2960-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3012-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2974-big-Data.db:level=0, ]
| O




Jul 10 14:44:20 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards








Jul 10 14:44:20 mw-scylla01 scylla: [shard 1] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2994-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3002-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2997-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2998-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3010-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3013-big-Data.db:level=0, ].
12812738 bytes to 12772626 (~99% of original) in 10881ms = 1.12MB/s. ~23552
total partitions merged to 22806.
Jul 10 14:44:20 mw-scylla01 scylla: [shard 5] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2959-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2994-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2966-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2903-big-Data.db:level=0, ]
Jul 10 14:44:20 mw-scylla01 scylla: [shard 1] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2948-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2997-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2969-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2985-big-Data.db:level=0, ]
Jul 10 14:44:21 mw-scylla01 scylla: [shard 5] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)






Jul 10 14:44:21 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...






Jul 10 14:44:23 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 2060b2d2e02b71d7e41186125d401d7fa1e02809:11395747440-
n6gemh-la flows (1589541 bytes)
Jul 10 14:44:23 mw-scylla01 scylla: [shard 2] large_data - Writing large cell
playground/stuff_data: f23de17f9cab659488452062f1eb03af648f4a09:11240848580-
a56nm1-la flows (1864187 bytes)
Jul 10 14:44:23 mw-scylla01 scylla: [shard 0] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3003-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3004-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3005-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3006-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2993-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3001-big-Data.db:level=0, ].
29872302 bytes to 27471391 (~91% of original) in 13946ms = 1.88MB/s. ~54912
total partitions merged to 49518.
Jul 10 14:44:24 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 49392a52d4662df024c6ac1ee96a2bad4ba026ed:11502073492-
i03alc-m1 flows (2319919 bytes)
Jul 10 14:44:24 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 8cec8be3888546e1a26e75d546071925c8eef5ba:11377040389-
rr81q2-la flows (1706113 bytes)
Jul 10 14:44:24 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: d836f229742291aba45d4ce61930d979bc4f1d00:11395818293-
iga3nk-la flows (1402701 bytes)
Jul 10 14:44:27 mw-scylla01 scylla: [shard 2] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3026-big-Data.db:level=0, ].
15111062 bytes to 8339556 (~55% of original) in 6867ms = 1.16MB/s. ~27520
total partitions merged to 15191.
Jul 10 14:44:27 mw-scylla01 scylla: [shard 5] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3022-big-Data.db:level=0, ].
13968618 bytes to 10534601 (~75% of original) in 6731ms = 1.49MB/s. ~24832
total partitions merged to 18279.
Jul 10 14:44:27 mw-scylla01 scylla: [shard 6] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3023-big-Data.db:level=0, ].
17854521 bytes to 13410102 (~75% of original) in 6868ms = 1.86MB/s. ~27776
total partitions merged to 21116.
Jul 10 14:44:27 mw-scylla01 scylla: [shard 2] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2989-big-Data.db:level=0, ]
Jul 10 14:44:27 mw-scylla01 scylla: [shard 0] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2982-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2983-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2981-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2980-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2991-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3020-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3015-big-Data.db:level=0, ]
Jul 10 14:44:27 mw-scylla01 scylla: [shard 1] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3016-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3009-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2996-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3008-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3000-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3007-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2995-big-Data.db:level=0, ]
Jul 10 14:44:28 mw-scylla01 scylla: [shard 1] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3018-big-Data.db:level=0, ].
19229946 bytes to 10765106 (~55% of original) in 7614ms = 1.35MB/s. ~34304
total partitions merged to 19409.
Jul 10 14:44:28 mw-scylla01 scylla: [shard 0] compaction - Compacted 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3017-big-Data.db:level=0, ].
18713479 bytes to 11949752 (~63% of original) in 7974ms = 1.43MB/s. ~32512
total partitions merged to 20953.
Jul 10 14:44:28 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:44:28 mw-scylla01 scylla: [shard 2] compaction - Resharded 1
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3027-big-Data.db:level=0, ].
17946 bytes to 18014 (~100% of original) in 800ms = 0.02MB/s. ~128 total
partitions merged to 2.


Jul 10 14:44:28 mw-scylla01 scylla: [shard 0] storage_service - Loading of
new tables failed to playground.stuff_data due to filesystem error: link failed:
File exists [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2892-big-Data.db] [/data/scylla-
data-dirs/scylla-data/playground/stuff_data-76b4ed50bba811eabe38000000000006/
mc-3027-big-Data.db]


Jul 10 14:44:28 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 6946999 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 6976897 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 6976923 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 6976905 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 6976907 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 6976907 usec
Jul 10 14:44:28 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 6976904 usec


Jul 10 14:44:28 mw-scylla01 scylla: [shard 3] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3027-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3013-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3006-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2992-big-Data.db:level=0, ]
Jul 10 14:44:29 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)
Jul 10 14:44:32 mw-scylla01 scylla: [shard 1] seastar_memory - oversized
allocation: 1712128 bytes. This is non-fatal, but could lead to latency and/or
fragmentation issues. Please report: at 0x33286dd#012 0x2e42c24#012
0x2e474ab#012 0x12c39d7#012 0x12d6812#012 0x12d94df#012 0x12da258#012
0x130b23e#012 0x12bacf9#012 0x130c7de#012 0x10c68c6#012 0x10c8558#012
0x10c8b68#012 0x1346d27#012 0x1348328#012 0x13489a7#012 0x314817c
Jul 10 14:44:32 mw-scylla01 scylla: [shard 1] large_data - Writing large cell
playground/stuff_data: 244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-
y1vu1z-la flows (1710805 bytes)
Jul 10 14:44:32 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: f23de17f9cab659488452062f1eb03af648f4a09:11240848580-
a56nm1-la flows (1864187 bytes)
Jul 10 14:44:33 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 148374a72654a5522b04006de2450026e403aadc:
11502140640-08932d-m1 flows (1696579 bytes)
Jul 10 14:44:35 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-
y1vu1z-la flows (1710805 bytes)
Jul 10 14:44:37 mw-scylla01 scylla: [shard 1] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3041-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3021-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3036-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3037-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3040-big-Data.db:level=0, ].
14320319 bytes to 14243443 (~99% of original) in 9789ms = 1.39MB/s. ~24192
total partitions merged to 23650.
Jul 10 14:44:37 mw-scylla01 scylla: [shard 4] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2993-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2965-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3021-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2979-big-Data.db:level=0, ]
Jul 10 14:44:37 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3040-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2998-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3005-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3026-big-Data.db:level=0, ]
Jul 10 14:44:39 mw-scylla01 scylla: [shard 3] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3048-big-Data.db:level=0, ].
14689072 bytes to 10537292 (~71% of original) in 10951ms = 0.92MB/s. ~27520
total partitions merged to 19376.
Jul 10 14:44:39 mw-scylla01 scylla: [shard 2] large_data - Writing large cell
playground/stuff_data: f23de17f9cab659488452062f1eb03af648f4a09:11240848580-
a56nm1-la flows (1864187 bytes)
Jul 10 14:44:41 mw-scylla01 scylla: [shard 4] large_data - Writing large cell
playground/stuff_data: 148374a72654a5522b04006de2450026e403aadc:
11502140640-08932d-m1 flows (1696579 bytes)
Jul 10 14:44:41 mw-scylla01 scylla: [shard 0] compaction - Resharded 7
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3034-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3014-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3029-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3030-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3025-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3033-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3024-big-Data.db:level=0, ].
27784209 bytes to 27775962 (~99% of original) in 13878ms = 1.91MB/s. ~48000
total partitions merged to 47585.
Jul 10 14:44:41 mw-scylla01 scylla: [shard 5] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3001-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3036-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3029-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3022-big-Data.db:level=0, ]
Jul 10 14:44:41 mw-scylla01 scylla: [shard 6] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3030-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3002-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3037-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3023-big-Data.db:level=0, ]
Jul 10 14:44:41 mw-scylla01 scylla: [shard 0] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3010-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3024-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3003-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3017-big-Data.db:level=0, ]
Jul 10 14:44:42 mw-scylla01 scylla: [shard 5] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)
Jul 10 14:44:42 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 2060b2d2e02b71d7e41186125d401d7fa1e02809:11395747440-
n6gemh-la flows (1589541 bytes)
Jul 10 14:44:42 mw-scylla01 scylla: [shard 4] large_data - Writing large cell
playground/stuff_data: 244e96767f7298fdfb7021a33a285daf8307cae7:11395276228-
y1vu1z-la flows (1710805 bytes)
Jul 10 14:44:43 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 49392a52d4662df024c6ac1ee96a2bad4ba026ed:11502073492-
i03alc-m1 flows (2319919 bytes)
Jul 10 14:44:43 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: 8cec8be3888546e1a26e75d546071925c8eef5ba:11377040389-
rr81q2-la flows (1706113 bytes)
Jul 10 14:44:43 mw-scylla01 scylla: [shard 6] large_data - Writing large cell
playground/stuff_data: d836f229742291aba45d4ce61930d979bc4f1d00:11395818293-
iga3nk-la flows (1402701 bytes)
Jul 10 14:44:44 mw-scylla01 scylla: [shard 2] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3047-big-Data.db:level=0, ].
17477479 bytes to 10543093 (~60% of original) in 6379ms = 1.58MB/s. ~32128
total partitions merged to 19221.
Jul 10 14:44:44 mw-scylla01 scylla: [shard 4] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3028-big-Data.db:level=0, ].
22147883 bytes to 13111408 (~59% of original) in 7283ms = 1.72MB/s. ~36224
total partitions merged to 21797.
Jul 10 14:44:45 mw-scylla01 scylla: [shard 6] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3044-big-Data.db:level=0, ].
15416093 bytes to 13916203 (~90% of original) in 3493ms = 3.80MB/s. ~24576
total partitions merged to 21934.
Jul 10 14:44:45 mw-scylla01 scylla: [shard 0] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3031-big-Data.db:level=0, ].
14416497 bytes to 12346715 (~85% of original) in 3402ms = 3.46MB/s. ~25344
total partitions merged to 21759.
Jul 10 14:44:45 mw-scylla01 scylla: [shard 5] compaction - Compacted 4
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3043-big-Data.db:level=0, ].
22860804 bytes to 12888351 (~56% of original) in 3951ms = 3.11MB/s. ~38912
total partitions merged to 21713.
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-30-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-23-big-Data.db:level=0, ]
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-37-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-44-big-Data.db:level=0, ]
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacted 2
sstables to [/data/scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-51-big-Data.db:level=0, ]. 10825
bytes to 5576 (~51% of original) in 518ms = 0.01MB/s. ~256 total partitions
merged to 1.
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacted 2
sstables to [/data/scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-58-big-Data.db:level=0, ]. 10478
bytes to 5550 (~52% of original) in 389ms = 0.01MB/s. ~256 total partitions
merged to 1.
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-58-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-51-big-Data.db:level=0, ]
Jul 10 14:49:04 mw-scylla01 scylla: [shard 2] compaction - Compacted 2
sstables to [/data/scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-65-big-Data.db:level=0, ]. 11126
bytes to 5573 (~50% of original) in 275ms = 0.02MB/s. ~256 total partitions
merged to 1.
Jul 10 14:49:21 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-72-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-65-big-Data.db:level=0, ]
Jul 10 14:49:21 mw-scylla01 scylla: [shard 2] compaction - Compacted 2
sstables to [/data/scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-86-big-Data.db:level=0, ]. 10811
bytes to 5590 (~51% of original) in 310ms = 0.02MB/s. ~256 total partitions
merged to 1.
Jul 10 14:49:21 mw-scylla01 scylla: [shard 2] compaction - Compacting [/data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-79-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-86-big-Data.db:level=0, ]
Jul 10 14:49:22 mw-scylla01 scylla: [shard 2] compaction - Compacted 2
sstables to [/data/scylla-data-dirs/scylla-data/system/
truncated-38c19fd0fb863310a4b70d0cc66628aa/mc-93-big-Data.db:level=0, ]. 10826
bytes to 5590 (~51% of original) in 317ms = 0.02MB/s. ~256 total partitions
merged to 1.








Jul 10 14:55:00 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...






Jul 10 14:55:00 mw-scylla01 scylla: [shard 0] table - Reshuffling SSTables in /
data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006...
Jul 10 14:55:00 mw-scylla01 scylla: [shard 0] storage_service - CF
playground.stuff_data at shard 0 had SSTables writes disabled for 201 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 6] storage_service - CF
playground.stuff_data at shard 6 had SSTables writes disabled for 157 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 4] storage_service - CF
playground.stuff_data at shard 4 had SSTables writes disabled for 153 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 3] storage_service - CF
playground.stuff_data at shard 3 had SSTables writes disabled for 167 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 5] storage_service - CF
playground.stuff_data at shard 5 had SSTables writes disabled for 153 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 1] storage_service - CF
playground.stuff_data at shard 1 had SSTables writes disabled for 174 usec
Jul 10 14:55:00 mw-scylla01 scylla: [shard 2] storage_service - CF
playground.stuff_data at shard 2 had SSTables writes disabled for 167 usec




Jul 10 14:55:01 mw-scylla01 scylla: [shard 0] storage_service - Done loading
new SSTables for playground.stuff_data for all shards



Jul 10 14:55:01 mw-scylla01 scylla: [shard 1] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3038-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3064-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3065-big-Data.db:level=0, ]
Jul 10 14:55:01 mw-scylla01 scylla: [shard 0] compaction - Resharding [/data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3055-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3049-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3042-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3057-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3051-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3068-big-Data.db:level=0, /data/
scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3039-big-Data.db:level=0, ]
Jul 10 14:55:01 mw-scylla01 scylla: [shard 1] large_data - Writing large cell
playground/stuff_data: 148374a72654a5522b04006de2450026e403aadc:
11502140640-08932d-m1 flows (1696579 bytes)
Jul 10 14:55:01 mw-scylla01 scylla: [shard 0] large_data - Writing large cell
playground/stuff_data: 110e0e7b0c721da04189bedbda0f603e5d6b0cf3:11501780316-
seq5j1-m1 flows (1458671 bytes)




Jul 10 14:55:02 mw-scylla01 scylla: [shard 0] storage_service - Loading new
SSTables for playground.stuff_data...










Benny Halevy

<bhalevy@scylladb.com>
unread,
Jul 14, 2020, 12:03:03 PM7/14/20
to scylladb-users@googlegroups.com, Avi Kivity, Raphael S. Carvalho
Thanks.

It looks like we have a collision on the target sstable
across different shards:

Jul 10 14:44:28 mw-scylla01 scylla: [shard 2] compaction - Resharded 1
sstables to [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-3027-big-Data.db:level=0, ].
17946 bytes to 18014 (~100% of original) in 800ms = 0.02MB/s. ~128 total
partitions merged to 2.


Jul 10 14:44:28 mw-scylla01 scylla: [shard 0] storage_service - Loading of
new tables failed to playground.stuff_data due to filesystem error: link failed:
File exists [/data/scylla-data-dirs/scylla-data/playground/
stuff_data-76b4ed50bba811eabe38000000000006/mc-2892-big-Data.db] [/data/scylla-
data-dirs/scylla-data/playground/stuff_data-76b4ed50bba811eabe38000000000006/
mc-3027-big-Data.db]

Raphael speculates that it may happen in 4.0 due to
lack of synchronization between refresh and resharding.

Raphael S. Carvalho

<raphaelsc@scylladb.com>
unread,
Jul 15, 2020, 12:24:37 PM7/15/20
to Benny Halevy, ScyllaDB users, Avi Kivity
Reproduced this exact problem with a script as simple as this:
#!/bin/bash

table_dir=/home/raphaelsc/scylla/data/data/keyspace4/standard1-a5e12920c60e11eaab49000000000000
keyspace=keyspace4
table=standard1
from=$table_dir/backup
to=$table_dir/upload

for i in {1..10};
do
echo "Running iteration $i..."
echo "Cleaning $to"
rm -rf $to/*
echo "Creating links from $from to $to"
ln $from/* $to
echo "Refreshing keyspace $keyspace.$table"
/home/raphaelsc/scylla-tools-java/bin/nodetool refresh $keyspace $table
echo "Done with iteration $i"
done


ERROR 2020-07-15 13:11:35,885 [shard 0] storage_service - Loading of
new tables failed to keyspace4.standard1 due to filesystem error: link
failed: No such file or directory
[./data/data/keyspace4/standard1-a5e12920c60e11eaab49000000000000/mc-5374-big-TOC.txt]
[./data/data/keyspace4/standard1-a5e12920c60e11eaab49000000000000/mc-5447-big-TOC.txt.tmp]


The problem is the lack of synchronization between refresh and
resharding. This problem will not happen anymore after we stopped
accepting loading from the main directory. Because refresh will not
pick up the SSTables being operated by resharding. Backporting the
patch "table: no longer accept online loading of SSTable files in the
main directory" to older versions implies that we prevent their users
from relying on a feature that may be part of their application. On
older versions, refresh only proceeds after disabling writes for a
table using table::disable_sstable_write(), perhaps we should fix
disable_sstable_write() to only return after resharding is done for
this particular table. Thoughts?!

Avi Kivity

<avi@scylladb.com>
unread,
Jul 15, 2020, 1:17:54 PM7/15/20
to Raphael S. Carvalho, Benny Halevy, ScyllaDB users
Sounds reasonable.

Raphael S. Carvalho

<raphaelsc@scylladb.com>
unread,
Jul 22, 2020, 9:48:01 AM7/22/20
to micha-1@fantasymail.de, ScyllaDB users, Avi Kivity
This issue can now be tracked via https://github.com/scylladb/scylla/issues/6903
Reply all
Reply to author
Forward
0 new messages