Metrics disk usage on sidecar mode
18 views
Skip to first unread message
Dan Sarisky
Feb 25, 2020, 7:35:04 PM2/25/20
to TLP Apache Cassandra Reaper users
When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do expect it will stabilize eventually.
Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious problems ...
Alexander Dejanovski
Feb 26, 2020, 3:08:05 AM2/26/20
to Dan Sarisky, TLP Apache Cassandra Reaper users
Hi Dan,
yes, the problem has been reported and we have a data modelling issue here. I've opened a PR to fix this : https://github.com/thelastpickle/cassandra-reaper/pull/856
Once it's merged, I'll cut out a new bug fix release.
Cheers,
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
On Wed, Feb 26, 2020 at 1:35 AM Dan Sarisky <dsar...@gmail.com> wrote:
When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do expect it will stabilize eventually.Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious problems ...
--
You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com.
Sarisky, Dan
Feb 26, 2020, 7:45:58 AM2/26/20
to Alexander Dejanovski, TLP Apache Cassandra Reaper users
Ah. Wonderful. Thanks for being on top of it so quickly.
On 2/26/2020 3:07 AM, Alexander Dejanovski wrote:
> Hi Dan,
>
> yes, the problem has been reported and we have a data modelling issue here. I've opened a PR to fix this : https://github.com/thelastpickle/cassandra-reaper/pull/856
> Once it's merged, I'll cut out a new bug fix release.
>
> Cheers,
>
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>
On 2/26/2020 3:07 AM, Alexander Dejanovski wrote:
> Hi Dan,
>
> yes, the problem has been reported and we have a data modelling issue here. I've opened a PR to fix this : https://github.com/thelastpickle/cassandra-reaper/pull/856
> Once it's merged, I'll cut out a new bug fix release.
>
> Cheers,
>
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
>
>
> On Wed, Feb 26, 2020 at 1:35 AM Dan Sarisky <dsar...@gmail.com <mailto:dsar...@gmail.com>> wrote:
>
> When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do expect it
> will stabilize eventually.
> Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious problems ...
>
> --
> You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> <mailto:tlp-apache-cassandra-r...@googlegroups.com>.
>
> On Wed, Feb 26, 2020 at 1:35 AM Dan Sarisky <dsar...@gmail.com <mailto:dsar...@gmail.com>> wrote:
>
> When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do expect it
> will stabilize eventually.
> Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious problems ...
>
> --
> You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com
> <https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com?utm_medium=email&utm_source=footer>.
>
Alexander Dejanovski
Mar 2, 2020, 4:48:47 AM3/2/20
to Sarisky, Dan, TLP Apache Cassandra Reaper users
Hi Dan,
2.0.3 was just released and fixes the issue.
Let us know how it goes for you.
Cheers,
-----------------
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
Alexander Dejanovski
France
@alexanderdeja
Consultant
Apache Cassandra Consulting
To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/6cf00335-2449-fdd9-4f5a-ce83cfe66dda%40gmail.com.
Sarisky, Dan
Mar 3, 2020, 10:39:00 AM3/3/20
to Alexander Dejanovski, TLP Apache Cassandra Reaper users
2.0.3 is working great thus far.
Repairs are succeeding across various clusters.
Disk space usage is in the <20MB range instead of on track to be in the 15GB range per node.
Thanks for turning this around so quick!
On 3/2/2020 4:48 AM, Alexander Dejanovski wrote:
> Hi Dan,
>
> > <mailto:tlp-apache-cassandra-r...@googlegroups.com <mailto:tlp-apache-cassandra-reaper-users%2Bunsu...@googlegroups.com>>.
> To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/6cf00335-2449-fdd9-4f5a-ce83cfe66dda%40gmail.com.
>
Repairs are succeeding across various clusters.
Disk space usage is in the <20MB range instead of on track to be in the 15GB range per node.
Thanks for turning this around so quick!
On 3/2/2020 4:48 AM, Alexander Dejanovski wrote:
> Hi Dan,
>
> 2.0.3 was just released and fixes the issue.
>
> Let us know how it goes for you.
>
> Cheers,
>
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
>
> Let us know how it goes for you.
>
> Cheers,
>
> -----------------
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com <http://www.thelastpickle.com/>
>
>
> On Wed, Feb 26, 2020 at 1:45 PM Sarisky, Dan <dsar...@gmail.com <mailto:dsar...@gmail.com>> wrote:
>
> Ah. Wonderful. Thanks for being on top of it so quickly.
>
> On 2/26/2020 3:07 AM, Alexander Dejanovski wrote:
> > Hi Dan,
> >
> > yes, the problem has been reported and we have a data modelling issue here. I've opened a PR to fix this : https://github.com/thelastpickle/cassandra-reaper/pull/856
> > Once it's merged, I'll cut out a new bug fix release.
> >
> > Cheers,
> >
> > -----------------
> > Alexander Dejanovski
> > France
> > @alexanderdeja
> >
> > Consultant
> > Apache Cassandra Consulting
> > http://www.thelastpickle.com <http://www.thelastpickle.com/>
> >
> >
> > On Wed, Feb 26, 2020 at 1:35 AM Dan Sarisky <dsar...@gmail.com <mailto:dsar...@gmail.com> <mailto:dsar...@gmail.com <mailto:dsar...@gmail.com>>> wrote:
> >
> > When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do
> expect it
> > will stabilize eventually.
> > Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious
> problems ...
> >
> > --
> > You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> <mailto:tlp-apache-cassandra-reaper-users%2Bunsu...@googlegroups.com>
>
>
> On Wed, Feb 26, 2020 at 1:45 PM Sarisky, Dan <dsar...@gmail.com <mailto:dsar...@gmail.com>> wrote:
>
> Ah. Wonderful. Thanks for being on top of it so quickly.
>
> On 2/26/2020 3:07 AM, Alexander Dejanovski wrote:
> > Hi Dan,
> >
> > yes, the problem has been reported and we have a data modelling issue here. I've opened a PR to fix this : https://github.com/thelastpickle/cassandra-reaper/pull/856
> > Once it's merged, I'll cut out a new bug fix release.
> >
> > Cheers,
> >
> > -----------------
> > Alexander Dejanovski
> > France
> > @alexanderdeja
> >
> > Consultant
> > Apache Cassandra Consulting
> > http://www.thelastpickle.com <http://www.thelastpickle.com/>
> >
> >
> > On Wed, Feb 26, 2020 at 1:35 AM Dan Sarisky <dsar...@gmail.com <mailto:dsar...@gmail.com> <mailto:dsar...@gmail.com <mailto:dsar...@gmail.com>>> wrote:
> >
> > When deploying sidecar mode, I see the table node_metrics_v2 growing to >10GB on each node in my cluster. I see it uses a default ttl of 864000 (10 days), and I do
> expect it
> > will stabilize eventually.
> > Is there any issue with setting that default ttl quite a bit lower to minimize the storage impact of reaper? I'm testing with that and I don't see any obvious
> problems ...
> >
> > --
> > You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> > To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> > <mailto:tlp-apache-cassandra-r...@googlegroups.com <mailto:tlp-apache-cassandra-reaper-users%2Bunsu...@googlegroups.com>>.
> > To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com
> > <https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >
>
> --
> You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> <mailto:tlp-apache-cassandra-reaper-users%2Bunsu...@googlegroups.com>.
> > <https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/916f1799-baf5-45c7-be52-7e19279dc3b6%40googlegroups.com?utm_medium=email&utm_source=footer>.
> >
>
> --
> You received this message because you are subscribed to the Google Groups "TLP Apache Cassandra Reaper users" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to tlp-apache-cassandra-r...@googlegroups.com
> To view this discussion on the web visit https://groups.google.com/d/msgid/tlp-apache-cassandra-reaper-users/6cf00335-2449-fdd9-4f5a-ce83cfe66dda%40gmail.com.
>
Reply all
Reply to author
Forward
0 new messages