TTL and Delete series feedbacks

[A. Hébert]

Hi,

  Since a year we put in production the Warp10 TTL and since a couple of month, a bot deletes our old inactive series. Since then, our production is very healthy and working well! Thanks for all those nice feature. I just have a mind a couple of small improvement that could be great for Warp10:

   - Could we improve the DELETE method on the exec endpoint to take meta, quiesce/active as parameters? (I didn't find the time yet to contribute it)

   - At our scale, we do have issues with "deletall" calls as it can produces twice more traffic on a query more than the usual ingest/fetch ones. To reduce the impact, a new limit on the amount of time series than can be deleted in the same way that LIMIT or MAXGTS works, we have tested it in production. A native support on Warp10 of such limits could serves other Warp10 user. Let me know your plan on it, I can contribute as if my first implementation if you are interested.

Best

[Mathias Herberts]

Limiting the number of series which are deleted is not something very practical as you will have a hard time actually selecting a maximum number.

There is a feature in the Store component to limit the number of messages which are processed, this is commonly used to throttle down the delete operations on dedicated Store components so as to preserve the health of RegionServers. 

The store throttling file is configured via the following configuration keys:

//

// Throttling file path

//

#store.throttling.file =

//

// How often to reread the throttling file (in ms, defaults to 60000).

//

#store.throttling.period = 

//

// How much to wait when the consumption was throttled, in ns (nanoseconds), defaults to 10 ms (milliseconds)

//

#store.throttling.delay =

The throttling file contains a single line with a DOUBLE between 0.0 and 1.0 which represents the percentage of requests whose processing should be delayed of 'store.throttling.delay'.

[A. Hébert]

If I understand correctly it's a manual operation that you have to do when deletes operation are occurring? No automatic delete throttling is done

[Mathias Herberts]

It is a one time configuration of the Store instance handling the deletes.