get the good rows and feed an alerting system

[tobias...@justwatch.com]

Hi,

i want to create an alert event (in prometheus), based on how many good and bad rows we import. It would be nice to have this data in one place, so either redshift or elastic. So my idea was, that there could be an option, that I can also import an event into elastic that holds informations about the number of good rows, that were implemented (if you will an accumulated event for that run, with more information). Or we somehow use a context for this with etl_tstamp,number_bad_rows and than we could write queries to measure the percentage of bad rows to good rows. Maybe I described it a little bit to specific for our use case, but I think it could be worth it for more people.


In general, it can be hard to say if the bad rows are high or low, if I don't know the number of good rows, for this import.

[Alex Dean]

Hi Tobias,

It's a cool idea! We are working on functionality for Snowplow which will let you add arbitrary data modeling jobs into your EMR run after the enrichment process has completed. Potentially you could add a job to perform that simple count and then upload it into Elasticsearch for comparison against the bad rows count.

Cheers,

Alex

--
You received this message because you are subscribed to the Google Groups "Snowplow" group.
To unsubscribe from this group and stop receiving emails from it, send an email to snowplow-use...@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--

Co-founder
Snowplow Analytics
The Roma Building, 32-38 Scrutton Street, London EC2A 4RQ, United Kingdom
+44 (0)203 589 6116
+44 7881 622 925
@alexcrdean

[tobias...@justwatch.com]

Hi Alex,

that's great, looking forward to that.