Notifications, web hooks and reliability

[Niklas Therning]

Hi,

I'm trying to find some docs on how notifications work in Kill Bill. Is there a list of all defined notifications somewhere? I've found BeatrixListener which seems to be where internal events are converted into external events. The ExtBusEventType enum seems to list the possible event types?

We're looking into using web hooks for getting notifications to our webapp from the Kill Bill server. One concern we have is what happens if our webapp should be down. IIUC PushNotificationListener is the class which forwards ExtBusEvents to the registered web hooks. Am I right to assume that notifications that happens while the web hook cannot be reached are just discarded? The org.killbill.persistent.bus.${instancename}.max.failure.retry config property suggests that event delivery will be retried but PushNotificationListener doesn't throw any exception when a web hook fails.

Would we be better off writing a notification plugin that handles the reliable delivery of notifications (like storing to our own db which the webapp can then read or using a persistent Hazelcast queue or similar)? Has anyone else done something like this already?

Another thing about the java client APIs. We're using the  KillBillClient class (0.2.0) but it doesn't seem to have any methods for listing and deleting notification callbacks? Also the endpoint for the DELETE endpoint in TenantResource looks incorrect:

    @Timed

    @DELETE

    @Path("/REGISTER_NOTIFICATION_CALLBACK")

    @ApiOperation(value = "Delete a push notification")

    @ApiResponses(value = {@ApiResponse(code = 400, message = "Invalid tenantId supplied")})

    public Response deletePushNotificationCallbacks(@PathParam("tenantId") final String tenantId,

I think the @Path should rather be

    @Path("/" + REGISTER_NOTIFICATION_CALLBACK)

We're using 0.13.4 of killbill-jaxrs but this doesn't seem to have been fixed in HEAD.

Thanks!

Niklas

[Pierre-Alexandre Meyer]

Hello Niklas,

On Mon, Mar 2, 2015 at 3:56 AM, Niklas Therning <nik...@therning.org> wrote:

I'm trying to find some docs on how notifications work in Kill Bill. Is there a list of all defined notifications somewhere? I've found BeatrixListener which seems to be where internal events are converted into external events. The ExtBusEventType enum seems to list the possible event types?

Correct. The full list can be seen here:

  https://github.com/killbill/killbill-plugin-api/blob/master/notification/src/main/java/org/killbill/billing/notification/plugin/api/ExtBusEventType.java 

We're looking into using web hooks for getting notifications to our webapp from the Kill Bill server. One concern we have is what happens if our webapp should be down. IIUC PushNotificationListener is the class which forwards ExtBusEvents to the registered web hooks. Am I right to assume that notifications that happens while the web hook cannot be reached are just discarded? The org.killbill.persistent.bus.${instancename}.max.failure.retry config property suggests that event delivery will be retried but PushNotificationListener doesn't throw any exception when a web hook fails.

Ah, you're right.  I'm not sure we should rely on the retry mechanism of the external bus though, because the retry would pretty much happen right away. If your webapp is down, it may take some time until it's back online.

What we should probably do is have the push notification listener use its own notification queue, and schedule retries with exponential backoff. I've created https://github.com/killbill/killbill/issues/285 for tracking.

Would we be better off writing a notification plugin that handles the reliable delivery of notifications (like storing to our own db which the webapp can then read or using a persistent Hazelcast queue or similar)? Has anyone else done something like this already?

That's another way, but I think reliable delivery would be a nice feature in the core. If you are interested in contributing the feature, we can finalize the design in the ticket and help you with the implementation. Let us know.

Another thing about the java client APIs. We're using the  KillBillClient class (0.2.0) but it doesn't seem to have any methods for listing and deleting notification callbacks?

Indeed, only the creation seems to be implemented (https://github.com/killbill/killbill-client-java/blob/master/src/main/java/org/killbill/billing/client/KillBillClient.java#L1368). Happy to merge a patch for GET/DELETE (associated tests can be found here: https://github.com/killbill/killbill/blob/master/profiles/killbill/src/test/java/org/killbill/billing/jaxrs/TestPushNotification.java)!

 

Also the endpoint for the DELETE endpoint in TenantResource looks incorrect:

    @Timed

    @DELETE

    @Path("/REGISTER_NOTIFICATION_CALLBACK")

    @ApiOperation(value = "Delete a push notification")

    @ApiResponses(value = {@ApiResponse(code = 400, message = "Invalid tenantId supplied")})

    public Response deletePushNotificationCallbacks(@PathParam("tenantId") final String tenantId,

I think the @Path should rather be

    @Path("/" + REGISTER_NOTIFICATION_CALLBACK)

We're using 0.13.4 of killbill-jaxrs but this doesn't seem to have been fixed in HEAD.

That's an unfortunate old typo :( It is tracked by https://github.com/killbill/killbill/issues/238. While the fix is trivial, it requires coordination to release it as client libraries have to be updated (and we also know companies with their own client implementations). We have quite a few things on our hands for 0.13.5, but we can probably still tackle it before the 0.14.0 release.

Thanks for the great feedback!

--

Pierre

[Niklas Therning]

Thanks! That last one already seems to have been fixed. Awesome! I'll let you know if I get some time to look into those new features (reliable web hook delivery and GET/DELETE for notification callbacks) and we can discuss how it should be done.

[Pierre-Alexandre Meyer]

On Tue, Mar 3, 2015 at 4:59 AM, Niklas Therning <nik...@therning.org> wrote:

Thanks! That last one already seems to have been fixed. Awesome! I'll let you know if I get some time to look into those new features (reliable web hook delivery and GET/DELETE for notification callbacks) and we can discuss how it should be done.

FYI, Stéphane also implemented yesterday GET/DELETE for notification callbacks in the Java client.

--

Pierre

[Pierre-Alexandre Meyer]

On Mon, Mar 2, 2015 at 7:04 AM, Pierre-Alexandre Meyer <pie...@kill-bill.org> wrote:

We're looking into using web hooks for getting notifications to our webapp from the Kill Bill server. One concern we have is what happens if our webapp should be down. IIUC PushNotificationListener is the class which forwards ExtBusEvents to the registered web hooks. Am I right to assume that notifications that happens while the web hook cannot be reached are just discarded? The org.killbill.persistent.bus.${instancename}.max.failure.retry config property suggests that event delivery will be retried but PushNotificationListener doesn't throw any exception when a web hook fails.

Ah, you're right.  I'm not sure we should rely on the retry mechanism of the external bus though, because the retry would pretty much happen right away. If your webapp is down, it may take some time until it's back online.

What we should probably do is have the push notification listener use its own notification queue, and schedule retries with exponential backoff. I've created https://github.com/killbill/killbill/issues/285 for tracking.

FYI, in case you still need the functionality, it's currently being actively worked on in this PR: https://github.com/killbill/killbill/pull/589.

--

Pierre