Funqy Knative events - filter based on source attribute

[swidersk...@gmail.com]

Hi,

I am working with Google Cloud Run and using Funqy with Knative events and ran into a limitation (mainly due to simplified use of cloud events from Cloud Run in my opinion).

When using cloud run you can configure different triggers (but you can’t create custom ones - like custom in terms you define what is the value of the type attribute) and in my case it’s the PubSub trigger. The problem here is that Cloud Run will use the same value for type attribute (google.cloud.pubsub.topic.v1.messagePublished) regardless of the topic it comes from. The topic itself is given as source attribute of the cloud event. On the other side Funqy with cloud events uses only type attribute of the cloud event as the trigger correlation so to say. That means it’s going to trigger all functions as they will use the same type defined via @CloudEventMapping.

See this docs page describing how the cloud event are used in Cloud Run

https://cloud.google.com/eventarc/docs/cloudevents#pubsub

Proposal

I was thinking to provide an optional attribute on the @CloudEventMapping annotation that allows to define the filter on source attribute. 

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filter=“//pubsub.googleapis.com/projects/my-project/topics/orders”)

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filter=“//pubsub.googleapis.com/projects/my-project/topics/customers")

In general it could be on other attributes as well so alternatively it could look like this

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filterAttribute=“ce-source” filterValue=“//pubsub.googleapis.com/projects/my-project/topics/orders”)

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filterAttribute=“custom-header” filterValue=“custom value”)

With this we could easily use Funqy with cloud events on Cloud Run and I believe this could be useful even outside of the Cloud Run context to allow to filter based on at least the source of the cloud event.

Thoughts?

Thanks

Maciej

[William Burke]

On Wed, May 19, 2021 at 11:52 AM <swidersk...@gmail.com> wrote:

Hi,

I am working with Google Cloud Run and using Funqy with Knative events and ran into a limitation (mainly due to simplified use of cloud events from Cloud Run in my opinion).

When using cloud run you can configure different triggers (but you can’t create custom ones - like custom in terms you define what is the value of the type attribute) and in my case it’s the PubSub trigger. The problem here is that Cloud Run will use the same value for type attribute (google.cloud.pubsub.topic.v1.messagePublished) regardless of the topic it comes from. The topic itself is given as source attribute of the cloud event. On the other side Funqy with cloud events uses only type attribute of the cloud event as the trigger correlation so to say. That means it’s going to trigger all functions as they will use the same type defined via @CloudEventMapping.

See this docs page describing how the cloud event are used in Cloud Run

https://cloud.google.com/eventarc/docs/cloudevents#pubsub

Proposal

I was thinking to provide an optional attribute on the @CloudEventMapping annotation that allows to define the filter on source attribute. 

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filter=“//pubsub.googleapis.com/projects/my-project/topics/orders”)

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filter=“//pubsub.googleapis.com/projects/my-project/topics/customers")

Sounds good.  I just don't understand "filter" syntax above.  I understand the "filterAttribute" syntax.

--

Bill Burke

Red Hat

[swidersk...@gmail.com]

Filter is just shorthand version that assumes the filtering is done only on ce-source attribute

So it’s either we want to allow to filter by any header/attribute of the cloud event or just by source.

Maciej

[Maciej Swiderski]

One another thought would be to support multiple filters as well something along the lines

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filters={

  @CloudEventFilter(attribute="ce-source", attributeValue="//pubsub.googleapis.com/projects/my-project/topics/customers"),

  @CloudEventFilter(attribute="custom-header", attributeValue="custom value")  

})

Would this be something we would like to have or too much hassle?

Let me know if I should go ahead and create an issue and if no one else is interesting I can give it a try with implementation.

Maciej

[William Burke]

Sounds good to me. 

[Matej Vasek]

btw there will be DSL for filtering events
https://github.com/cloudevents/spec/blob/master/expression-language.md

[Matej Vasek]

I know that for HTTP Funq we have function name based routing:
```
/fn1 -> fn1()
/fn2 -> fn2()
```

Not sure if that's the case for Knative Funq, but  if it was then you can use knative `triggers` for filtering.
https://knative.dev/docs/eventing/flows/sequence/sequence-with-broker-trigger/#create-the-trigger-targeting-the-sequence

Just instead of `ref` use `uri`.
Trigger can filter on ce-type and ce-source, in addition in the future it will allow to use aforementioned DSL.

[swidersk...@gmail.com]

Thanks Matej for the link to the SQL like filtering. It does remind me the JMS selector kind of thing which I thought is too much but I’d like to hear other opinions.

When it comes to filtering based on the source and type you mentioned - I assume you refer to the trigger definition of Knative, right? If so that’s the issue - Cloud Run does not expose “such low level” details. You can create trigger based on the eventrac thingy and that’s pretty much it. Or I simply don’t know how to do it and can’t find docs describing that. 

Maciej

-- 
You received this message because you are subscribed to the Google Groups "Quarkus Development mailing list" group.
To unsubscribe from this group and stop receiving emails from it, send an email to quarkus-dev...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/e8d81b30-4124-405d-8253-8bb7b2daa510n%40googlegroups.com.

[Matej Vasek]

Yes, I meant knative trigger you usually create using yaml similar to the one in link.
Also the expression language already have some implementation in Java https://github.com/cloudevents/sdk-java/blob/master/sql/src/main/java/io/cloudevents/sql/Expression.java#L51

[swidersk...@gmail.com]

Bill, Matej, so how do we want to tackle it? I see three options

1. Stick to simple @CloudEventFilter annotation(s) that allow to have only attribute name and attribute value to be compared.

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filters={

  @CloudEventFilter(attribute="source", attributeValue="//pubsub.googleapis.com/projects/my-project/topics/customers"),

  @CloudEventFilter(attribute="custom-header", attributeValue="custom value")  

})

2. Prepare for expressions and have @CloudEventFilter with single field “expression” and for now it will only support equals based comparison and expand over time when the spec is ready

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filters={

  @CloudEventFilter(expression="subject = 'my topic'")  

})

Why it is a list of filters? Mainly as it can be easier to express complex filter expressions as they would be evaluated in AND manner so to say.

3. Combined approach where user can specify either attribute name and value or expression

@CloudEventMapping(trigger = "google.cloud.pubsub.topic.v1.messagePublished”, filters={

  @CloudEventFilter(attribute="source", attributeValue="//pubsub.googleapis.com/projects/my-project/topics/customers"),

  @CloudEventFilter(expression="source = 'my topic'")  

})

I am not sure what is the interest for Funqy to implement the cloud event filter expression and potentially use the java sdk for it. As I believe this would have quite substantial impact on the approach.

Maciej

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/225969c2-576e-4a88-8275-ec0900ad03ddn%40googlegroups.com.

[William Burke]

@CloudEventMapping(trigger="xxx" attributes = {

    @EventAttribute(name="source", value="//pubsub"),

    @EventAttribute(name="custom-header", value="xxx")

    }

)

@CloudEventMapping(trigger="xxx", expression="source = 'my topic' && header='foo'")

Just trying to reduce what people have to type.  I don't know, you guys know the spec better.

[Matej Vasek]

I think that solution 1. is good for now. Latter we can extend it to  accept both as proposed in solution 3.

[Maciej Swiderski]

So I guess our selected option be like this

@CloudEventMapping(trigger="xxx" attributes = {

    @EventAttribute(name="source", value="//pubsub"),

    @EventAttribute(name="custom-header", value="xxx")

    }

)

Correct?

If so I will create issue describing what we discussed here and give it a go on impl side. 

Maciej

Wiadomość napisana przez Matej Vasek <mva...@redhat.com> w dniu 21.05.2021, o godz. 19:21:

I think that solution 1. is good for now. Latter we can extend it to  accept both as proposed in solution 3.

To view this discussion on the web visit https://groups.google.com/d/msgid/quarkus-dev/073db9ff-6447-4c8f-9a80-b480c7421453n%40googlegroups.com.

[swidersk...@gmail.com]

So here is a PR for this feature

https://github.com/quarkusio/quarkus/pull/17655

Please have a look when you have some time

Thanks

Maciej