Relevance custom TrackingPolicy implementation

[dennis.nijssen]

My client has a custom consent / privacy settings cookie, and we would like for relevance to listen to this cookie in order to know whether the visitor wants to be tracked or not.

When looking at the tracking cookie consent documentation [1], I didnt find a good explanation on how to use a custom cookie for this. Although after looking in the code I noted we could probably create our custom implementation of the TrackingPolicy interface. But I can't seem to override this via custom spring configuration, as a hst-assembly override.

package com.example.site.targeting;

import javax.servlet.http.HttpServletRequest;
import com.onehippo.cms7.targeting.TrackingPolicy;

public class CustomTrackingPolicy implements TrackingPolicy {

    // ....

    @Override
    public boolean trackingAllowed(HttpServletRequest request) {
        // ....
    }
}

This is the targeting.xml override in \hst-assembly\overrides\targeting.xml

<?xml version="1.0" encoding="UTF-8"?>
<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="com.onehippo.cms7.targeting.DefaultTrackingPolicy" class="com.exaple.site.targeting.CustomTrackingPolicy"/>

</beans>

But the bean is not getting loaded, and still the DefaultTrackingPolicy is being called.

Any ideas on why I can't seem to override the tracking policy bean?

[1] https://www.onehippo.org/library/enterprise/enterprise-features/targeting/gdpr/implement-tracking-cookie-consent.html

Kind regards,

Dennis

[dennis.nijssen]

Nevermind! I've been looking 1 hour into this, and 30 minutes after I posted this I found the solution in the addon\module.xml:

<config-location>classpath*:META-INF/hst-assembly/overrides/targeting-addon-*.xml</config-location>
<config-location>classpath*:META-INF/hst-assembly/overrides/addon/com/onehippo/cms7/targeting/*.xml</config-location>

Seems I have to name my Spring override xml like "targeting-addon-*.xml".

Maybe we can add such an example to the relevance documentaton.

Kind regards,

Dennis

[Ard Schrijvers]

Hey Dennis,

On Mon, Dec 11, 2017 at 3:29 PM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> Nevermind! I've been looking 1 hour into this, and 30 minutes after I posted
> this I found the solution in the addon\module.xml:
>
> <config-location>classpath*:META-INF/hst-assembly/overrides/targeting-addon-*.xml</config-location>
> <config-location>classpath*:META-INF/hst-assembly/overrides/addon/com/onehippo/cms7/targeting/*.xml</config-location>
>
>
> Seems I have to name my Spring override xml like "targeting-addon-*.xml".
> Maybe we can add such an example to the relevance documentaton.

Great you found it, admittedly it is a bit confusing how it is set up
in targeting since when that project started we did not yet have
'spring addon' support. Hence we have two different override options,
although it might just be all doable just by placing your overrides in
hst-assembly/overrides/addon/com/onehippo/cms7/targeting

Regards Ard

> --
> Hippo Community Group: The place for all discussions and announcements about
> Hippo CMS (and HST, repository etc. etc.)
>
> To post to this group, send email to hippo-c...@googlegroups.com
> RSS:
> https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
> ---
> You received this message because you are subscribed to the Google Groups
> "Hippo Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to hippo-communi...@googlegroups.com.
> Visit this group at https://groups.google.com/group/hippo-community.
> For more options, visit https://groups.google.com/d/optout.



--
Hippo Netherlands, Oosteinde 11, 1017 WT Amsterdam, Netherlands
Hippo USA, Inc. 71 Summer Street, 2nd Floor Boston, MA 02110, United
states of America.

US +1 877 414 4776 (toll free)
Europe +31(0)20 522 4466
www.onehippo.com

[Ard Schrijvers]

On Tue, Dec 12, 2017 at 10:36 AM, Ard Schrijvers
<ard.sch...@bloomreach.com> wrote:
> Hey Dennis,
>
> On Mon, Dec 11, 2017 at 3:29 PM, 'dennis.nijssen' via Hippo Community
> <hippo-c...@googlegroups.com> wrote:
>> Nevermind! I've been looking 1 hour into this, and 30 minutes after I posted
>> this I found the solution in the addon\module.xml:
>>
>> <config-location>classpath*:META-INF/hst-assembly/overrides/targeting-addon-*.xml</config-location>
>> <config-location>classpath*:META-INF/hst-assembly/overrides/addon/com/onehippo/cms7/targeting/*.xml</config-location>
>>
>>
>> Seems I have to name my Spring override xml like "targeting-addon-*.xml".
>> Maybe we can add such an example to the relevance documentaton.
>
> Great you found it, admittedly it is a bit confusing how it is set up
> in targeting since when that project started we did not yet have
> 'spring addon' support. Hence we have two different override options,
> although it might just be all doable just by placing your overrides in
> hst-assembly/overrides/addon/com/onehippo/cms7/targeting

ps wrt documentation...we do not give hard guarantees on overriding
targeting spring configurations during upgrades. If we document it we
also make it more formally cast in concrete and supported 100% wrt
backwards compatibility....which I am hesitant to do so because it
might impose too hard constraints on future development. This is a
hard choice and delicate to balance. Hope you agree

Regards Ard

[dennis.nijssen]

I agree and disagree :) I mainly disagree because developers (like me) can find the override feature. And when its documented, it will also get in the release notes when a change has been made. Which will help me ofcourse!

And I do believe implementing a custom cookie consent is not the weirdest thing to do, especially with GDPR coming next year. 

One more thing I noticed was when a user decides not to be tracked, there is still a "_visitor" cookie placed on the response. Is that intended? 

Kind regards,

Dennis

[Ard Schrijvers]

On Tue, Dec 12, 2017 at 11:33 AM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> I agree and disagree :) I mainly disagree because developers (like me) can
> find the override feature. And when its documented, it will also get in the
> release notes when a change has been made. Which will help me ofcourse!

I agree and disagree :-)

The biggest problem is that we ourselves typically don't really
realize what changes can probably result in an end project failure. If
we for example do some restructuring of spring bean wiring, then end
projects that did override spring beans might break. Spring bean
shuffling can now be done as part as some refactoring or bug fix. We
don't explicitly document this. Also as opposed to javadoc, there is
no way we can easily communicate deprecation of some api....and we
don't formally support the spring wiring as api although of course we
*try* to avoid unnecessary end project pain. Also making spring wiring
api is very difficult....only take the method invoking factory bean as
example : It is very difficult to support this as api if devs can
pretty much use reflection to hook in pretty much everything.

I hope you see my point above. Now, given that, *if* we formally
document how you can override all kind of targeting internals, we
pretty much might end up in everything being cast in concrete because
we can't touch anything any more...or think really hard about impact
and write complex upgrade documentation.

> And I do believe implementing a custom cookie consent is not the weirdest
> thing to do, especially with GDPR coming next year.

which class do you override for custom cookie consent? I thought
typically custom cookie consent is done by client side javascript code
and the only thing you modify in targeting is the name of the consent
cookie. If there needs to be more flexibility, we can create a factory
or something and then document how to plugin a custom cookie consent
class (and that then becomes 'api').

>
>
> One more thing I noticed was when a user decides not to be tracked, there is
> still a "_visitor" cookie placed on the response. Is that intended?

That is odd. I agree there should not be created a visitor cookie
because that is misleading and might seem to imply that the user is
still tracked. I think from glancing at the code that the _visitor
cookie is used to store the consent cookie value (and thus that the
visitor doesn't want to be tracked). That doesn't make sense to me.
I'll create an issue for it that we stop doing this. I'll do some
extra checks to validate this but I don't think we need it and I
consider the current logic undesirable. Thanks for noticing!

[Ard Schrijvers]

Hey,

wrt to the _visitor cookie: Turns out that apart from writing the
cookie, nothing else was done with it. It is an identifier to a
targeting data blob and request log we never store. It did however
result in a needless lookup for whether we had any data for the
visitor, so not storing the cookie actually results in a performance
improvement. I've fixed it in [1] which is not public. Most likely we
will backport it (though it did not store anything)

Thanks again for mentioning

Regards Ard

[1] https://issues.onehippo.com/browse/BT-2022

[dennis.nijssen]

I have been implementing the TrackingPolicy interface, and in order to have relevance use my custom implementation, I have now created a "targeting-addon-override.xml" in hst-assembly\overrides, which contains the following Spring configuration.

\hst-assembly\overrides\targeting-addon-override.xml

<?xml version="1.0" encoding="UTF-8"?>

<beans xmlns="http://www.springframework.org/schema/beans" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"

       xsi:schemaLocation="http://www.springframework.org/schema/beans http://www.springframework.org/schema/beans/spring-beans.xsd">

  <bean id="com.onehippo.cms7.targeting.DefaultTrackingPolicy" class="com.example.site.targeting.CustomTrackingPolicy"/>

</beans>

My use case was to have relevance' its tracking policy listen to our own cookie, and not the cookieconsent cookie set by the relevance. 

But to do this it seemed the best to implement our own TrackingPolicy, as it seemed like a nice extension point.

However I can agree with your points as well, cause one thing in my example could already be the bean identifier... Why in godsname is it the fully qualified class name. ;)

Kind regards,

Dennis

[dennis.nijssen]

So the _visitor cookie will not be set anymore, when tracking is not allowed?

In what version is this fixed? 

[Ard Schrijvers]

On Tue, Dec 12, 2017 at 3:55 PM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> I have been implementing the TrackingPolicy interface, and in order to have
> relevance use my custom implementation, I have now created a
> "targeting-addon-override.xml" in hst-assembly\overrides, which contains the
> following Spring configuration.
>
> \hst-assembly\overrides\targeting-addon-override.xml
> <?xml version="1.0" encoding="UTF-8"?>
> <beans xmlns="http://www.springframework.org/schema/beans"
> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> xsi:schemaLocation="http://www.springframework.org/schema/beans
> http://www.springframework.org/schema/beans/spring-beans.xsd">
>
> <bean id="com.onehippo.cms7.targeting.DefaultTrackingPolicy"
> class="com.example.site.targeting.CustomTrackingPolicy"/>
>
> </beans>
>
>
> My use case was to have relevance' its tracking policy listen to our own
> cookie, and not the cookieconsent cookie set by the relevance.
> But to do this it seemed the best to implement our own TrackingPolicy, as it
> seemed like a nice extension point.
>
> However I can agree with your points as well, cause one thing in my example
> could already be the bean identifier... Why in godsname is it the fully
> qualified class name. ;)

I wholeheartedly agree!! I can't inform the person who did this any
more since he doesn't work he any more :-). Of course, in 99% of the
cases, the point is to use the interface
com.onehippo.cms7.targeting.TrackingPolicy as the bean identifier.

And now we in a shitty area: If we rename DefaultTrackingPolicy to a
more sensible name (because 'default' doesn't tell me anything), then
we might easily rename the bean id also.....or if we now want to fix
the obvious wrong bean id, it means that we'd break your project. This
is why it is such a shitty part to 'document as api' .... it is very
hard for us to realize what we might possibly break in downstream
projects if internal spring beans are overridden. Although we provide
override capabilities, making it some sort of public api is a way more
difficult task.

So, now we are between the rock and the hard place: The
id="com.onehippo.cms7.targeting.DefaultTrackingPolicy" is wrong....but
if I 'fix' it, I now know it will break your code.

Perhaps we should for every release, even minor releases, generate a
Spring bean definition diff. Would that help perhaps? For example that
we publish Spring bean definition diff file *but* also keep our
current strategy of not claiming the spring wiring is api?

Apart from this all, did you know you can configure which cookie name
to use for consent? You can configure it on the
[targeting:configuration] node (/targeting:targeting node):

- targeting:consentCookieEnabled (boolean)
- targeting:consentCookieOptIn (boolean)
- targeting:consentCookieValue (string)
- targeting:consentCookieName (string)

Also see https://www.onehippo.org/library/enterprise/enterprise-features/targeting/gdpr/implement-tracking-cookie-consent.html

[Ard Schrijvers]

On Tue, Dec 12, 2017 at 4:00 PM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> So the _visitor cookie will not be set anymore, when tracking is not
> allowed?
> In what version is this fixed?

I now did it in 12.2.0, will be backported to the next 12.1.x, 12.0.x
and 11.2.x versions. Do you need it in 10 as well?

Regards Ard

[fpieper.n...@gmail.com]

Hi Ard,

we too would love to know when this is available, as our client is very strict on the GDPR stuff and having a _visitor cookie while not allowing tracking is a no-go.

Thanks,

Felix

[Ard Schrijvers]

Hey Felix,

On Wed, Dec 13, 2017 at 1:39 PM, <fpieper.n...@gmail.com> wrote:
> Hi Ard,
>
> we too would love to know when this is available, as our client is very
> strict on the GDPR stuff and having a _visitor cookie while not allowing
> tracking is a no-go.

To be sure, the _visitor cookie is written to the http response but
nothing is actually stored. I understand it might raise eyebrows by
end users, but for what it is worth, we do not track any data. Not
sure if just writing the _visitor would already violate GDPR. Writing
the consent cookie for example is allowed so it is not the case that
according GDPR no cookie at all is allowed to be set.

If it turns out to be mandatory, we can already set the targeting tags
containing the fix. Also we can also do complete release cycle in
January.

HTH,

Regards Ard

[dennis.nijssen]

Yea I know it will break our project once we upgrade, but I think for this one "override" it is something we should take for granted. Ofcoure every upgrade we need to check this point.

So I won't mind if the bean id gets renamed, as it seems such a simple override at this point. Ofcourse when the whole TargetingPolicy interface implementation in the TargetingService gets changed I might end up in more trouble. 

Also I do know about the targetin:consentCookie* options, but our cookie is a little more sophisticated. It is basically an url encoded json object like {"analytics":false,"personalisation":true"} (but then url encoded ofc...) 

So the targeting:consentCookieValue wont work with this. That is why I have added my project specific TrackingPolicy for now. 

[dennis.nijssen]

For our use we most likely need it in 12.1.x, as we started yesterday with our upgrade from 11.2.x to 12.1.x.

So no, I don't need this in 10 ;) 

And about the release date, it would ofcourse be best if this can be released as soon as possible, Because otherwise my client has to explain why we store a _visitor cookie, without its being used. ;) 

We could of course ask the visitors to trust us on our blue eyes ;) But I believe its better to not have the cookie set at all. 

[Ard Schrijvers]

On Wed, Dec 13, 2017 at 3:54 PM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> Yea I know it will break our project once we upgrade, but I think for this
> one "override" it is something we should take for granted. Ofcoure every
> upgrade we need to check this point.
> So I won't mind if the bean id gets renamed, as it seems such a simple
> override at this point. Ofcourse when the whole TargetingPolicy interface
> implementation in the TargetingService gets changed I might end up in more
> trouble.

When we'd modify the TargetingPolicy we'd create upgrade documentation
for it. The interface will in minor version never have BC-incompatible
changes. Also with java api when possible we work with deprecation
annotations and javadoc of course.

I don't plan to 'fix' the bean id name: Let's leave it as is.

>
> Also I do know about the targetin:consentCookie* options, but our cookie is
> a little more sophisticated. It is basically an url encoded json object like
> {"analytics":false,"personalisation":true"} (but then url encoded ofc...)
> So the targeting:consentCookieValue wont work with this. That is why I have
> added my project specific TrackingPolicy for now.

Ok makes sense!

Thanks for the feedback

Regards Ard

[Ard Schrijvers]

On Wed, Dec 13, 2017 at 3:58 PM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> For our use we most likely need it in 12.1.x, as we started yesterday with
> our upgrade from 11.2.x to 12.1.x.
> So no, I don't need this in 10 ;)
>
> And about the release date, it would ofcourse be best if this can be
> released as soon as possible, Because otherwise my client has to explain why
> we store a _visitor cookie, without its being used. ;)
> We could of course ask the visitors to trust us on our blue eyes ;) But I
> believe its better to not have the cookie set at all.

I agree. Would January be soon enough?

@AJ : What do you think? Cut a targeting only release and have the
option for now that end projects who want this fixed bump targeting
only or do a full blown release? Think the last option might be a bit
too much for this

Regards Ard

[Woonsan Ko]

On Wed, Dec 13, 2017 at 10:05 AM, Ard Schrijvers <ard.sch...@bloomreach.com> wrote:

On Wed, Dec 13, 2017 at 3:54 PM, 'dennis.nijssen' via Hippo Community
<hippo-community@googlegroups.com> wrote:
> Yea I know it will break our project once we upgrade, but I think for this
> one "override" it is something we should take for granted. Ofcoure every
> upgrade we need to check this point.
> So I won't mind if the bean id gets renamed, as it seems such a simple
> override at this point. Ofcourse when the whole TargetingPolicy interface
> implementation in the TargetingService gets changed I might end up in more
> trouble.

When we'd modify the TargetingPolicy we'd create upgrade documentation
for it. The interface will in minor version never have BC-incompatible
changes. Also with java api when possible we work with deprecation
annotations and javadoc of course.

I don't plan to 'fix' the bean id name: Let's leave it as is.

I think we can change the ID, but keep the old ID in name or alias:

- https://docs.spring.io/spring/docs/4.2.8.RELEASE/spring-framework-reference/htmlsingle/#beans-beanname-alias

In the targeting module, the bean might want to set 'name' attribute to the old id simply.

Regards,

Woonsan

 

> To post to this group, send email to hippo-community@googlegroups.com

> RSS:
> https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
> ---
> You received this message because you are subscribed to the Google Groups
> "Hippo Community" group.
> To unsubscribe from this group and stop receiving emails from it, send an

> email to hippo-community+unsubscribe@googlegroups.com.

> Visit this group at https://groups.google.com/group/hippo-community.
> For more options, visit https://groups.google.com/d/optout.



--
Hippo Netherlands, Oosteinde 11, 1017 WT Amsterdam, Netherlands
Hippo USA, Inc. 71 Summer Street, 2nd Floor Boston, MA 02110, United
states of America.

US +1 877 414 4776 (toll free)
Europe +31(0)20 522 4466
www.onehippo.com

--
Hippo Community Group: The place for all discussions and announcements about Hippo CMS (and HST, repository etc. etc.)

To post to this group, send email to hippo-community@googlegroups.com

RSS: https://groups.google.com/group/hippo-community/feed/rss_v2_0_msgs.xml?num=50
---
You received this message because you are subscribed to the Google Groups "Hippo Community" group.

To unsubscribe from this group and stop receiving emails from it, send an email to hippo-community+unsubscribe@googlegroups.com.

Visit this group at https://groups.google.com/group/hippo-community.
For more options, visit https://groups.google.com/d/optout.

--

Woonsan Ko

woons...@bloomreach.com

[dennis.nijssen]

If it will be somewhere in the first week of january, but that might be a little tricky with vacations and such. 

[Ard Schrijvers]

Yes, but at some point in time, you would like to delete the old bean,
but we can afaik never 'communicate' via intellij that some bean id
has been deprecated : That means it will become a documentation
task...meaning the bean ids will pretty much become api and as an
effect most likely a lot of the spring wiring. I still doubt whether
we can support this

> Woonsan Ko
> woons...@bloomreach.com
> US +1 877 414 4776 (toll free)
> Europe +31(0)20 522 4466
>
>

[dennis.nijssen]

Hello Ard,

Do you have any idea regarding the release date for the _visitor cookie fix? 

Kind regards,

Dennis

[Arent-Jan Banck]

We do not yet have a date set for the next maintenance release, likely this will be later this month. We can create a patch release for the relevance module earlier. If you add a Jira issue for the project I can update it with the release information.

Thanks & regards,

- Arent-Jan

[dennis.nijssen]

Hi Ard,

First of all thank you for creating the fix so soon regarding the _visitor cookie that was set regardless of the tracking consent. 

I was also wondering earlier in this discussion you were referring to a possible improvement / factory that could be made to have a custom implementation for the TrackingPolicy, instead of overriding the "com.onehippo.cms7.targeting.DefaultTrackingPolicy" bean. Is this something that has been put on the backlog? As I can not view the relevance project on issues.onehippo.com myself ;)

Kind regards,

Dennis

[Ard Schrijvers]

Hey Dennis,

On Wed, Jan 17, 2018 at 8:59 AM, 'dennis.nijssen' via Hippo Community
<hippo-c...@googlegroups.com> wrote:
> Hi Ard,
>
> First of all thank you for creating the fix so soon regarding the _visitor
> cookie that was set regardless of the tracking consent.
>
> I was also wondering earlier in this discussion you were referring to a
> possible improvement / factory that could be made to have a custom
> implementation for the TrackingPolicy, instead of overriding the
> "com.onehippo.cms7.targeting.DefaultTrackingPolicy" bean. Is this something
> that has been put on the backlog? As I can not view the relevance project on
> issues.onehippo.com myself ;)

no this is not on the backlog. Could you otherwise create an issue in
your own jira project and refer to this thread and assign it to
helpdesk?