How does Archipelago do analytics?

[Bryan Brown]

Hi all,

I was wondering how Archipelago Commons is doing analytics. Is it making use of Matomo, or something else? I'm currently working with Matomo in the context of a vanilla Drupal 8 data repository and I have some ideas for how it should be done, but thought this might be an opportunity to look at how others in the Drupal 8 repository space are handling this and figured it might be an opportunity for collaboration and code sharing.

[Nate Hill]

Hey Bryan, is your project live? Can you share a link to it?

Nate

On Tue, Jun 2, 2020 at 1:17 PM Bryan Brown <bryj...@gmail.com> wrote:

Hi all,

I was wondering how Archipelago Commons is doing analytics. Is it making use of Matomo, or something else? I'm currently working with Matomo in the context of a vanilla Drupal 8 data repository and I have some ideas for how it should be done, but thought this might be an opportunity to look at how others in the Drupal 8 repository space are handling this and figured it might be an opportunity for collaboration and code sharing.

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.
To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-com...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/021dc8aa-ddf1-4502-bfa9-5942b7724a80%40googlegroups.com.

--

Nate Hill

Executive Director

Metropolitan New York Library Council

metro.org

917-375-0332

[Bryan Brown]

Hi Nate,

Its still in the development phase so not public yet, but with plans to do a soft launch for external user testing in November. 

[Bryan Brown]

Just so it doesn't seem like I'm hiding anything, we do have a placeholder site at https://ldbase.org/ that describes the project, and all of our code and configuration is in https://github.com/ldbase, which includes a development environment Vagrant VM at https://github.com/ldbase/ldbase_deployment.

On Tuesday, June 2, 2020 at 1:40:03 PM UTC-4, Bryan Brown wrote:

Hi Nate,

Its still in the development phase so not public yet, but with plans to do a soft launch for external user testing in November. 

On Tue, Jun 2, 2020 at 1:36 PM Nate Hill <nh...@metro.org> wrote:

Hey Bryan, is your project live? Can you share a link to it?

Nate

On Tue, Jun 2, 2020 at 1:17 PM Bryan Brown <bryj...@gmail.com> wrote:

Hi all,

I was wondering how Archipelago Commons is doing analytics. Is it making use of Matomo, or something else? I'm currently working with Matomo in the context of a vanilla Drupal 8 data repository and I have some ideas for how it should be done, but thought this might be an opportunity to look at how others in the Drupal 8 repository space are handling this and figured it might be an opportunity for collaboration and code sharing.

--
You received this message because you are subscribed to the Google Groups "archipelago commons" group.

To unsubscribe from this group and stop receiving emails from it, send an email to archipelago-commons+unsub...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/archipelago-commons/021dc8aa-ddf1-4502-bfa9-5942b7724a80%40googlegroups.com.

[Diego Pino]

Hi Bryan,

Hope all is well there and congrats on ldbase! Happy to discuss approaches, share info and interact with the D8 community on this, or any other, topic. 

On our side what we do: Stock Matomo is our approach for now, with very similar customization to the code we wrote for Drupal 7/Islandora 7 and async calls via ajax instead of the PHP/HTTP api.

Custom Dimensions v/s variables is still a larger question because of the multi value v/s single value per item constraints but also raises the question of how much we plan on providing our own code v/s reuse the existing modules and just make our data structures easier to be exposed natively to D8.

Since in Archipelago we have way more metadata because of strawberryfields (and hierarchical, JSON) than in a vanilla Drupal 8 (well 9 starting tomorrow June 3rd!), i'm planning on using our exposed keyname provider plugin system, in specific the one that already parses https://jmespath.org through native drupal token info as options so Matomo module can directly be used to keep track of internal metadata coming via JSON, like wikidata subjects, geographic metadata, etc. Rest should be decently solved by vanilla D8 Matomo module really (like users)

I don't see this token development as a priority in our roadmap since we are approaching Version 1.0 under Drupal9 very fast and i'm more concerned with an easy multi importer equivalent for the expect July release but i guess it is still handy to have and also useful in other places. In every tracking module (matomo and google one), all happens on the page attachments hooks and it is also how we will provide schema.org page header info and script info for google scholar and friends via a twig template, so if we code something custom it will go there. As you know twig templates are key to our workflow so probably options for matomo will be made available as such to admins too.

A larger difference and i actually feel core difference is that in Archipelago we don't use '/node/1' as paths. We wrote our own UUID argument to entity resolver. So metadata endpoints (the ones that cast in realtime JSON to IIIF, or to MODS, or SCHEMA.ORG, in the form or 'do/someuuid/metadata/iiifv3/manifest.json' or do/someuuid/file/someohteruuod/default.jpeg' ) can live inside the same HTTP URL path and thus be extracted directly inside matomo to belonging to the same Archipelago Digital Object via an URL pattern!, making keeping track of metadata, Persistent URLs of course and even file attachments (also accessible from the /do/uuid path as subpaths) all together quite simple. I feel that piece of code already simplified a lot of us. Imagine keeping track of file downloads that happened on a media entity and being able to say . "Hey, all this happened in association of this node!" Quite difficult in a stock Drupal8, if not, impossible and needed in an IR/repository environment to get all things-related together.

So to your post, what are the tracking needs you have identified that are not solved by the code Matomo or google stats module? Have you found roadblocks or red-flags? Are you thinking on customizing the backend or just the front end?How much of your code is generic v/s tied to your particular fields and configs?

Let us know what are those ideas you have and i would be happy to discuss them in the more general Drupal 8/9 context.

 

Take care and thanks a lot for reaching out!

Diego Pino

Metro.org

[Bryan Brown]

Diego,

My first foray into the matter was using the stock D8 Matomo module, but I was already worried about it being technical debt from the start since it only provided access to custom variables but not custom dimensions (which seem to be the future of Matomo). I had attempted to set up some custom variables just to try to understand Matomo and set up custom variables like UUID and NID (with future plans to expose a 'parent' object ID as a token so that could be communicated and tracked as well), but found the data that Matomo was returning via segmentation to be very inaccurate; for example I would create a fresh basic page and it would tell me that it had several views already which makes me think that I'm using the segmentation API wrong. I ended up giving up on this approach with the idea that maybe it would be easier/more accurate via custom dimensions and that I should try it that way, but since custom dimensions don't exist in the core Matomo D8 module I would either need to make the decision to stick with custom variables, or write code to add custom dimension capability to Matomo (but really hoping someone else has done that already if this is truly the future of Matomo).

Another major issue as you pointed out is making the connection between a downloaded file and the page it was downloaded from since in repository terms this is isn't a download of the file, but a download of the object the referrer page represents. 

Between all of these issues, I decided to kick the can of Matomo work in LDbase down the road a bit and work on things that had clearer paths to success, but with the public testing date becoming closer every day I'm realizing that I need to make a decision and go with it. I don't think the Islandora 8 community has done any Matomo work beyond getting a Matomo server installed as part of the basic stack and installing the module to point to it, but I think a lot more custom data will need to be pushed to Matomo in order to get it to generate usage reports that will be valuable to users and administrators, and I see a lot of overlap in terms of use cases even if the data model underneath isn't the same so I figured I'd give it a shot and ask here to so I don't reinvent the wheel.

So based on what you said, would it be fair to say that you are currently using default Matomo data with plans to do more custom stuff in the future?

- Bryan

[Diego Pino]

Hi Bryan,

Time passes at a different speed lately, sorry for the late reply. 

Yes, we have plans on adding more custom stuff regarding exposing data to modules like Matomo.  But the changes needed to do what i read you need (and i could be misunderstanding, you probably have other needs) are not really big (good news) so the code will also not too much on our side.

Example approach based on: https://git.drupalcode.org/project/matomo/-/blob/8.x-1.x/js/matomo.js

That is the core JS and its super slim. So what i pretend is to do something similar. simply attach on every Digital Object page an extra Javascript settings/custom tiny JS with the things i want to track, including Dimensions but also some custom Variables given the weird fact that dimensions can not be multivalued and will never be, see:

https://github.com/matomo-org/plugin-CustomDimensions/issues/102

basically we will have a hard dependency on the matomo module, inject our own JS and out settings to attach some extra values. Then alter the Matomo provided admin form to allow some extra UI facing settings from our side, specific to Archipelago Digital Objects (fancy name for Any Node bearing a Strawberry field, easy to relate identity concept). Still Its a hard choice (v/s coding the whole thing again because, well, the module itself is not written fast enough for my own standards and adds some overhead)

The catch (or a few catch-es). I'm not sure i want to have too many modules in Archipelago as a statement of sustainability and role separation (we already have 3, pesistance/datatyping, output, input)  but also how hard it is to keep anything backwards and forward not-deprecating in the Drupal world. I spend more time making sure all still works on every update of Drupal than coding new things. 

Still, all this could eventually go into an Archipelago SEO module, where i would want to expose also the JSON-LD/Structured metadata we generate on the fly via the twig template/format_strawberryfield module (quite different to the schema module everyone uses that depends fields and tokens), but if so it will make the later a dependency and i believe you folks could see that as a problem. basically i'm mostly sure our solution even when Drupal 9 aware and totally compatible, will be quite specific to Archipelago Digital objects and metadata needs. For D8/D9 normal content types and everyday fields we feel there is enough in the market and would make supporting that code for us quite complex (unless we get contributors/and maintainers!). 

Second catch . There is another issue with Matomo. It requires Matomo Server side/deploymennt configuring and some of those configurations are very complex to deliver in an automated way. Means you have to go step by step and deploy the server and enabled plugins via the symfony console. I had to do some circus magic and double-mortal-flips without a safety net on the LASIR/Islandora 7 project to reach an in-between state, but it also makes me feel that if i can not automate the deployment, then i need to write very good documentation and make the Drupal side of things quite aware that the other side could be not well deployed. Extra logic and extra failure points.

As you see i have been thinking a lot about this and will for sure provide code and solutions. But all of them will, in some way or other, serve the Archipelago needs and because of that to some extend less the generic Drupal/ Islandora 8 / content types with many fields/ use case.

Take care and please let us know if this answers some of your questions ideas, and if you have further thoughts, comments or suggestions. Also thanks for bringing this up

Cheers

Diego