Questions about the Px stack

[Sverker Abrahamsson]

Hi,
we at the Swedish Crime Prevention Council (Brottsvförebyggande rådet, BRÅ) are working on a PoC investigating if the Px stack can be used for publication of BRÅ public data.

It has been a bit confusing as there seem to be different discussion groups, and the issues sections on the modules under https://github.com/PxTools/ but it seems this group is the most recent active and the correct place for general discussions.

Current state is that we have an internal demo portal deployed in our kubernetes cluster with PxWeb2, PxWebApi, CNMM and our internal publishing pipelines. These are set up via a helm chart in a very convenient way and the intention is that we will be able to release this helm chart to the community for others to use when we got all the dots connected (there is no desition on that yet but it's the intention). We also have docker files for our local development containers.

We find the other modules fairly well described, but our main challenge is understanding the data model in the backend. I.e. to where we publish our data. We do understand there are two ways to handle data, either via PX files or CNMM database. A plausible explanation why there are very little information except for very old documents on both PX and CNMM is that they are so trivial that everybody except us understands them immediatly.. :-D

First question: Are publishing via PX files or CNMM equivalent feature wise? I.e. there are no capabilities that are availible in one datasource and not in the other? I found somewhere a system diagram where PX files were used as transfer format from an internal CNMM database to public CNMM database and hence they should have equivalent feature set.

Second question: We currently use the image for CNMM cloned from https://github.com/statisticssweden/CNMM, I don't find any repo for that under https://github.com/PxTools. Is this repo still valid or outdated?

Third question(s): On https://www.scb.se/vara-tjanster/statistikprogram-for-px-filer/PxWeb/px-web-med-sql-databas/ there are documentation for CNMM 2.3. Elsewhere CNMM 2.4 is mentioned. Are there any documentation and reference implementation for CNMM 2.4, or later? Which is compatible with latest release of PxWebApi v2.2.0?

Forth questions: Regarding PX files the most recent specification I find is from 2013, is this still valid or outdated? I saw discussion about that the specification is not very exact and an attempt from Mikael Gueck to create machine readable specs and tools but not sure how complete they are.

Please bear with our (possibly) noob questions, these are to ensure we are on the right track and not dancing in the dark... :-D

[Stefan Jul Gunnersen]

Hello,

I cannot answer all (or maybe any) of your questions, but I will try.

First of all, this is (probably) the most active group besides a dedicated Slack group for developers (which you might want to join - I hope somebody else can provide more information - it is found at pxtools.slack.com - you need an invite to join.

Second, great that you got it (somewhat) working with Kubernetes and containers. We at Statistics Denmark (who currently are in the (long) process of implementing such a setup) might contact you ;-)

As regards CNMM database and PX Files, my belief is that they provide the same functionality, just in wildly different ways. The PX file is a different approach from the database. But they have different strengths and weaknesses. Files are fast, selfcontained and can be read efficient. Databases provide easier update and publishing options (we use it), but can be more difficult to setup and handle - files can easily be exported as PX files, by the way (using PxWebApi). I am no expert (no, really, I am not), but it totally depends on your needs and capabilities what you choose. I really do think they support the same options/functionality, but support for file/database in tools may vary a lot.

About the CNMM image, I think you are right, and I think it is the plan to move/create it in PxTools, but I do not know if this is the correct status.

We in Statistics Denmark have also considered using CNMM 2.4 instead of 2.3 (or even come with suggestions for a 2.5, that would be fun, but no promises at all - above my paygrade), and we, too, have wondered about documentation for this. So +1 on that subject.

PX File format is somewhat being "locally" extended/expanded/revised, it seems. I believe some new stuff has made its way (or is on the way - or was talked about) to PxWebApi, but I am unsure. We talked about it at the annual PX Meeting (in Helsinki, Finland, back in October,  I think it was), but where it ended, I am unsure. So the latest official version is probably still from 2013. Also, PxWebApi uses JSON-STAT2 as its "local" data format for use with PxWeb 2. 

Other members of this group - which are acutal experts - will most probably be able to fill in the blanks and answer your questions more precisely than my answers :-)

Kind regards

Stefan Jul Gunnersen

Chief adviser (IT developer) at Statistics Denmark