Jira (PUP-1829) Define PCore - the Puppet Meta Meta Model

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model Change By: Henrik Lindberg Summary: Implement a Define PCore - the  Puppet  Core  Meta  Meta  Model  (PCore) Add Comment

This message was sent by Atlassian JIRA (v6.4.12#64027-sha1:e3691cc)

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model Change By: Henrik Lindberg

The PCore model is the puppet meta  meta  model which  is  will be  used to describe  all puppet meta models. If your head is spinning: * A meta model is like a schema e.g.  the  types in the  json-schema for a  puppet  system  catalog * A meta meta model is like the jsonschema json-schema, that defines what you can define in a json-schema In Puppet we now have a very rich Type System .  It plays  PCore takes advantage of that and this allows us to define very compact models. The first model that we express is  the  same role  PCore meta meta model  as  ECore  it defines what follows .  This  The first real  model  we aim to express using PCore   is  required  the AST model. We are doing this  because  existing modeling Ecore, jsonschema, yaml-schema, etc. are based on assumptions about  the  Puppet  target language (java, json, yaml) and thus contain features suitable for each of those technologies (and naturally also lacking features). While it is possible to model anything with those, it requires yet another abstraction layer since the basic  types  available  are not  (easily) directly describable in ECore  enough to describe the puppet types .   Currently With our own model  we  have,  require fewer abstraction layers. What we process is already typed by  the  Puppet::Pops::Types Ecore model  type system , and  this model  it  should be  expanded  just as easy to describe a model as it is to define a value's type. When something is typed by the type system ,  enough information is known to enable that the value is serialized. This means that objects do not have to support a variety of serialization protocols (to_yaml, to_json, to_msgpack, to_this, to_that). To work around that, in Puppet the protocol is now "to_data_hash"  and  renamed  "from_data_hash" and that step reduces the information  to  PCore  a hash of serializable values and that is in turned serialized with a to_yaml etc .  When doing this we created another problem for us; this is a lossy transformation ! (This is one reason it is hard to serialize anything but the most rudimentary data types9). The work consists of: *  implementing the  Specifying  PCore  metamodel *  implementing a validator  Implementing PCore in PCore * * writing issues  Specifying how an instance model  ( i.e. formal error messages) * implementing  a  tree dumper (textual representation  catalog, an AST, etc ) * implementing factory / helper methods * PCore to Ecore generator to Ruby (to realize the classes into running code) * "scraping" 3.x types (and maybe? providers) to form  +  a  new model of "old" implementations PCore  model  is used  by the Evaluator  to serialize  and  the new Loaders.  deserialize that model, The functionality is experimental in 3.6. and is expected to be released in 4.0. Other tickets will contain the more concrete things (like writing a serializer)

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Story Points: 24 5

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model Change By: Henrik Lindberg

The PCore model is the puppet meta meta model which will be used to describe all puppet meta models.

If your head is spinning:

* A meta model is like a schema e.g. the json-schema for a puppet catalog

* A meta meta model is like the jsonschema json-schema, that defines what you can define in a json-schema

In Puppet we now have a very rich Type System. PCore takes advantage of that and this allows us to define very compact models. The first model that we express is the PCore meta meta model as it defines what follows. The first real model we aim to express using PCore   is the AST model. We are doing this because existing modeling Ecore, jsonschema, yaml-schema, etc. are based on assumptions about the target language (java, json, yaml) and thus contain features suitable for each of those technologies (and naturally also lacking features). While it is possible to model anything with those, it requires yet another abstraction layer since the basic types available are not enough to describe the puppet types.  With our own model we require fewer abstraction layers. What we process is already typed by the type system, and it should be just as easy to describe a model as it is to define a value's type. When something is typed by the type system, enough information is known to enable that the value is serialized. This means that objects do not have to support a variety of serialization protocols (to_yaml, to_json, to_msgpack, to_this, to_that). To work around that, in Puppet the protocol is now "to_data_hash" and "from_data_hash" and that step reduces the information to a hash of serializable values and that is in turned serialized with a to_yaml etc. When doing this we created another problem for us; this is a lossy transformation ! (This is one reason it is hard to serialize anything but the most rudimentary data types9). The work consists of: * Specifying PCore * Implementing PCore in PCore * Specifying how an instance model (a catalog, an AST, etc) + a PCore model is used to serialize and deserialize that model, * It is expected that this work will spin off requirements for enhancements to the type system (Some already identified and worked on).

Other tickets will contain the more concrete things (like writing a serializer)

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg assigned an issue to Thomas Hallgren

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Assignee: Henrik Lindberg Thomas Hallgren

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model Thomas Hallgren Assigned this to you to review the ticket itself; does it contain enough scope and detail. (Serialization will be dealt with separately).

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829

Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg

PCore is an enabler for many future features: * partitioning results to make what is required in each communication to be smaller * efficient sending of a machine state (the RAL) as a fact * a more efficient replacement for plugin-sync  * describing resource types in puppet * polyglot implementations (native resource types) * functions on the agent (without distributing the pp source to all agents) * and many more This ticket is about none of the future things - only the basic processing needed to support the "Native Puppet Parser" but with the required concepts in place for higher order functionality (or at least that the implementation does not block the higher order functionality by being so simplistic that it has to be immediately redone). The design has to be as future proof as we can reasonably make it. As this sits at the core it is both hard and takes a long time to change when done wrong. We should rather err on the side of being too cautious and e.g. include extra meta information than skip it in the belief that what we implement first will last for ever).

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model Henrik Lindberg, here are some general thoughts that I'd like to convey, just to verify that the approach I'm taking is what you'd expect. All types can be described in terms of the new Object type described in PUP-5844. E.g. Numeric could be describe like this (in practice, it already exists, so this would be an error): type Numeric inherits Scalar { 'from' => Numeric, 'to' => Numeric 'unbounded?' => Callable[{} >> Boolean] } PCore consists of the current basic types in the Puppet Type system. A serialized meta-schema will consist of such type instances, possibly with the addition of defined Object type instances. Implementing PCore in PCore is all about describing all the PCore types as Object types. A serialized meta-meta schema consist of those Object types. It will, for instance, contain the Object type that describes the Object type. The Object types for the classes in an RGen model can be dynamically generated since all features of each class are described in detail (with type information). Other runtime implementations in Ruby, for which we want to share type information, will need to be associated with a handcrafted Object type since the type of their members is unknown but very likely something that we will want to specify. All Object types must be added to a TypeImplementationRegistry and the TypeCalculator must consult this registry when it performs type inference (in essence, the fallback to Runtime('ruby', <some class>) must first consult the registry). We will need a global registry in combination with registries local to a ModelSet. The TypeCalculator must hence be ModelSet aware (this might be a challenge, given the somewhat global nature of it).

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model

That sounds about right. A couple of thoughts: While we at runtime probably do not need the PCore equivalence of the RGen Ecore models, we do need to generate them and make them available with correct model name and version. Our implementation in Ruby happens to have an implementation of that model that is built with RGen. For the things not in RGen a model is needed. The Locator in the AST is a good example - the RGen model cheats as it is just saying that the Locator is a Ruby object. Naturally if you try to serialize that (with RGen at least) it breaks. Clearly a ModelSet is needed to handle a) the RGen AST, and b) the supporting model for Locator (possibly more stuff like the type system i.e. PCore itself. If we were to do this like in Ecore the models would have model references to the models it depends on. Maybe we can do that with RGen too ? i.e. the AST model has references (URIs), and we know how to resolve those. Thus, if you need to use something like Locator, you would need a model that says that there is a type called Puppet::AST::Locator and what is, and in Ruby there needs to be a model that maps that ie. type Puppet::AST::Locator = Runtime[Ruby, Locator] (or something like that). I would like those to be documents (URIs) rather than a sequence of calls. It is a kind of "Bindings Model". Not sure if the type calculator needs to be ModelSet aware - it operates on the Runtime objects. They are what they are all the time. It is when you read/write via a ModelSet that you map them.

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model

Regarding the TypeCalculator and ModelSet. We can do this in one of two ways. 1. The mapping between a runtime object (say, Locator) and it's associated Puppet type is registered in a global registry. This means that all serialization and type inference can function without a ModelSet. It also means that all types live in a global namespace which means that two versions of the same type cannot co-exist in the same runtime, even if they are mapped to different runtime implementations. I don't know if this matters, but this is IMO one motivation for having ModelSet instances in the first place. 2. The mapping is not global. It's local to a ModelSet so of course the ModelSet must be present in order to use the mapping. The mapping is necessary when doing type inference so the TypeCalculator (and the Serializer) must be able to somehow use what the ModelSet provides.

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model

1. We must support different mapping of name to type in different contexts (at runtime this would be an environment, but we should not build the mechanism on that). I think we have a hierarchy (two levels). The runtime itself has one mapping - something like the AST, the Locator etc. It will not change between environments contexts - we only have one implementation (or at least - we do not intend to have multiple versions). I think that a ModelSet refers to a parent set and that it is possible to override in the derived ModelSet. In many cases, you can just use the master runtime ModelSet (e.g. AST and locators as there are no user defined types there, only instructions that when evaluated creates types). 2. What I proposed at 1. probably solves this.

Add Comment

[Thomas Hallgren (JIRA)]

Thomas Hallgren assigned an issue to Unassigned

Puppet / PUP-1829

Define PCore - the Puppet Meta Meta Model

Change By: Thomas Hallgren Assignee: Thomas Hallgren

Add Comment

This message was sent by Atlassian JIRA (v6.4.13#64028-sha1:b7939e9)

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Team: Puppet Developer Support

Add Comment

This message was sent by Atlassian JIRA (v6.4.14#64029-sha1:ae256fe)

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829

Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Scrum Team: Language Team: Puppet Developer Experience Team/s: Puppet Developer Experience Epic Status: In Progress Epic Name: Pcore Status: In Progress Developing Workflow: Scrum Team Engineering Epic  Workflow Issue Type: Story Epic

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Story Points: 5

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Fix Version/s: PUP 4.y Fix Version/s: PUP 5.y

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model Change By: Henrik Lindberg

The PCore model is the puppet meta meta model which will be used to describe all puppet meta models. If your head is spinning: * A meta model is like a schema e.g. the json-schema for a puppet catalog * A meta meta model is like the jsonschema json-schema, that defines what you can define in a json-schema In Puppet we now have a very rich Type System. PCore takes advantage of that and this allows us to define very compact models. The first model that we express is the PCore meta meta model as it defines what follows. The first real model we aim to express using PCore   is the AST model. We are doing this because existing modeling Ecore, jsonschema, yaml-schema, etc. are based on assumptions about the target language (java, json, yaml) and thus contain features suitable for each of those technologies (and naturally also lacking features). While it is possible to model anything with those, it requires yet another abstraction layer since the basic types available are not enough to describe the puppet types.  With our own model we require fewer abstraction layers. What we process is already typed by the type system, and it should be just as easy to describe a model as it is to define a value's type. When something is typed by the type system, enough information is known to enable that the value is serialized. This means that objects do not have to support a variety of serialization protocols (to_yaml, to_json, to_msgpack, to_this, to_that). To work around that, in Puppet the protocol is now "to_data_hash" and "from_data_hash" and that step reduces the information to a hash of serializable values and that is in turned serialized with a to_yaml etc. When doing this we created another problem for us; this is a lossy transformation ! (This is one reason it is hard to serialize anything but the most rudimentary data types9). The work consists of: * Specifying PCore * Implementing PCore in PCore * Specifying how an instance model (a catalog, an AST, etc) + a PCore model is used to serialize and deserialize that model, * It is expected that this work will spin off requirements for enhancements to the type system (Some already identified and worked on). Other tickets will contain the more concrete things (like writing a serializer) PCore is an enabler for many future features: * partitioning results to make what is required in each communication to be smaller * efficient sending of a machine state (the RAL) as a fact * a more efficient replacement for plugin-sync  * describing resource types in puppet * polyglot implementations (native resource types) * functions on the agent (without distributing the pp source to all agents) * and many more

This ticket is about none of the future things - only the basic processing needed to support the " Native Puppet Parser Rich Data Types "  needed for various purposes (binary data, encrypted data, rich resource attributes values in general),  but with the required concepts in place for higher order functionality (or at least that the implementation does not block the higher order functionality by being so simplistic that it has to be immediately redone).

The design has to be as future proof as we can reasonably make it. As this sits at the core it is both hard and takes a long time to change when done wrong. We should rather err on the side of being too cautious and e.g. include extra meta information than skip it in the belief that what we implement first will last for ever).

Add Comment

[Lindsey Smith (JIRA)]

Lindsey Smith updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Lindsey Smith Team/s: Puppet Developer Experience Agent

Add Comment

[John Duarte (JIRA)]

John Duarte updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: John Duarte Labels: triaged

Add Comment

[Moses Mendoza (JIRA)]

Moses Mendoza updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Moses Mendoza Labels: triaged

Add Comment

[Henrik Lindberg (JIRA)]

Henrik Lindberg updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Henrik Lindberg Component/s: Type System

Add Comment

[Geoff Nichols (JIRA)]

Geoff Nichols updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Geoff Nichols Team/s: Agent Platform Core

Add Comment

[Justin Stoller (JIRA)]

Justin Stoller updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Justin Stoller Component/s: Server

Add Comment

This message was sent by Atlassian JIRA (v7.5.1#75006-sha1:7df2574)

[Josh Cooper (Jira)]

Josh Cooper updated an issue

Puppet / PUP-1829 Define PCore - the Puppet Meta Meta Model

Change By: Josh Cooper Fix Version/s: PUP 5.y

Add Comment

This message was sent by Atlassian Jira (v8.5.2#805002-sha1:a66f935)

[Josh Cooper (Jira)]

Josh Cooper commented on PUP-1829

Re: Define PCore - the Puppet Meta Meta Model We don't have plans to implement this in the near future, so I'm going to close this out. Feel free to reopen if this feature is needed in the future.

Add Comment