Spec without global registry?

[Mark]

I'm embarking on a new project and I think spec can be a central component not just to the developer-users of the system but to my end-users as well.  I'm thinking of providing something like a graphical mechanism to describe specs in EDN to bring spec goodness to user created data pipelines.  My only concern is the global registry.  Quickly browsing spec alpha-17's code, it appears the that global registry is an implementation detail of the map spec and, perhaps, others. 

Would it make sense to provide an additional arity for the conform*, unform* and explain* fns which take a user-supplied registry?

[Alex Miller]

We don't have any plans at the moment to support anything but the global registry.

[Mark]

I'm a bit surprised by this.  It seems that the use of the global registry limits spec to development use cases.  Is that intentional?  Maybe I'm worried over nothing

[Kevin Baldor]

I'm interested in the answer to whether it is just an accident of implementation or if there is some compelling reason for the global registry.

I'm still new to Clojure and it would be good to hear the tradeoffs and design process that led to the current implementation.

--
You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to
clojure+unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.
To unsubscribe from this group and stop receiving emails from it, send an email to clojure+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

[Gregg Reynolds]

On Jun 10, 2017 11:57 AM, "Mark" <markad...@gmail.com> wrote:

I'm embarking on a new project and I think spec can be a central component not just to the developer-users of the system but to my end-users as well.  I'm thinking of providing something like a graphical mechanism to describe specs in EDN to bring spec goodness to user created data pipelines.  My only concern is the global registry.  Quickly browsing spec alpha-17's code, it appears the that global registry is an implementation detail of the map spec and, perhaps, others. 

Would it make sense to provide an additional arity for the conform*, unform* and explain* fns which take a user-supplied registry?

what about using a global config var like *spec-registry* to control which registry gets used?

[Sean Corfield]

Can you explain why you think this is the case Mark?

 

We use spec heavily in production (and have been doing so for months) so I’m not following your logic here I’m afraid…

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Alex Miller]

On Monday, June 12, 2017 at 1:36:53 PM UTC-5, Kevin Baldor wrote:

I'm interested in the answer to whether it is just an accident of implementation or if there is some compelling reason for the global registry. 

With spec we wish to encourage the use of attributes with good (qualified) names and global semantics.

While I haven't seen a strong use case for something beyond the global registry, that also doesn't rule out the idea of something else in the future.

On Mon, Jun 12, 2017 at 12:37 PM, Mark <markad...@gmail.com> wrote:

I'm a bit surprised by this.  It seems that the use of the global registry limits spec to development use cases.  Is that intentional?  Maybe I'm worried over nothing

I don't see how that limits it to dev use cases. Can you explain more why you say that? 

[Mark]

'm thinking of exposing the spec machinery to my app's end users who would build specs (probably using a graphical tool) to facilitate building data pipelines.  It seems that the global registry assumes a particular lifecycle for specs which is probably perfect for developer-centric use cases.  I believe I'll want a different lifecycle for specs written by end users. 

[Sean Corfield]

So, you would give all those end-user-created specs a unique namespace prefix which identified them as part of your application. As Alex indicated, spec is predicated on the use of appropriately qualified names, so that they have global meaning.

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

[Mark]

> I don't see how that limits it to dev use cases. Can you explain more why you say that? 

I understand (and completely agree with) the assumption of a global namespace for spec names.  The scope of that namespace is all Clojure developers.  I want a different scope:  the users of my app.  To continue on this thinking, if my app was shared among different customers, each customer would have their own namespace - their own registry.  Further, I suspect that the lifecycle for specs in my app's scope will be a bit different than the Clojure developer scope.  Specifically, I can see that some customers will insist on retracting specs.

Make sense?

[Mark]

Yeah, I can see how to make it work but I'm worried about lifecycle issues.  What if my customers insist on the ability to retract a spec?  Showing them a Rich Hickey video is not a particularly good response :)  Leaving that aside, users who just play around risk polluting the global registry and, in the worst case, creating memory issues.

[Alan Forrester]

On 12 Jun 2017, at 20:41, Mark <markad...@gmail.com> wrote:

> > I don't see how that limits it to dev use cases. Can you explain more why you say that?
>
> I understand (and completely agree with) the assumption of a global namespace for spec names. The scope of that namespace is all Clojure developers. I want a different scope: the users of my app. To continue on this thinking, if my app was shared among different customers, each customer would have their own namespace - their own registry.

What problem would be solved by each customer having his own registry?

> Further, I suspect that the lifecycle for specs in my app's scope will be a bit different than the Clojure developer scope. Specifically, I can see that some customers will insist on retracting specs.

What do you mean by retracting specs? And what problem would this solve?

Alan

[Mark]

> What problem would be solved by each customer having his own registry? 

Name clashes (to be fair, elsewhere on this thread, Sean Corfield suggested prefixing and this would certainly work).  

> What do you mean by retracting specs? And what problem would this solve? 

Retracting a spec means that it's not in use anymore. In this instance, I'm just imagining my users' requirements.  I can foresee them wanted to indicate that a particular spec is no longer in use.  I suppose I could always "change" the spec to be (constantly false).  

[Alex Miller]

I think it's your responsibility to make specs "sufficiently unique". Prefixing with a standard namespace you control seems like it would work.

There is an enhancement winding through jira to support the ability to remove a spec from the registry by doing (s/def ::foo nil) and I expect that to go in soon. https://dev.clojure.org/jira/browse/CLJ-2060

[Gregg Reynolds]

On Jun 12, 2017 4:55 PM, "Alex Miller" <al...@puredanger.com> wrote:

I think it's your responsibility to make specs "sufficiently unique". Prefixing with a standard namespace you control seems like it would work.

pls excuse me for butting in, but i wonder what happens when i require 14 namespaces and 7 of them register foo.bar/baz in the global registry? who wins? we don't have this prob with vars; if i require foo.bar and foo.baz, and they both define x, no prob.  but spec namespacing is different, no?  the namespace you use for a spec is independent of the ns in which you define/specify it.  which defeats the purpose of namespacing.  clojure namespacing is controlled; spec namespacing is not.  which leads me to think that maybe spec registries should be namespaced, just like everything else.

--

[Steve Buikhuizen]

I can think of a use-case in support of Mark's position. 

If you are building a hosted ETL web service (like Mulesoft) and you want users to be able to use spec to validate records flowing through it.

One customer wants :org/postcode to be integer (i.e. US) and another wants it to be string (i.e. UK) With a global registry this is not possible. 

It's possible to spec this but then you need to predict all the data shapes supported i.e. not an open validation system

You can prefix the key with an org id e.g. :<id>.org/postcode but if you are hosting a lot of orgs the registry will become large. 

Removal could help manage space but it would require some kind of registry garbage collection i.e. incidental complexity.

Any thoughts on how best to do this with spec? Could a multi-spec help here?

[Mark Addleman]

Yes, I think most of the problems can be solved through prefixing (although the solution is a bit hacky, IMO) but the real problem with the global registry is that its not based on an abstraction but a concrete implementation.  The only specific problem I can think of right now is the incidental complexity related to garbage collection - like Steve points out.

If the registry were based on an abstraction, I can imagine a few fun experiments.  Right now, the implementation assumes that the registry is loaded when a namespace is initialized.  Why not load the registry from a database, edn files or off the network?  Piling onto the network idea, why not have a true global registry which gets updated as Clojure developers publish specs in real time? 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---

You received this message because you are subscribed to a topic in the Google Groups "Clojure" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/clojure/4jhSCZaFQFY/unsubscribe.
To unsubscribe from this group and all its topics, send an email to clojure+u...@googlegroups.com.

[Sean Corfield]

If you’re building a multi-tenant system, then this is not how you do it:

 

> One customer wants :org/postcode to be integer (i.e. US) and another wants it to be string (i.e. UK)

 

In a multi-tenant system, you have very few “global truths” and for _anything_ that an end-user may touch, it is not a “global truth” by definition. Therefore you _must_ partition per-user specs either by process (entirely) or via their user ID (or similar) so that they absolutely cannot clash.

 

Also, remember that all of the clojure.core specs _are_ global truths – so they can’t have a different meaning for each user and, more importantly, your system must not allow an end user spec  to be created that overwrites a clojure.core spec (and that’s _your_ responsibility when designing a multi-tenant system).

 

It is the responsibility of the multi-tenant system to manage loading and unloading of per-user specs, not just “leave it up to the JVM”.

 

To be honest, if you want per-user isolation on a per-ETL-job basis, your best bet may be to spin each job off in its own JVM container anyway since different jobs could require radically different JVM configurations and resources. That also ensures you can’t have any in-memory data conflicts.

 

This is no different from multi-tenant databases: you either completely partition your per-user data (separate databases/schemas) or you ensure that the user ID is part and parcel of every single key reference into the data store.

 

(for reference, we have a multi-tenant system that runs around 100 different websites with millions of unique user accounts, so keeping per-user data isolated while allowing for certain global shared resources goes with the territory)

 

Sean Corfield -- (970) FOR-SEAN -- (904) 302-SEAN
An Architect's View -- http://corfield.org/

"If you're not annoying somebody, you're not really alive."
-- Margaret Atwood

 

--

You received this message because you are subscribed to the Google
Groups "Clojure" group.
To post to this group, send email to clo...@googlegroups.com
Note that posts from new members are moderated - please be patient with your first post.
To unsubscribe from this group, send email to

clojure+u...@googlegroups.com

For more options, visit this group at
http://groups.google.com/group/clojure?hl=en
---
You received this message because you are subscribed to the Google Groups "Clojure" group.

To unsubscribe from this group and stop receiving emails from it, send an email to clojure+u...@googlegroups.com.

[Alex Miller]

On Monday, June 12, 2017 at 5:24:23 PM UTC-5, Gregg Reynolds wrote:

On Jun 12, 2017 4:55 PM, "Alex Miller" <al...@puredanger.com> wrote:

I think it's your responsibility to make specs "sufficiently unique". Prefixing with a standard namespace you control seems like it would work.

pls excuse me for butting in, but i wonder what happens when i require 14 namespaces and 7 of them register foo.bar/baz in the global registry? who wins? we don't have this prob with vars; if i require foo.bar and foo.baz, and they both define x, no prob.  but spec namespacing is different, no?  the namespace you use for a spec is independent of the ns in which you define/specify it.  which defeats the purpose of namespacing.  clojure namespacing is controlled; spec namespacing is not.  which leads me to think that maybe spec registries should be namespaced, just like everything else.

Well, I completely disagree - these are the identical problem with the identical scoping mechanism - namespace qualifiers. You should create sufficiently unique names by using qualifiers you "control" (either by delegating to a url that you own as the root or a trademark, etc). This is the recommended approach for Clojure namespaces (and for Java packages), and it's the identical recommendation for spec namespaces.

Sure, you can be deliberately harmful, but then stuff will break and not work... so, don't do that? 

 

[Alex Miller]

On Monday, June 12, 2017 at 7:42:53 PM UTC-5, Sean Corfield wrote:

If you’re building a multi-tenant system, then this is not how you do it:

I think Sean covered it - this seems like a problem with the design of this service.

 

[Alex Miller]

On Monday, June 12, 2017 at 7:03:13 PM UTC-5, Mark wrote:

Yes, I think most of the problems can be solved through prefixing (although the solution is a bit hacky, IMO) but the real problem with the global registry is that its not based on an abstraction but a concrete implementation.  The only specific problem I can think of right now is the incidental complexity related to garbage collection - like Steve points out.

I look forward to when that is an issue. :)  

 

If the registry were based on an abstraction, I can imagine a few fun experiments.  Right now, the implementation assumes that the registry is loaded when a namespace is initialized.  Why not load the registry from a database, edn files or off the network?  Piling onto the network idea, why not have a true global registry which gets updated as Clojure developers publish specs in real time? 

Nothing is stopping you from running code to load the registry via a database, edn files, or off a network. s/form exists exactly for the purpose of allowing you to extract specs as edn suitable for sending through a network, files, etc. Also, saying no now does not mean saying no forever - we can always expand support for more registry support later.

[Mark]

> I look forward to when that is an issue. :)  

Heh.  Fair enough!

> Also, saying no now does not mean saying no forever - we can always expand support for more registry support later

Got it.  Thanks!

[Mario T. Lanza]

First of all, Clojure, core.async, spec, the whole of what you guys produce is fabulous.  I have nothing but respect for the work you're all doing.  It's Clojure's simple design that has me enjoying programming more than ever.

That said, I'd like to add a new perspective to the discussion Mark started.  It has a wider application than spec alone.

Baldridge in Core Async in Use (great talk!) exemplifies how to do apis that are asynchronous.  He encourages developers to avoid asynchronous code, pushing its necessary appearance to as few strategic points as possible.

This parallels the fundamental Clojure separation of identity from state.  It is by the nature of pushing the necessary manipulation of state to as few strategic points as possible that we can lean on pure functions.

Using globals reduces this separation and makes a design more imperative.  The possibility of fine-grained control is lost when state is pushed into globals.  I can appreciate that globals have a place in simplifying a public api, but why not fully expose the underlying data structures?  This was done with hierarchies.

Hierarchies (the defining of is-a relationships via `derive`) are directly related to spec as evidenced by the existence of `multi-spec`.  So if hiearchies, which have their own global counterpart, are exposed, it would appear consistent to expose the registry.  Both are related.  They're both used to define ontologies.

So while spec's use of a global registry suits most cases, it complicates things for those like me who want first-class schemas.  I am building a CMS where schemas are a core feature.  I need to manage their lifetimes and scopes at runtime.

I can't imagine it would have been much harder to implement spec using what what Gary Bernhardt refers to as a Functional Core, Imperative Shell.  It's a good principle.  I find that it aligns  closely with idea of deferring the messy parts for later found in both Baldridge's talk and Clojure's state management strategy.

I've found that designing around a functional core increases possibilities.  It's what:

• makes implementing undo/redo mostly trivial.
  
• allows control over the lifetime and scope of constructs.
  
• enables the use of single state atoms that host nested data structures
  

When we only have access to the imperative shell, these things are not possible.

To help illustrate, the following lines are listed in the order of increasing possibility.

(derive tag parent)
(derive h tag parent)
(swap! h (derive tag parent))

Only one of these options -- the one using a pure function -- accommodates all of the above possibilities.  The longer I've done Clojure, the closer I've come to a consistent use of this approach.

spec's design unnecessarily takes things off the table.

I know I'm being dogmatic, but why not consistently expose a functional core as a matter of principle?

Thanks for your consideration.

Mario