Versionning proposal
mozer
"http://expath.org/ns/core"
"http://expath.org/ns/<module>"
we need a function
core:function-available(AnyURI namespace, String name) as boolean
core:function-available(AnyURI namespace, String name, int arity) as boolean
core:function-available(QName function) as boolean
core:function-available(QName function, int arity) as boolean
Extending
We can add (because we have function-available())
* new name function (as soon as it make sens in the current package)
* same name function with different arity than existing one
We can create new package
We can deprecate existing function in a package trough documentation
If we want to refactor completly we have the following choices
Having a major new branch
"http://v2.expath.org/ns/core"
or local branching
"http://expath.org/ns/core/v2"
or new module
Xmlizer
David Carlisle
we need a function
core:function-available(AnyURI namespace, String name) as boolean
core:function-available(AnyURI namespace, String name, int arity) as boolean
core:function-available(QName function) as boolean
core:function-available(QName function, int arity) as boolean
Yes, I think we need these. But actually function-available() isn't as easily usable in xpath2 as it was xpath1
as if(core:function-available(QName('math:sin')) then math:sin($x) else $x is likely to give a compile time error
on the math:sin($x) even if function-available would return false when run.
I'm not sure if it's possible to devise an expath (or exquery) pragma syntax that works something like use-when in xslt and allows compile time guarding of expressions that might not be portable?
David
--
http://dpcarlisle.blogspot.com/
Florent Georges
Hi,
> versioning proposal
> "http://expath.org/ns/core"
> "http://expath.org/ns/<module>"
Yes, I agree with that, providing as you said that a module can
introduce a version number if absolutely necessary. I am not sure the
"core" ns should be treated specially, but who knows?
> core:function-available(AnyURI namespace, String name) as boolean
> core:function-available(AnyURI namespace, String name, int arity) as boolean
> core:function-available(QName function) as boolean
> core:function-available(QName function, int arity) as boolean
I think xs:string would be better than xs:anyURI (for the same
reasons F&O functions takes strings for URIs.)
> Extending
> We can add (because we have function-available())
> * new name function (as soon as it make sens in the current package)
> * same name function with different arity than existing one
Those cases do not really need function-available() for versioning
purpose. If the implementation does not support them, there would be
a compilation error.
But if one has an alternative, as David said, it would maybe be
interesting for her to be able to detect it at compile-time. But I am
not sure how we could handle that in an elegant way. If we backport
XQuery pragmas, we'll generate compile-errors if the implementation
does not support them, and if we use something like (:# ... #:) we
could generate compile-time errors for legitimate XPath comments, or
not reach our goals if the implementation does not recognize it.
I wonder if this is not the responsibility of the host language to
ensure the existence of the function (name or arity,) and to take
adequate behavior if not (use-when in XSLT, pragma in XQuery, or a
separate call to function-available() if, say, in Java.)
So about namespace URIs: +1. About function-available(): +1. About
preventing compile-time errors for unsupported functions: yes, but
how?
Regards,
--
Florent Georges
http://www.fgeorges.org/
Kurt Cagle
Kurt Cagle
Managing Editor
http://xmlToday.org
Martin Probst
>> preventing compile-time errors for unsupported functions: yes, but
>> how?
>
> util:eval() or util:load-module() capability.
I think that will not quite be possible in XQuery. The spec says it's
a static, compile time error if a function is not available. E.g. in
case we have a newer http-get function that supports an additional
cookie parameter, this would not work:
if (function-available("http://expath.org/ns/core", "http-get", 2)) then
expath:http-get('http://www.example.com', $mycookie)
else
expath:http-get('http://www.example.com')
It might somehow work if we have higher order functions, and they
support dynamically instantiating functions:
let $http-call as function() :=
if (function-available("http://expath.org/ns/core", "http-get", 2)) then
get-function('expath:http-get#2')
else
get-function('expath:http-get#1')
return ...
... but with plain XQuery as it is now, such a dynamic fall back is
impossible. Maybe this would make a good case for a first extension
function.
Martin
Florent Georges
> >> So about namespace URIs: +1. About function-available(): +1. About
> >> preventing compile-time errors for unsupported functions: yes, but
> >> how?
> > I think function-available() plays a bigger role if there is some type of
> > util:eval() or util:load-module() capability.
> I think that will not quite be possible in XQuery. The spec says it's
> a static, compile time error if a function is not available.
particularly useful with a dynamic evaluation function:
if ( util:function-available("http:send-request", 1) ) then
util:eval("http:send-request(...)")
else
...
Martin Probst
> util:eval("http:send-request(...)")
Or rather util:eval($namespace, $functionname, $args...)?
Martin
Florent Georges
able to evaluate any XPath expression: util:eval("1 + 2").
Martin Probst
>> > util:eval("http:send-request(...)")
>
>> Or rather util:eval($namespace, $functionname, $args...)?
>
> I am note sure to understand; a dynamic evaluation function would be
> able to evaluate any XPath expression: util:eval("1 + 2").
Eh yes, I'd just rather see a dynamic function invocation function.
That seems less invasive, and we all know eval is evil.
util:invoke($namespace, $functionname, $args...)
That way you don't have to take care you get the syntax right, you
don't have to invoke a parser, and so on.
Martin
James Fuller
Eh yes, I'd just rather see a dynamic function invocation function.
>> > if ( util:function-available("http:send-request", 1) ) then
>> > util:eval("http:send-request(...)")
>
>> Or rather util:eval($namespace, $functionname, $args...)?
>
> I am note sure to understand; a dynamic evaluation function would be
> able to evaluate any XPath expression: util:eval("1 + 2").
That seems less invasive, and we all know eval is evil.
util:invoke($namespace, $functionname, $args...)
util:invoke(....) or util:eval(...) would work off the default context ... we might also want to consider having an overloaded version be able to pass in context e,g, serialization params, pragmas, etc ....we may even want to pass in a default context or named variables instead of doing $somevar/util:invoke(...)
cheers, Jim Fuller
we might also want to consider an overloaded version to arbitrarily pass in default context as well
Adam Retter
> That seems less invasive, and we all know eval is evil.
>
> util:invoke($namespace, $functionname, $args...)
>
> That way you don't have to take care you get the syntax right, you
> don't have to invoke a parser, and so on.
Blowing the eXist trumpet for a second (sorry), we have something
similar already -
util:function() and util:call()
util:call(util:function(xs:QName("http:send-request"), 1), $args)
Documentation is available here -
http://demo.exist-db.org/exist/xquery/functions.xql
--
Adam Retter
EXQuery Founder
t: +44 (0)7714 330069
e: adam....@exquery.org
w: www.exquery.org
Kurt Cagle
I think the entire eXist util: namespace is worth some serious consideration here (of course that could be said for a lot of extensions in eXist, but that one in particular is just chock full of goodness). For those that don't have an implementation handy here's what's supported:
binary-doc-available
binary-to-string
binary-to-string
catch
declare-namespace
declare-option
eval-with-context
eval-with-context
exclusive-lock
get-fragment-between
get-module-description
get-sequence-type
hash
import-module
index-key-documents
index-key-documents
index-key-occurrences
index-key-occurrences
index-keys
index-keys
is-module-registered
log-app
qname-index-lookup
registered-functions
serialize
serialize
string-to-binary
string-to-binary
unescape-uri
uuid
I find that a significant amount of the code that I need for XQuery app development comes from util:
Florent Georges
> > I am note sure to understand; a dynamic evaluation function
> > would be able to evaluate any XPath expression: util:eval("1
> > + 2").
> Eh yes, I'd just rather see a dynamic function invocation
> function. That seems less invasive, and we all know eval is
> evil.
> util:invoke($namespace, $functionname, $args...)
the behavior of which could be provided by a general purpose
util:eval() function or by first-class function objects. But
actually, I guess that could be useful for that exact issue of
detecting at runtime if a function is supported, without
requiring the whole machinery for full evaluation or first-class
object functions.
Could someone comment on that?
Florent Georges
> util:invoke(....) or util:eval(...) would work off the default context ...
> we might also want to consider having an overloaded version be able to pass
> in context e,g, serialization params, pragmas, etc ....we may even want to
> pass in a default context or named variables instead of doing
> $somevar/util:invoke(...)
define the interaction between eval() and the XPath context.
For invoke(), that less obvious to me. This just provide an
alternative way to write a function call (util:invoke('my:name', $ar1,
$arg2) instead of my:name($ar1, $arg2)) so I think everything should
be the same between both, except that the first one won't generate a
compile time error, but rather a runtime error.
Did I misunderstood something?
Florent Georges
> > Eh yes, I'd just rather see a dynamic function invocation function.
> > That seems less invasive, and we all know eval is evil.
> > util:invoke($namespace, $functionname, $args...)
> similar already -
> util:function() and util:call()
> util:call(util:function(xs:QName("http:send-request"), 1), $args)
class function object system. I wonder after all if this is really
more work than just invoke(). If not, I guess there's no point to
invoke(). For first-class function objects, I guess the best approach
would be to take John Snelson's "Higher Order Functions for XQuery"
<http://snelson.org.uk/~jpcs/higher-order-functions.html> as a basis,
but defining functions instead of grammar changes.
That's definitely something that would be very beneficial for the
community, I think.
Dimitre Novatchev
It will be an overkill to implement dynamic evaluation just for the
purpose of making function-available() possible.
One could think of less grandiose mechanisms, such as "deferred
evaluation" or "suppressed static typechecking".
If static type checking could be suppressed (and this would be a nice
meta-function of EXPath), and there was a "try... catch..."
capability, one could write something like ( excuse my ignorance of
XQuery syntactic rules :) )
$foo-available =
suppress-static-typechecking
{
try
{ (: simple invocation of foo :) foo(); return true;}
catch
{return false;}
}
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Florent Georges
> I think the entire eXist util: namespace is worth some serious consideration
> here (of course that could be said for a lot of extensions in eXist, but
> that one in particular is just chock full of goodness). For those that don't
> have an implementation handy here's what's supported:
the eXist function documentation system, they can go to
http://demo.exist-db.org/exist/xquery/functions.xql, choose in
the list "Or display all in: http://exist-db.org/xquery/util,"
then click "Browse."
Dimitre Novatchev
> would be to take John Snelson's "Higher Order Functions for XQuery"
> <http://snelson.org.uk/~jpcs/higher-order-functions.html> as a basis
seems that HOF will be present in the next version of XQuery, so it
would not be useful to create another, competing HOF implementation in
EXPath.
Florent Georges
> It will be an overkill to implement dynamic evaluation just for the
> purpose of making function-available() possible.
functions wouldn't be another approach. At first glance, I am not
sure this would require much more work than invoke(), while providing
a far more useful feature.
> One could think of less grandiose mechanisms, such as "deferred
> evaluation" or "suppressed static typechecking".
grandiose ;-)
> If static type checking could be suppressed (and this would be a nice
> meta-function of EXPath), and there was a "try... catch..."
Florent Georges
> Also, it seems that HOF will be present in the next version of
> XQuery, so it would not be useful to create another, competing
> HOF implementation in EXPath.
try-catch) have been added only in the last XQuery 1.1 WD. I can
be totally wrong, but I don't expect it to pass as a REC this
year. Having a similar definition in EXPath would be also the
opportunity to get feedback before its publication, besides
providing the functionality to not-XQuery 1.1 languages...
But once again, that's more a matter of who's going to work on that
(or not.)
Kurt Cagle
I"m also going to commit a bit of heresy here, but WHY in this case should eval() be considered evil? So long as you properly sandbox for permissions, you're dealing with a language that is already implicitly distributed, where your specific code trajectories may come from multiple distributed sources. It is less efficient, yes, and written badly you can screw yourself over, but that's true of XQuery in general.
As a case in point - suppose that you wanted to set up a query such that the predicate and sorting parameters for that query were fed from external resources. In this case, using an eval makes a huge deal of sense:
let $collection := collection("bar/bat")
let $predicate := myfunc:get-external-predicate("mysource","$entry//myChildElement/text() = 'foo'")
let $sort := myfunc:get-external-sort("mysource","$entry//myChildSortElement ascending")
let $expr := concat("for $entry in $collection where ",$predicate," order by ",$sort," return $entry")
let $result := eval($expr)
return $result
This is of course a not terribly secure case - I see predicate as being a mechanism for converting something like a keyword set from a Search box on a browser into a much more secure query, but the idea is there nonetheless.
I think we need to be careful about not getting so caught up in the notion that something is bad in imperative languages (and there I would agree that eval() is evil) should also be seen as being bad in declarative ones. I'll probably be taken to task for this, but I'm not necessarily ready to abandon eval() without their being a rational reason to do so in the context of the language.
Martin Probst
like unit tests (a la unit-test($test as function()).
However the current proposal at
http://snelson.org.uk/~jpcs/higher-order-functions.html will not work
for the use case of function-available, as it cannot do dynamic
function calls.
About eval: to make this actually useful, we would need to use some
eval function that evaluates the whole query within the current
context, with namespaces and variables and all. I don't know about
other XQuery implementations, but in xDB this is not going to be easy,
as at run time we have already "forgotten" much of the needed
information and/or transformed it into a more execution-suited format.
I really have a strong dislike to concatenating program statements at
run time, and I think it doesn't make a difference whether we're in a
declarative or imperative programming language. The kind of bugs, the
implementation overhead, and the security problems are all the same.
> This is of course a not terribly secure case - I see predicate as being a
> mechanism for converting something like a keyword set from a Search box on a
> browser into a much more secure query, but the idea is there nonetheless.
That is exactly part of the problem. Having written three different
XQuery parsers, I'd have true pity for someone trying to escape
literals to keep an XQuery injection attack away.
> I think we need to be careful about not getting so caught up in the notion
> that something is bad in imperative languages (and there I would agree that
> eval() is evil) should also be seen as being bad in declarative ones. I'll
> probably be taken to task for this, but I'm not necessarily ready to abandon
> eval() without their being a rational reason to do so in the context of the
> language.
There are use cases for eval, granted. But I think our use case at
hand here - dynamically picking a function to call - is much simpler.
My rule of thumb is if you can get away without using eval, do so. And
I think this is such a case.
Martin
Adam Retter
> here (of course that could be said for a lot of extensions in eXist, but
> that one in particular is just chock full of goodness). For those that don't
> have an implementation handy here's what's supported:
A great endorsement for eXist ;-) But hangon just a second... I agree
that there are a lot of good general purpose functions in there (I am
sure other implementations probably have many of them as well),
however there are several that are very eXist specific (locking,
indexing, optimisation, some of the eval stuff) or out of reach of a
pure XPath effort (just too complicated to make use of in XPath) and
more relevant to EXQuery.
Also the util module has suffered more than any module in eXist for
being a general purpose dump for anything that doesnt really fit into
its own module. Because we had no versioning (am I really starting
that conversation again?!?), we have never been able to remove or tidy
that module. Personally in maybe release 1.6 of eXist (1.4 will be out
soonish and odd versions are development branches), I want to do a
complete overhaul and reorganisation of the modules whilst culling
anything legacy. Yes it may break applications, but we are carrying a
lot of cruft around and its getting to the point where its hard for
developers to locate the function they need from the documentation -
actually EXPath and EXQuery modules could be the perfect excuse ;-)
Adam Retter
> I was going to say I didn't see the point of such a function,
> the behavior of which could be provided by a general purpose
> util:eval() function or by first-class function objects. But
> actually, I guess that could be useful for that exact issue of
> detecting at runtime if a function is supported, without
> requiring the whole machinery for full evaluation or first-class
> object functions.
>
> Could someone comment on that?
I would always avoid full evaluation where possible.
As a developer with an eye on security I see util:eval() as the devils
own work, for me this is right up there beside native language binding
in XQuery or XSLT. If I can provide constructs which are more limited
then I will always do so. I am not saying that it should not be
available, but in an implementation I am providing, such functionality
will be disabled by default, as it enables too much risk.
I am sure we are all aware of SQL injection attacks, has anyone tried
an XQuery injection attack? I am sure it is possible. With util:eval()
enabled in something like a HTTP REST context and people having easy
access to an eval function and using parameters from POST or GET then
I can see a lot of problems occuring.
Sure such issues are perhaps a developer training issue, but I prefer
perhaps a more proactive approach where if I can prevent stupid
mistakes from happening then everyones life becomes easier hopefully.
Adam Retter
> meta-function of EXPath), and there was a "try... catch..."
> capability, one could write something like ( excuse my ignorance of
> XQuery syntactic rules :) )
>
> $foo-available =
> suppress-static-typechecking
> {
> try
> { (: simple invocation of foo :) foo(); return true;}
> catch
> {return false;}
> }
>
Perhaps I am mistaken, but I wonder if we are not over stepping the
scope of EXPath, EXPath is for XPath and should develop only what is
permissible in XPath surely?
Adam Retter
> run time, and I think it doesn't make a difference whether we're in a
> declarative or imperative programming language. The kind of bugs, the
> implementation overhead, and the security problems are all the same.
> That is exactly part of the problem. Having written three different
> XQuery parsers, I'd have true pity for someone trying to escape
> literals to keep an XQuery injection attack away.
> There are use cases for eval, granted. But I think our use case at
> hand here - dynamically picking a function to call - is much simpler.
> My rule of thumb is if you can get away without using eval, do so. And
> I think this is such a case.
I completely agree with all of these statements, some excellent points.
Dimitre Novatchev
>
>> If static type checking could be suppressed (and this would be a nice
>> meta-function of EXPath), and there was a "try... catch..."
>> capability, one could write something like ( excuse my ignorance of
>> XQuery syntactic rules :) )
>>
>> $foo-available =
>> suppress-static-typechecking
>> {
>> try
>> { (: simple invocation of foo :) foo(); return true;}
>> catch
>> {return false;}
>> }
>>
>
> Perhaps I am mistaken, but I wonder if we are not over stepping the
> scope of EXPath, EXPath is for XPath and should develop only what is
> permissible in XPath surely?
Purpose and Scope.
For me the best results from EXPath would come not so much from
providing a stadard library of extension functions (certainly this
would be a very good and useful result in itself), but from also
extending the XPath language with at least some new features that were
most asked for .
At present many people have identified at least one feature as badly
neaded: the support for a nested sequence -- a sequence that can have
another sequence as an individual item.
Another often sited feature is HOF (more for XQuery as in XSLT we have
had FXSL for many years).
It seems that W3C are doing something for HOF in XQuery, but why not
address nested sequences in EXPath? An implementation is
straightforward and an average developer (me included) can provide
such in a couple of days.
Some people (most notably Florent) have proposed other extensions to
XPath (such as the Try ... Catch mechanism). While this seems a lot
more tricky to implement in a functional setting, why not address the
issue at all? If implemented properly, the benefits would be huge. A
lot of people are asking how to determine if the document() function
was successful or if there was timeout when consuming a remote
service. I believe that having try... catch would provide a good
mechanism to address such kind of issues.
To conclude, I propose that extending XPath itself (not only by
recommending a set of extension functions) is high priority in the
Vision, Goals and Scope of EXPath.
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
>
>
>
>
Florent Georges
> However the current proposal athttp://snelson.org.uk/~jpcs/higher-order-functions.htmlwill not work
> function calls.
but that's because it is not possible to get a function item at
runtime, from a string.)
> About eval: to make this actually useful, we would need to use some
> eval function that evaluates the whole query within the current
> context, with namespaces and variables and all.
act on a different context (context item, in-scope namespaces, in-
scope variable bindings, etc.) Even maybe providing the ability to
modify (especially restrict) the available function signatures in the
static context.
I see a general purpose eval module as defining a function to
compile an expression, and other to evaluate the compiled expression,
allowing one to set the exact context to use. A simple eval
(xs:string) function would only be a convenient shortcut then.
> There are use cases for eval, granted. But I think our use case at
> hand here - dynamically picking a function to call - is much simpler.
> My rule of thumb is if you can get away without using eval, do so. And
> I think this is such a case.
Florent Georges
> For me the best results from EXPath would come not so much from
> providing a stadard library of extension functions (certainly this
> would be a very good and useful result in itself), but from also
> extending the XPath language with at least some new features that were
> most asked for .
> At present many people have identified at least one feature as badly
> neaded: the support for a nested sequence -- a sequence that can have
> another sequence as an individual item.
XPath community(ies,) especially allowing complex data structure
preserving the identity of its items. And clearly such features are
more than welcome here.
But I am convinced extension functions remain the best option to
implement them. I would be reluctant to change the XPath grammar
within a processor-independent project.
If you take the example of John's HOF proposal, something similar
could be achieved using extension functions. Instead of my:f#1, use
fun:function('ns:f', 1), providing the function's behavior was
properly defined regarding the static and dynamic contexts.
Of course, discussing how the function-based feature in EXPath could
be translated into a grammar change in, say, XPath 2.1 would be
interesting. But IMHO proposing features in EXPath through grammar
changes would work against an important goal: portability.
I agree you get a point: this is not clear, and should maybe be
clearly stated on the website.
> To conclude, I propose that extending XPath itself (not only by
> recommending a set of extension functions) is high priority in the
> Vision, Goals and Scope of EXPath.
function is the *highly* preferred way of providing them (opposed to
grammar changes in particular.)
Dimitre Novatchev
In just a few words, we agree that a couple of extension functions
(one for a reference to a sequence and another for dereferencing a
sequence) are sufficient for a very basic, rude implementation of
nested sequences.
If nobody would extend XPath itself, I don't mind extending it at the
function level :)
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Adam Retter
> providing a stadard library of extension functions (certainly this
> would be a very good and useful result in itself), but from also
> extending the XPath language with at least some new features that were
> most asked for .
I understand the purpose of the extension functions. But I have some
questions otherwise - When you start extending XPath as a language are
you not really turning it more and more into XQuery (or what the next
version of XQuery is likely to include). Why not just use XQuery
instead of XPath if you need language functionality that XPath does
not have?
> At present many people have identified at least one feature as badly
> neaded: the support for a nested sequence -- a sequence that can have
> another sequence as an individual item.
Excuse my ignorance but I am not a pure XPath developer, my background
is really in XQuery. This may not be the place to ask, but for my sake
can you explain this need briefly - why a sequence of sequences rather
than a node set?
((1,2,3), (8,9,0)) =
<sequence><sequence><item>1</item><item>2</item><item>3</item></sequence><sequence><item>8</item><item>9</item><item>0</item></sequence></sequence>
> Some people (most notably Florent) have proposed other extensions to
> XPath (such as the Try ... Catch mechanism). While this seems a lot
> more tricky to implement in a functional setting, why not address the
> issue at all? If implemented properly, the benefits would be huge. A
> lot of people are asking how to determine if the document() function
> was successful or if there was timeout when consuming a remote
> service. I believe that having try... catch would provide a good
> mechanism to address such kind of issues.
try/catch would certainly be welcome in XQuery ;-)
Florent Georges
> In just a few words, we agree that a couple of extension functions
> (one for a reference to a sequence and another for dereferencing a
> sequence) are sufficient for a very basic, rude implementation of
> nested sequences.
That's at least the first step. Once we have that, we could always
consider its drawbacks and new reasonable orientations :-)
Dimitre Novatchev
>
>> For me the best results from EXPath would come not so much from
>> providing a stadard library of extension functions (certainly this
>> would be a very good and useful result in itself), but from also
>> extending the XPath language with at least some new features that were
>> most asked for .
>
> I understand the purpose of the extension functions. But I have some
> questions otherwise - When you start extending XPath as a language are
> you not really turning it more and more into XQuery (or what the next
> version of XQuery is likely to include). Why not just use XQuery
> instead of XPath if you need language functionality that XPath does
> not have?
and try...catch, just to name a few.
>
>
>> At present many people have identified at least one feature as badly
>> neaded: the support for a nested sequence -- a sequence that can have
>> another sequence as an individual item.
>
> Excuse my ignorance but I am not a pure XPath developer, my background
> is really in XQuery. This may not be the place to ask, but for my sake
> can you explain this need briefly - why a sequence of sequences rather
> than a node set?
>
> ((1,2,3), (8,9,0)) =
> <sequence><sequence><item>1</item><item>2</item><item>3</item></sequence><sequence><item>8</item><item>9</item><item>0</item></sequence></sequence>
>
is how a nested sequence *has to be modelled* at present in FXSL, and
this is quite painful).
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
>
Florent Georges
> Excuse my ignorance but I am not a pure XPath developer, my background
> is really in XQuery. This may not be the place to ask, but for my sake
> can you explain this need briefly - why a sequence of sequences rather
> than a node set?
> ((1,2,3), (8,9,0)) =
> <sequence><sequence><item>1</item><item>2</item><item>3</item></sequence><sequence><item>8</item><item>9</item><item>0</item></sequence></sequence>
But then, you cannot have complex data structure that preserve every
properties of nodes that you would like to reference from there. For
example their identity or their validation info. Because when you add
an item to a node, it is copied and some properties of the resulting
node are changed from the original one.
> try/catch would certainly be welcome in XQuery ;-)
It is on its way:
http://www.w3.org/TR/2008/WD-xquery-11-20081203/#id-try-catch :-)
Adam Retter
> and try...catch, just to name a few.
Sure but I think it is only a matter of time before HOF and try/catch
will make it into XQuery, we can assist it along the way of course. My
point was that XPath should not turn into XQuery. XPath fulfills a
very simple need (and does it very well) and if you need more
complexity then there is XQuery.
>> ((1,2,3), (8,9,0)) =
>> <sequence><sequence><item>1</item><item>2</item><item>3</item></sequence><sequence><item>8</item><item>9</item><item>0</item></sequence></sequence>
>>
>
> Because this is both more awkward and prohibitively expensive. (This
> is how a nested sequence *has to be modelled* at present in FXSL, and
> this is quite painful).
Whilst trying not to be unrealistic, I am wondering if this is this
not a detail of the underlying implementation? Is it not the case that
if the implementation was more efficient at handling nodes then you
would not need a nested sequence?
If that is the case (it might not be) would it not be better to invest
time improving the performance of the underlying implementation?
Dimitre Novatchev
>> is how a nested sequence *has to be modelled* at present in FXSL, and
>> this is quite painful).
>
> Whilst trying not to be unrealistic, I am wondering if this is this
> not a detail of the underlying implementation? Is it not the case that
> if the implementation was more efficient at handling nodes then you
> would not need a nested sequence?
> If that is the case (it might not be) would it not be better to invest
> time improving the performance of the underlying implementation?
>
>
Other people can perhaps expalin best, how expensive it is to create
new elements and documents.
And there is the architectural problem: Why should we have a
(non-nested) sequence as a basic datatype but need to represent a
nested sequence (including its non-nested subsequences) in a
completely different way? Even in a student project I would mark this
as a bad decision.
On the other side a nested sequence has so simple implementations that
they are trivial.
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Florent Georges
> 2009/4/19 Adam Retter wrote:
>> Excuse my ignorance but I am not a pure XPath developer, my background
>> is really in XQuery. This may not be the place to ask, but for my sake
>> can you explain this need briefly - why a sequence of sequences rather
>> than a node set?
>> ((1,2,3), (8,9,0)) =
>> <sequence><sequence><item>1</item><item>2</item><item>3</item></sequence><sequence><item>8</item><item>9</item><item>0</item></sequence></sequence>
> But then, you cannot have complex data structure that preserve every
> properties of nodes that you would like to reference from there. For
> example their identity or their validation info. Because when you add
> an item to a node, it is copied and some properties of the resulting
> node are changed from the original one.
Besides, of course, you cannot create nodes in XPath :-)
Kurt Cagle
Not disagreeing about util: being a dumping ground, and there's really no question that about 2/3 of those could/should be repartitioned into new namespaces. I just thing they represent a nice sample of utility functions that we should be looking at for expath.
I don't necessarily see the need for versioning here in the namespace - what you need more than anything is an obsolescence mechanism, documentation that can be placed within the existing namespaces docs saying "WARNING: This function is obsolete. Consider using foo() instead. This function will be phased out on Jan. 1, 2010".
My big concern with versioning is that if you have specific versions, you also force vendors to support all versions, which is actually counter to the intent. I'd be more inclined to say that what may need versioning is EXPath itself and that only at major version level. This way, there's a very clearcut division between all libraries, and people aren't playing version whack-a-mole trying to debug their code.
Adam Retter
> you need more than anything is an obsolescence mechanism, documentation that
> can be placed within the existing namespaces docs saying "WARNING: This
> function is obsolete. Consider using foo() instead. This function will be
> phased out on Jan. 1, 2010".
In eXist we do have a mechanism for marking functions as deprecated,
with some accompanying documentation - just as you suggested actually
;-) But we then encounter the problem of not knowing how long we
should wait before removing a function in its entirety.
Kurt Cagle
I've been thinking about this for a while, and I think it comes back to a data type that isn't in XPath at all - a hash type. The problem is that it begins to get into XPath as an OOP language, though the implementation even in XQuery wouldn't be hard:
hash:create() as ^hash^
hash:get-value($hash-obj as ^hash^,$key as xs:string) as xs:any*
hash:set-value($hash-obj as ^hash^,$key as xs:string, $obj as xs:any*)
hash:serialize($hash-obj as ^hash^) as node()
hash:parse($hashnode as node()) as ^hash^
I'm using the rather odd ^hash^ notation to indicate an object, something which can sort of get away with as long as we specifically don't ask what that object is. serialize and parse would make it possible to convert this object into and out of XML representation. I understand the rational for not doing it in the original spec and its a valid one - such hashes are definitely not declarative, which is why they'd never end up specifically in the W3C spec, but they could definitely serve in an ancillary fashion.
Florent Georges
> hash:set-value($hash-obj as ^hash^,$key as xs:string, $obj as xs:any*)
Do you mean the hash passed as parameter should be modified as a
side-effect of the function call?
Kurt Cagle
let $hash := hash:set-value($hash-obj as ^hash^,$key as xs:string,$obj as xs:any*) as ^hash^
but that places significant restrictions on scope usage.
However, its the only way that I can see creating sequences within sequences or compound structures, for that matter. Not that happy with the solution myself, which is why I suspect it never made its way into XPath 2, but a hash is a persistent data structure.
Take a look as well at the XQuery Update Facility code - it seems to recognize that side-effect modifications may be unavoidable once you start dealing with update facilities.
Dimitre Novatchev
Moreover, this is an implementation detail and, by definition, any
implementation details are best to be avoided in a specification.
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Kurt Cagle
James Fuller
So how would you create a nested sequence? Short of encoding such a sequence as a sequence of XML structures, each of which are an encoding of a sequence, I can't see how'd you do it.
references ?
ta, Jim Fuller
Florent Georges
First, about side-effects of an extension function, it would be
better to avoid them as much as possible. They introduce
unpredictability in code, and optimizers in existing processors
actually show problems with such extensions.
> So how would you create a nested sequence? Short of encoding such a sequence
> as a sequence of XML structures, each of which are an encoding of a
> sequence, I can't see how'd you do it.
By using what XSLT 2.0 calls an external object:
http://www.w3.org/TR/xslt20/#external-objects
This is an item(), but neither an atomic type nor a node(). Exactly
as the proposed function() item type. So the exact type is a black
box (and in plain, standard XPath 2.0 can only be represented as
item() in an ItemType:
http://www.w3.org/TR/xpath20/#prod-xpath-ItemType
The user can only pass it to another functions defined as such by
the module (besides adding it to a sequence, for instance in a
variable.) For instance (let's abbreviate a nested sequence by ns,
even if it is not the best option in the XML world, but I don't have
enough imagination right now):
let $ns1 as item() := ns:make-ns((1, 2, 3))
let $ns2 as item() := ns:make-ns((5, 6, 7))
let $s as item()+ := ($ns1, 4, $ns2)
return
( count($s), ns:atomize($s[1]), $s[2], ns:atomize($s[3]) )
that would produce:
3, 1, 2, 3, 4, 5, 6, 7
3 is the length of $s (one nested sequence, one integer, and one
other ns.) Then 1, 2, 3 is the content of the first nested sequence
(that is, the first item of $s,) which must be explicitly atomized
before using its items. 4 is the second item of $s, the integer. And
finally 5, 6, 7 is the content of the last ns (the third item of $s.)
Regards,
Adam Retter
> as the proposed function() item type. So the exact type is a black
> box (and in plain, standard XPath 2.0 can only be represented as
> item() in an ItemType:
Ouch - It would be my preference to avoid the use of Item at all
possible costs. Passing around an unknown object just feels very
wrong. We used to have a similar situation in eXist where we had a
type to repesent a Java File Object and you could pass that in and out
of various functions - we have subsequently chased that out of the
extension functions almost entirely now.
Although that said, it would seem that Item is appropriate to this
situation if we really cant come up with anything better.
> The user can only pass it to another functions defined as such by
> the module (besides adding it to a sequence, for instance in a
> variable.)
> even if it is not the best option in the XML world, but I don't have
> enough imagination right now):
Me neither, will give it some thought tomorrow...
Dimitre Novatchev
>
>> This is an item(), but neither an atomic type nor a node(). Exactly
>> as the proposed function() item type. So the exact type is a black
>> box (and in plain, standard XPath 2.0 can only be represented as
>> item() in an ItemType:
>
> Ouch - It would be my preference to avoid the use of Item at all
> possible costs. Passing around an unknown object just feels very
> wrong. We used to have a similar situation in eXist where we had a
> type to repesent a Java File Object and you could pass that in and out
> of various functions - we have subsequently chased that out of the
> extension functions almost entirely now.
topic is avoided in XPath/XQuery/XSLT by not speaking what is inside a
variable that contains a sequence:
<xsl:variable name="mySeq" select="1 to 1000"/>
my:foo($mySeq)
what is contained in $mySeq and is passed to my:foo() above are not
all numbers from 1 to 1000, but it is rather a *reference* to the
sequence 1 to 1000.
XPath and all its implementations already have the reference type, but
they are not speaking about it openly.
Michael Kay calls the reference withs its proper name in a number of
places in his books.
So we do not have to invent anything new, anything frightening -- it
already exists.
We possibly would not even an item() for this type if we cover it well
in expressions like:
(1, makeRef(2 to 10), 3)
or
subsequence(deRef($seq[3]), 2,1)
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
>
Kurt Cagle
We can avoid it by just not speaking about it -- the same way this
topic is avoided in XPath/XQuery/XSLT by not speaking what is inside a
variable that contains a sequence:
<xsl:variable name="mySeq" select="1 to 1000"/>
my:foo($mySeq)
what is contained in $mySeq and is passed to my:foo() above are not
all numbers from 1 to 1000, but it is rather a *reference* to the
sequence 1 to 1000.
Well, sort of. In XQuery it's a little more explicit:
for $myNum in (1 to 1000) return $myNum
But the point is taken.
We possibly would not even an item() for this type if we cover it well
in expressions like:
(1, makeRef(2 to 10), 3)
or
subsequence(deRef($seq[3]), 2,1)
It sounds like we're verging dangerously into the realm of <> templates here ;-)
I think one way to think about it here is that we are walking on the edge of formalizing something that has remained unformalized for a while - the underlying relationship between a namespace and an object definition in XSLT/XQuery. Right now, existing modules are pre-OOP - think of them as global static classes. This changes the moment we get into the modality
let $oRef := ns:new($paramSeq...)
let $result := ns:someFn($oRef,$paramSeq...)
At that stage, a module becomes an instantiable class definition. There are a few situations already where this has taken place, the aforementioned anonymous sequence *to* operator being one of them, and we'll run into more once we break non-fungibility rules with XQUF.
I think the question that needs to be determined is whether in fact this design pattern is one that EX* should endorse. It solves a number of problems, but it also introduces a number as well. For instance, even assuming that we establish formal restrictions to make hash() strictly declarative - you can only make a hash table indirectly via two sets of collections, akin to the XSLT index() function, then you can go with such a formal declaration of an instanced object:
let $hash := hash:new($collection1/labels,$collection2/values)
let $val1 := hash:get($hash,"label1")
let $val2 := hash:exists($hash,"label-in-hash")
In a similar vein, you could make explicit the subsequence notation:
let $seq1 := seq:new(("d","e","f"))
let $seq2 := ("a","b",$seq1)
let $val := seq:get($seq2[3],2)
The advantage, as I see it, are that we objectify the informal don't look don't tell rule, essentially identifying OOP constructs as being legitimate XPath entities (and note that this can be used in XPath itself with the for statement, as Dimitre laid out earlier, or via anonymous reference upon a singleton).
The disadvantage is that we end up having to also formalize the type declaration mechanisms, and up with first and second class types. Such objects by reference could only by specified by item() within the established framework, and would require that each type explicitly import some kind of "type:" library in order to handle introspection, or that a convention of including ns:type() and ns:serialize() exist for each such module namespace.
I think its a legitimate either/or question that should probably be decided earlier. My own druthers are to make such objects explicit, and push this back into XPath 2.1 as part of a serious question for that particular effort, but I can see arguments pro and con here.
-- Kurt Cagle
mozer
>
>> We can avoid it by just not speaking about it -- the same way this
>> topic is avoided in XPath/XQuery/XSLT by not speaking what is inside a
>> variable that contains a sequence:
>>
>> <xsl:variable name="mySeq" select="1 to 1000"/>
>>
>> my:foo($mySeq)
>>
>>
>> what is contained in $mySeq and is passed to my:foo() above are not
>> all numbers from 1 to 1000, but it is rather a *reference* to the
>> sequence 1 to 1000.
>
> Well, sort of. In XQuery it's a little more explicit:
>
> for $myNum in (1 to 1000) return $myNum
Well this is also obviously XPath
Xmlizer
Dimitre Novatchev
>
>> We can avoid it by just not speaking about it -- the same way this
>> topic is avoided in XPath/XQuery/XSLT by not speaking what is inside a
>> variable that contains a sequence:
>>
>> <xsl:variable name="mySeq" select="1 to 1000"/>
>>
>> my:foo($mySeq)
>>
>>
>> what is contained in $mySeq and is passed to my:foo() above are not
>> all numbers from 1 to 1000, but it is rather a *reference* to the
>> sequence 1 to 1000.
>
> Well, sort of. In XQuery it's a little more explicit:
>
> for $myNum in (1 to 1000) return $myNum
I think in XQuery one has to use the "let" clause.
In pure XPath it is not possible at all to define a variable that
contains a sequence. This:
> for $myNum in (1 to 1000) return $myNum
is simply equivalent to:
1 to 1000
and this is not a definition of a variable :(
> I think one way to think about it here is that we are walking on the edge of
> formalizing something that has remained unformalized for a while - the
> underlying relationship between a namespace and an object definition in
> XSLT/XQuery. Right now, existing modules are pre-OOP - think of them as
> global static classes. This changes the moment we get into the modality
>
> let $oRef := ns:new($paramSeq...)
> let $result := ns:someFn($oRef,$paramSeq...)
>
There's absolutely nothing special in the above lines and in fact a
person who doesn't know English (doesn't know the meaning of "new")
and OOP, will not make any difference between the above and:
let $myRef := ref($someSeq)
let $result := foo($myRef,$paramSeq...)
The fact that you *think* "new" means calling a constructor to create
a new instance of a type does not at all mean that this will *really*
take place in an actual implementation. In a particular
implementation there might be lazy evaluation, optimization and due to
various reasons creating a ref may actually be a no-op, as this ref
may already exist and be used for various other purposes.
> I think the question that needs to be determined is whether in fact this
> design pattern is one that EX* should endorse. It solves a number of
> problems, but it also introduces a number as well. For instance, even
The reasoning above proves that this is only an imaginary problem. It
would be good to warn any people that might be too influenced by OOP
that no such problem actually exists.
Kurt Cagle
I think in XQuery one has to use the "let" clause.
In pure XPath it is not possible at all to define a variable that
contains a sequence. This:
is simply equivalent to:
> for $myNum in (1 to 1000) return $myNum
1 to 1000
and this is not a definition of a variable :(
Admittedly - the problem with the discussion here is that the inability to create variables in XPath means that any entity that IS created has to be anonymous:
(for $myNum in (1 to 1000)[. mod 2 =0] return $myNum)
creates two sequences (the entire sequence and (1 to 1000)), but because you can't create variables, they are simply expressions. However, there are still operations that can be defined upon the expressions overall.
> I think one way to think about it here is that we are walking on the edge of
> formalizing something that has remained unformalized for a while - the
> underlying relationship between a namespace and an object definition in
> XSLT/XQuery. Right now, existing modules are pre-OOP - think of them as
> global static classes. This changes the moment we get into the modality
>
> let $oRef := ns:new($paramSeq...)
> let $result := ns:someFn($oRef,$paramSeq...)
There's absolutely nothing special in the above lines and in fact a
person who doesn't know English (doesn't know the meaning of "new")
and OOP, will not make any difference between the above and:
let $myRef := ref($someSeq)
let $result := foo($myRef,$paramSeq...)
Well, sort of. The point I'm making is not in the functional refs but in the assumption that these are in a distinct namespace. That is to say, whether using new(), create(), instantiate(), ref() or whatever, there is a difference between:
let $ref1 := ns1:ref()
and
let $ref1 := ns2:ref()
Once you create the idea of a core object class that is tied to a given namespace implementation, that can store an instantiation of that class, and that, require that the instance variable participate in the methods of that class, then you're into the realm of OOP, even if only marginally:
let $result := ns2:foo($ref1)
is meaningless, unless ns2: is designed as a subclass of ns1:.
The fact that you *think* "new" means calling a constructor to create
a new instance of a type does not at all mean that this will *really*
take place in an actual implementation. In a particular
implementation there might be lazy evaluation, optimization and due to
various reasons creating a ref may actually be a no-op, as this ref
may already exist and be used for various other purposes.
Oh, absolutely agreed, but that wasn't my point. My point is that if you have a namespace that has the potential to create one or more instances of a preferred format for that namespace, that such instances are necessary for the execution of one or more functions within that namespace, and that non-preferred objects when passed into such namespaces generate nonsensical results all point to the potential that namespaces can be conflated with OOP classes. That holds true whether such instances are rendered as Java objects or as XML, though obviously in the latter case such XML can be generated in an ad-hoc fashion and passed in (in other words, such instances are not encapsulated). Whether they are formal OOP is debatable - but I'd argue that they pass the duck test wrt usage.
> I think the question that needs to be determined is whether in fact this
> design pattern is one that EX* should endorse. It solves a number of
> problems, but it also introduces a number as well. For instance, even
The reasoning above proves that this is only an imaginary problem. It
would be good to warn any people that might be too influenced by OOP
that no such problem actually exists.
The problem that we face with XQuery/XPath modularization is that it IS conceptually possible to use an OOP like construction, the question is whether it is advisable. Most languages out there have no such distinction - their structure and syntax is such that they either have no formal mechanism internally to pull off inheritance and as a consequence rely upon syntactical convention to establish namespaces (think PHP4) or they have formal namespace mechanisms and an OOP syntax (think PHP5). XQuery straddles the line between libraries existing in an ad hoc namespace and full OOP. You have a strong namespace mechanism that encourages modularization, but at the same time has no established syntax for doing so.
In effect, we've sidestepped the shift of syntax here (which may actually prove to be a blessing in disguise) but have the mechanism to support OOP with a minimum of formalisms. The reason that I see this as being important is that this makes it possible to build ad hoc types. Should this be a role that EXPath gets involved in?
For instance, suppose that I create a sequence "type" which could be readily stored in an XML format or just as easily in an instantiated Java class:
declare function seq:ref($p_seq as item()*) as node() {
let $seq := for $entry in $p_seq return <entry>{$entry}</entry>
return <sequence expath:ns="http://expath.org/module/sequence" xmlns:expath="http://expath.org">{$seq}</sequence>
};
declare function seq:item-at($seq as node(),$pos as xs:double) as node() {
return $seq/entry[$pos]
};
declare function seq:count($seq as node()) as node() {
return count($seq/entry[$pos])
};
declare function seq:deref($seq as node()) as node() {
let $r_seq := for $entry in $seq return xs:string($entry)
return $r_seq
};
Note that I'm simplifying shamelessly here - you would be looking at some mechanism to identify and encode existing type for each entry, bounds checking, schema validation and possibly recursion if a given encapsulated sequence contained another encapsulated sequence. However, this is for illustration purposes only.
The point to consider here is the this is an object description - an instantiator (or parser), an accessor, a serializer. As you indicate, the names are less relevant than the larger issue that this is an object, and as an object it will be treated in different ways than a library module of utility functions (global static methods) would.
<xsl:copy-of select="seq:deref(seq:ref((seq:ref(("a","b")),seq:ref(("c","d"))))"/>
This would actually create two XML entities as a sequence (<sequence><entry>a</entry><entry>b</entry></sequence>,<sequence><entry>c</entry><entry>d</entry><sequence>) (removing the namespace info), in this implementation, but that's largely unimportant - it could readily be assigned as a neutral item() in another.
The question I would have is whether this modality should be encouraged within EX*, or not. If so, then it suggests that we need to think about the mechanics of such a system and how they could be made consistent across platforms - otherwise, we continue to punt and run into a problem down the road when we have half a dozen different object (inconsistent) instantiation systems. This is why I bring it up.
Kurt
Dimitre Novatchev
>
> Admittedly - the problem with the discussion here is that the inability to
> create variables in XPath means that any entity that IS created has to be
> anonymous:
>
> (for $myNum in (1 to 1000)[. mod 2 =0] return $myNum)
>
> creates two sequences (the entire sequence and (1 to 1000)), but because you
> can't create variables, they are simply expressions. However, there are
> still operations that can be defined upon the expressions overall.
Umm... it may be simpler, just:
(1 to 1000)[. mod 2 =0]
> The point I'm making is not in the functional refs but in the
> assumption that these are in a distinct namespace. That is to say, whether
> using new(), create(), instantiate(), ref() or whatever, there is a
> difference between:
>
> let $ref1 := ns1:ref()
>
> and
>
> let $ref1 := ns2:ref()
>
> Once you create the idea of a core object class that is tied to a given
> namespace implementation,
There is nothing that makes us associate a namespace to an object.
Haskell has its modules and a module is like a namespace. If you
import tow modules that have functions with the same name you need to
specify some disambiguation (isn't this exactly so with the XQuery
modules?).
This still doesn't make a module object.
>
> The problem that we face with XQuery/XPath modularization is that it IS
> conceptually possible to use an OOP like construction, the question is
> whether it is advisable.
As explained above, this is not "OOP-like", but just a good
modularization practice.
> The point to consider here is the this is an object description - an
> instantiator (or parser), an accessor, a serializer. As you indicate, the
> names are less relevant than the larger issue that this is an object, and as
> an object it will be treated in different ways than a library module of
> utility functions (global static methods) would.
You may *think* of it as an object description, but in fact it isn't.
People, who don't know about OOP wouldn't agree to this being a
specification of an object and wouldn't care, or worse may be
confused.
They will certainly point out this as a good modularization practice.
It is good to pray to God, no matter that this God is different to
different people :)
>
> The question I would have is whether this modality should be encouraged
> within EX*, or not. If so, then it suggests that we need to think about the
> mechanics of such a system and how they could be made consistent across
> platforms - otherwise, we continue to punt and run into a problem down the
> road when we have half a dozen different object (inconsistent) instantiation
> systems. This is why I bring it up.
I strongly believe that EXPath will better serve its mission if it
embraces a number of good design practices, one of which is
modularity.Not preaching a single religion will also be a definite
plus.
Kurt Cagle
Umm... it may be simpler, just:
(1 to 1000)[. mod 2 =0]
It is. It's Sunday, I'm operating on about 4 hours of sleep, sometimes the obvious takes a little while to force its way through the brain.
There is nothing that makes us associate a namespace to an object.
Haskell has its modules and a module is like a namespace. If you
import tow modules that have functions with the same name you need to
specify some disambiguation (isn't this exactly so with the XQuery
modules?).
This still doesn't make a module object.
You may *think* of it as an object description, but in fact it isn't.
People, who don't know about OOP wouldn't agree to this being a
specification of an object and wouldn't care, or worse may be
confused.
They will certainly point out this as a good modularization practice.
It is good to pray to God, no matter that this God is different to
different people :)
Of course, if you're a Taoist ... :-)
Object programming in general is good modularization practice, save when it becomes so heavily encoded into the language that it allows for no other perspective. One of the reasons I'm not a big fan of Bjarne Stroustrup. However, for many, many people, the idea that you can in fact apply an object-like approach to XQuery development is new and consequently frightening. I'm additionally not sure whether it imposes too heavy a conceptual weight on XQueries as a consequence. I'd be interested in hearing other people's perspectives wrt that.
>> The question I would have is whether this modality should be encouragedI strongly believe that EXPath will better serve its mission if it
> within EX*, or not. If so, then it suggests that we need to think about the
> mechanics of such a system and how they could be made consistent across
> platforms - otherwise, we continue to punt and run into a problem down the
> road when we have half a dozen different object (inconsistent) instantiation
> systems. This is why I bring it up.
embraces a number of good design practices, one of which is
modularity.Not preaching a single religion will also be a definite
plus.
One can only hope.
Martin Probst
> conceptual weight on XQueries as a consequence. I'd be interested in hearing
> other people's perspectives wrt that.
I'd be very, very careful in doing things like these.
If you add nested sequences to the type system of XPath, you will have
a lot of specification work to do. How do all the built in operators
work on nested sequences (=, eq, ..)? How do they interact with the
standard function library? Of course you could always say "they don't,
and it's an error", but that might not be very useful then.
Same goes for object orientation. XQuery is a functional language, so
side effects are forbidden. And that is for a very good reason - if
you drop this rule, you loose lazy evaluation and statement
reordering, which are really nice and really important optimization
techniques. The sql:connect extension would for example leak
connections in xDB as lazy evaluation would prevent the disconnect
call from happening.
But object orientation is going to be hardly useful without objects
carrying around state. I know it's possible to have immutable objects
only, but 99% of your users will not expect it to work that way.
If you are extending the language this way, you better very well know
what you're doing. Otherwise, you might end up with a variant of Java
with a really ugly syntax and some special XML processing
capabilities.
I know that quite some people like to use XQuery as a general purpose
programming language, and of course that is where requests for nested
sequences and objects come from, but I'm really not sure if that is a
good idea. I can't quite see a way to extend the language to handle
these use cases without breaking it beyond repair in some areas.
Martin
James Fuller
I know that quite some people like to use XQuery as a general purpose
programming language, and of course that is where requests for nested
sequences and objects come from, but I'm really not sure if that is a
good idea. I can't quite see a way to extend the language to handle
these use cases without breaking it beyond repair in some areas.
I especially do not want to reformulate and embed OO concepts at expath or exquery level; but I do want to have 'just enough' added to the language to make it a pragmatic tool.
stepping back a bit, the real problem is that in web programming we have 3 domains in terms of scope/scale ...
I) the web: for 90% of the web a simple general purpose language on the server side just needs to do a few things and do them well. Dynamic languages means you can opt into a watered down version of OO or opt out ... Perl gets this right by allowing just about any computing approach (imperative, functional, OO, etc...) but thats my own personal opinion.
II) enterpise: at the enterprise level, OO rules because, well we need to enable hundreds of developers to have some useful output and I will admit things like Java and its embedded OO notion is one way to go. A lot of specifications and software get a bias towards enterprise level requirements because the larger standards orgs have representation whose interests are firmly at this scale. The funny thing I have found though is that most of the effective work at the enterprise scale gets done by teams of less then 5 senior people ... the hundreds tend to work on the periphery as 'nurses' to the chief surgeons. Sometimes I think there is a bit of 'chinese wall' building but in general I can see how OO promises to enable more output with more developers but I do think our toolchain, specs, and technologies are bent towards doing the will of the 'few' versus the 'many'.
III) planet wide: at this scale we see that people are gaining most success with simple algorithms ... mapreduce, etc .... it will be interesting to see what computing paradigm emerges with virtualisation/parallel computing.
Out of the above 3 domains, there are many many more of the I) type and thats what interests me and what I see as problematic with standards/specification processes mainly being pitched at II) type scenarios.
Getting back to the point, allowing references would enable nested sequences and we could make it very straightforward in that a reference is just a number and existing type; then provide a function to work on that number and retrieve data associated with it ... this is simple and would require no tinkering at any deep level within any existing spec.
The reasons why I/we are drifting towards considering higher order abstractions with XQuery is that solving problems by functional decomposition is difficult to manage over time, especially if reuse mechanisms are lacking as they are in the original XQuery spec. If we had modules a bit more useful then things would be fine ... but thats a discussion I will bring over to EXQuery.
cheers, Jim Fuller
Martin
Martin Probst
> I especially do not want to reformulate and embed OO concepts at expath or
> exquery level; but I do want to have 'just enough' added to the language to
> make it a pragmatic tool.
My problem with a pragmatic tool is that pragmatism tends to break
certain kinds of scrutiny, and you might well end up with designing
language extensions that are extremely pragmatic, but totally break
certain use cases.
One of these use cases are certain optimizations, and that is what
XQuery was designed to do well in the first place. It's already a hell
to optimize currently, I'd rather not see more complexity added there.
I'm particularly wary of the side-effects adding things. They are
indeed very pragmatic, but IMHO totally break the language in very
important aspects. They do work in one certain implementation, but
that's clearly not enough, at least to me.
About application use cases: there are many interesting domains, and
certainly you can achieve a lot with XQuery. But I'd rather have a
programming environment that lets me fall back to Java (Python, Ruby,
...) or something else when necessary than shoe-horning lots of
features into XQuery.
It's very easy to break a language beyond any recognition by
pragmatically adding features - see PHP for an example. We shouldn't
do that. I'm all in favor of adding small things here and there that
enable certain well defined use cases (like HOF or dynamic function
invocation, better HTTP access and so on), but IMHO we should restrain
from significantly changing the language.
Martin
James Fuller
CC'ing exquery, even though I guess most people are on both lists anyways?
My problem with a pragmatic tool is that pragmatism tends to break
> I especially do not want to reformulate and embed OO concepts at expath or
> exquery level; but I do want to have 'just enough' added to the language to
> make it a pragmatic tool.
certain kinds of scrutiny, and you might well end up with designing
language extensions that are extremely pragmatic, but totally break
certain use cases.
yes, but at the end of the day we need to get work done; we will soon be getting to the cathedral/marketplace permatopic done this line ... instead of upfront design lets let community do what they *want*.
One of these use cases are certain optimizations, and that is what
XQuery was designed to do well in the first place. It's already a hell
to optimize currently, I'd rather not see more complexity added there.
I'm particularly wary of the side-effects adding things. They are
indeed very pragmatic, but IMHO totally break the language in very
important aspects. They do work in one certain implementation, but
that's clearly not enough, at least to me.
I think you and I are in agreement, but perhaps you are looking via imple pov ... there will always be an option to not implement possibly contentious modules, which I wouldn't consider in core list.
What I don't think expath/exquery should do is make decisions against lets say having someone link XQuery up to a coffee machine ... lets embrace the community let them do whatever they want ... as long as we only allow modules in the core that are not issues then we avoid the issue of debating contentious things only at the point we have something to debate.
About application use cases: there are many interesting domains, and
certainly you can achieve a lot with XQuery. But I'd rather have a
programming environment that lets me fall back to Java (Python, Ruby,
...) or something else when necessary than shoe-horning lots of
features into XQuery.
I am not suggesting to shoe horn anything, I am discussing procedure.
It's very easy to break a language beyond any recognition by
pragmatically adding features - see PHP for an example. We shouldn't
completely agree with you,
do that. I'm all in favor of adding small things here and there that
enable certain well defined use cases (like HOF or dynamic function
invocation, better HTTP access and so on), but IMHO we should restrain
from significantly changing the language.
I propose the same outcome but a laissez faire approach... I would rather allow everything and let social democracy dictate if;
a) a human steps up to come forward to define a module and then an implementation becomes available, all choices down to said humans doing the work
*and*
b) expath/exquery members review and vote for inclusion inside of core expath/exquery at that point in time
lets focus on modules that we can agree on now ...
Math module ? anything else ?
IMO expath core should look like the tip of an iceberg, where the visible modules are core and there being an active community underneath cooking up ideas along any manner of their choosing ... lets not stifle anything by presupposing now anything (past quality and technical correctness) and just provide infrastructure, metadata, and simple procedures.
cheers, Jim
Florent Georges
>> This is an item(), but neither an atomic type nor a node(). Exactly
>> as the proposed function() item type. So the exact type is a black
>> box (and in plain, standard XPath 2.0 can only be represented as
>> item() in an ItemType:
> Ouch - It would be my preference to avoid the use of Item at all
> possible costs.
I understand this concern. But on the other hand, that's really on
purpose here. We do not want to impose more restriction on this type
(like "it must be a node()," or "it must be a xs:string.") We just
want *exactly one* item. And one cannot do anything else than passing
it to some other functions of this module.
If we were defining a new version of XPath, we could add a new type,
sibling to node(). And provide an ItemType as "sequence-ref()" or
"nested-sequence()." But to only introduce a set of extension
functions, without changing the grammar, we can just stay with
"item()."
Furthermore, nested sequences will typically used to build
heterogeneous sequences (where the SequenceType can not really be more
precise than "item()+".) When you have a sequence of strings and
integers, you cannot precise that fact, only say that you can have
several items: item()+.
Of course, it must be carefully designed to be sure standard F&O are
properly handled (you could do almost nothing with this type excepted
with the functions explicitly designed to deal with it.)
The goal is to enable building complex data structures. But then,
this is the responsibility of the developer to explicitly handle
conversions between those structures and the traditional sequences of
XDM.
Florent Georges
> XPath and all its implementations already have the reference type, but
> they are not speaking about it openly.
I am a bit surprised to see this affirmation on a quite regular
basis. In my very humble opinion, XPath does not have anything like
this. Indeed, implementations will not copy all sequence's values if
it is used at several places, but this is just an implementation
"detail." Sequences are defined by value, and do not have any
identity.
But I might be wrong.
> Michael Kay calls the reference withs its proper name in a number of
> places in his books.
Do you have any precise reference (well, I mean pointer, well, you know :-p) ?
Florent Georges
>> The problem that we face with XQuery/XPath modularization is
>> that it IS conceptually possible to use an OOP like
>> construction, the question is whether it is advisable.
> As explained above, this is not "OOP-like", but just a good
> modularization practice.
Yes, I agree with Dimitre. IMHO this is "just" encapsulation.
While this is one very important principle of object orientation,
this is not the only one, and this does not make something object
oriented by itself.
And I do think encapsulation is very valuable in X*, and that's
one reason I'd really like to have the ability to nest sequences:
to be able to define complex data structures, encapsulated by
function libraries.
Florent Georges
> If you add nested sequences to the type system of XPath, you
> will have a lot of specification work to do. How do all the
> built in operators work on nested sequences (=, eq, ..)? How do
> they interact with the standard function library? Of course you
> could always say "they don't, and it's an error", but that
> might not be very useful then.
Funny, I do think this is exactly how it would be useful.
IMHO the goal is not to define black magic rules to decide
automagically when a sequence has to be nested or unnested, but
instead to provide a way to explicitly nest or unnest sequences.
In almost any other cases, an error should be thrown: with value
comparison operators (maybe also with general comparison ops, or
just return false,) with fn:data() and fn:string(), if tried to
be added to a tree, etc.
That would guarantee to not have any silent, unwanted result:
either the handling is explicit, or we get an error.
Dimitre Novatchev
>> places in his books.
>
> Do you have any precise reference (well, I mean pointer, well, you know :-p) ?
probably remember well, in the EXSLT list where I provided a number of
citations (including page numbers) from a number of books by Dr. Kay.
AFAIK his statements to the same effect can be found in his
lists/newsgroups replies, too.
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Florent Georges
>>> Michael Kay calls the reference withs its proper name in a
>>> number of places in his books.
>> Do you have any precise reference (well, I mean pointer, well,
>> you know :-p) ?
> There was another discussion on this topic not long ago, which
> you probably remember well, in the EXSLT list where I provided
> a number of citations (including page numbers) from a number of
> books by Dr. Kay. AFAIK his statements to the same effect can
> be found in his lists/newsgroups replies, too.
Thanks. I found for instance the following from EXSLT list
http://lists.fourthought.com/pipermail/exslt/2008-December/001782.html:
|> The <xsl:sequence> instruction ... is the only XSLT
|> instruction (with the exception of <xsl:perform-sort>) that
|> can return references to existing nodes, as distinct from
|> newly constructed nodes.
Well, in this case, I would say the important words are "as
distinct from newly constructed nodes," using "references" is
mainly to contrast with that. You can equally say "that can
return existing nodes..." I am not sure this is enough to say
"XPath has the reference type."
Nodes are added to sequences by identity (aka by reference) but
I don't think that's the same thing as having a reference type.
Regards,
Dimitre Novatchev
> |> instruction (with the exception of <xsl:perform-sort>) that
> |> can return references to existing nodes, as distinct from
> |> newly constructed nodes.
>
> Well, in this case, I would say the important words are "as
> distinct from newly constructed nodes," using "references" is
> mainly to contrast with that. You can equally say "that can
> return existing nodes..." I am not sure this is enough to say
> "XPath has the reference type."
it is not so important how you (or I) feel about that, what is
important is *the fact* that references are *de facto* already there
in XPath.
We may curve and force our tongues not to say the "dirty word", but
this will not change reality. :)
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
Florent Georges
> it is not so important how you (or I) feel about that, what is
> important is *the fact* that references are *de facto* already there
> in XPath.
Sure. You just haven't convinced me yet about that fact :-)
Actually, I simply don't see the point to say "there is a reference
type in XPath; it is not defined in the spec, you cannot use it, but
it is there." Or I completely misunderstood you.
Dimitre Novatchev
>
> 2009/4/20 Dimitre Novatchev wrote:
>
>> it is not so important how you (or I) feel about that, what is
>> important is *the fact* that references are *de facto* already there
>> in XPath.
>
> Sure. You just haven't convinced me yet about that fact :-)
>
> Actually, I simply don't see the point to say "there is a reference
> type in XPath; it is not defined in the spec,
> you cannot use it, but
A single node cannot be in two different items, at two different
positions, at the same time, but its references can.
One could prefer to regard a sequence that contains a given node
referenced more than once, as something containing "black holes", only
in order to avoid using the word "reference": this is their
problem.
If you prefer to use the term "black holes" instead of "references",
we could still have a function:
black-hole(node)
and
de-black-hole(black-hole)
Or, to use less extravagant terminology, we could most likely agree
that a sequence that contains a node has "something that uniquely
identifies the node". It is difficult to disagree with this fact,
doing so will contradict the capability of a sequence to select that
node at that particular position.
So, we can capture this "something that uniquely identifies the node"
in a function:
something-that-uniquely-identifies-the-node(node)
and
de-something-that-uniquely-identifies-the-node(something-that-uniquely-identifies-the-node)
We could use any other words that are of your liking and I promise
never to write any more the word "***erence".
--
Cheers,
Dimitre Novatchev
---------------------------------------
Truly great madness cannot be achieved without significant intelligence.
---------------------------------------
To invent, you need a good imagination and a pile of junk
-------------------------------------
Never fight an inanimate object
-------------------------------------
You've achieved success in your field when you don't know whether what
you're doing is work or play
>
Florent Georges
> A single node cannot be in two different items, at two different
> positions, at the same time, but its references can.
That's not what I understand from
http://www.w3.org/TR/xpath-datamodel/#sequences. Maybe, if we
continue this discussion, we should do so on XSL List?
Adam Retter
> a lot of specification work to do. How do all the built in operators
> work on nested sequences (=, eq, ..)? How do they interact with the
> standard function library? Of course you could always say "they don't,
> and it's an error", but that might not be very useful then.
Personally, I think true nested sequences should be the responsibility
of the W3C WG. However, if someone wishes to add an extension module
specification to EXPath in the mean time for something that imitates
the functionality of a nested sequence then they are more than
welcome. Please note that I said 'extension module', in my opinion
unofficial extensions to the XPath language itself will prove too
costly.
> Same goes for object orientation. XQuery is a functional language, so
> side effects are forbidden. And that is for a very good reason - if
> you drop this rule, you loose lazy evaluation and statement
> reordering, which are really nice and really important optimization
> techniques. The sql:connect extension would for example leak
> connections in xDB as lazy evaluation would prevent the disconnect
> call from happening.
I have no interest in seeing Object Orientation added. I completely
agree that side-effects in XQuery are a bad idea from the point of
lazy evaluation and optimization and I will always try and avoid such
things if possible.
However this is where I will start to diverge from your view - These
are extension modules for doing things that the language never
intended. People want these additions in my experience, which is what
has led me to this point today. If I need to provide some
functionality that can only be achieved with side effects, then I will
add that functionality. The assumption that I make is that it is only
where I have this functionality that the implementation will loose
optimal performance - is that a correct assumption? If so then that is
a price I am happy with.
With regards the sql:connect() example I used, perhaps the example
could be improved with regards to side-effects, it is not actually the
sql module that I implemented for eXist. The sql module in eXist has
no close() function, it automatically cleans up open connections when
the XQuery finishes execution - more side effects ;-p
sql:execute($connectionString as xs:string, $username as xs:string,
$password as xs:string, $sql as xs:string) as element(sql:results)
Perhaps that would be more side-effect free? It would then be upto the
implementation whether to do that as an atomic side-effect free
operation, or whether to perhaps underneath the hood use some sort of
sql connection pooling etc.
> If you are extending the language this way, you better very well know
> what you're doing. Otherwise, you might end up with a variant of Java
> with a really ugly syntax and some special XML processing
> capabilities.
Yuck!
> I know that quite some people like to use XQuery as a general purpose
> programming language, and of course that is where requests for nested
> sequences and objects come from, but I'm really not sure if that is a
> good idea. I can't quite see a way to extend the language to handle
> these use cases without breaking it beyond repair in some areas.
I do a lot of web application development in XQuery, personally I
havent wanted nested sequences or objects yet. But isnt this about
XPath?
Florent Georges
> sql:execute($connectionString as xs:string, $username as xs:string,
> $password as xs:string, $sql as xs:string) as element(sql:results)
> Perhaps that would be more side-effect free?
Well, a SQL extension shouldn't be totally free of side-effect ;-)