Hi! As a part of my Masters Thesis, I am trying to build a formal model for YAML so that parsers can use it as a base for furthering their work.
I was wondering whether you could tell me what model/grammar has been used to develop the PyYAML parser.
-- Regards,
Bodhisatta Barman Roy International Institute of Information Technology Bangalore
--------------------------------------------------------------------------- --- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2
You can look at the YamlReference Haskell implementation, and of course the spec it self is in BNF (well, with some attributes thrown in for good measure).
Have fun,
Oren Ben-Kiki
On Fri, Jun 3, 2011 at 1:38 PM, Bodhi <bodhisatta.barman...@iiitb.net>wrote:
> Hi! > As a part of my Masters Thesis, I am trying to build a formal model for > YAML so that parsers can use it as a base for furthering their work.
> I was wondering whether you could tell me what model/grammar has been used > to develop the PyYAML parser.
> -- > Regards,
> Bodhisatta Barman Roy > International Institute of Information Technology > Bangalore
> --------------------------------------------------------------------------- --- > Simplify data backup and recovery for your virtual environment with > vRanger. > Installation's a snap, and flexible recovery options mean your data is > safe, > secure and there when you need it. Discover what all the cheering's about. > Get your free trial download today. > http://p.sf.net/sfu/quest-dev2dev2 > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2
On Fri, Jun 3, 2011 at 5:54 PM, Oren Ben-Kiki <o...@ben-kiki.org> wrote: > You can look at the YamlReference Haskell implementation, and of course the > spec it self is in BNF (well, with some attributes thrown in for good > measure).
> Have fun,
> Oren Ben-Kiki
> On Fri, Jun 3, 2011 at 1:38 PM, Bodhi <bodhisatta.barman...@iiitb.net>wrote:
>> Hi! >> As a part of my Masters Thesis, I am trying to build a formal model for >> YAML so that parsers can use it as a base for furthering their work.
>> I was wondering whether you could tell me what model/grammar has been used >> to develop the PyYAML parser.
>> -- >> Regards,
>> Bodhisatta Barman Roy >> International Institute of Information Technology >> Bangalore
>> --------------------------------------------------------------------------- --- >> Simplify data backup and recovery for your virtual environment with >> vRanger. >> Installation's a snap, and flexible recovery options mean your data is >> safe, >> secure and there when you need it. Discover what all the cheering's about. >> Get your free trial download today. >> http://p.sf.net/sfu/quest-dev2dev2 >> _______________________________________________ >> Yaml-core mailing list >> Yaml-c...@lists.sourceforge.net >> https://lists.sourceforge.net/lists/listinfo/yaml-core
-- Regards,
Bodhisatta Barman Roy International Institute of Information Technology Bangalore
--------------------------------------------------------------------------- --- Simplify data backup and recovery for your virtual environment with vRanger. Installation's a snap, and flexible recovery options mean your data is safe, secure and there when you need it. Discover what all the cheering's about. Get your free trial download today. http://p.sf.net/sfu/quest-dev2dev2
So ... I've only recently joined and been tracking this list, so please excuse what might be off-topic.
I'm working on a security standard and trying to use YAML. However, I need a schema - not to enforce a schema on YAML, but primarily to be able to describe in YAML subsets of a possible schema.
The standard is focused on describing "who can do what". Who is cryptographic, what can be objects describable in YAML. Delegation requires the ability to define set of information within a schema.
I've looked at Doctrine - it's close but not quite what I need. Are there other efforts in place that could be leveraged?
I'm also looking at stronger typing and binary encoding (akin to Protobuf). The use of schemas, strong types and alternate encodings may be contrary to YAML culture and goals (of staying simple). Is YAML the right choice in this context or should I just start something new that is YAML-like?
Thanks in advance,
Paul
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
There hasn't been much work on YAML schemas. There is an issue here that people mean different things when they say "schema" and "validation".
If you are looking for validating your specific input files in your specific application, YAML is very well suited for that. A minimal amount of !tag-ing combined with some implicit tagging rules, plus adding verification code in the matching classes in your favorite implementation language, and you are all set for as strong (or as weak) a verification as you want. This is what most people do.
If you are looking for a generic "validate a data file with a schema file" ability. There's no such thing for YAML, perhaps because it isn't that useful in practice after all...
The same arguments can be made about JSON, except there you are completely at the mercy of "implicit" tagging, and the whole notion of deserialization into your application classes is an afterthought rather than being an inherent part of the spec.
Binary encoding... sorry. YAML is first and foremost about readability. Sure, you can produce pretty dense (and unreadable) YAML files using the flow styles (think JSON but with the !tag-ing, such as [!foo{bar: baz}]. That's not _too_ bad size-wise, as long as all your data is textual anyway - and you can zip it for additional shrinkage. If you have true binary data (e.g., a PNG image), you'd need to base-64 it, and zipping the result would only help so far... YAML really wasn't designed for this sort of thing.
On Fri, Jun 10, 2011 at 9:52 PM, Paul Lambert <p...@marvell.com> wrote: > So … I’ve only recently joined and been tracking this list, so please > excuse what might be off-topic.
> I’m working on a security standard and trying to use YAML. However, I need > a schema – not to enforce a schema on YAML, but primarily to be able to > describe in YAML subsets of a possible schema.
> The standard is focused on describing “who can do what”. Who is > cryptographic, what can be objects describable in YAML. Delegation requires > the ability to define set of information within a schema.
> I’ve looked at Doctrine – it’s close but not quite what I need. Are there > other efforts in place that could be leveraged?
> I’m also looking at stronger typing and binary encoding (akin to > Protobuf). The use of schemas, strong types and alternate encodings may be > contrary to YAML culture and goals (of staying simple). Is YAML the right > choice in this context or should I just start something new that is > YAML-like?
> Thanks in advance,
> Paul
> --------------------------------------------------------------------------- --- > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
> Binary encoding... sorry. YAML is first and foremost about readability.
The idea is to have both canonical human readable and binary encodings. It's pretty straight forward to define a TLV approach that simply takes a tag and converts it to an enumerated integer. YAML as it stands could have such an efficient encoding with a suitable definition of ways to do tag mapping. A Schema could help define the tag to integer mapping (ASN.1 or Protobuf as examples).
I'll sent a sample schema to the list as it gets put together ...
Paul
From: Oren Ben-Kiki [mailto:o...@ben-kiki.org] Sent: Friday, June 10, 2011 12:30 PM To: Paul Lambert Cc: yaml-c...@lists.sourceforge.net Subject: Re: [Yaml-core] Reg: PyYAML grammar
There hasn't been much work on YAML schemas. There is an issue here that people mean different things when they say "schema" and "validation".
If you are looking for validating your specific input files in your specific application, YAML is very well suited for that. A minimal amount of !tag-ing combined with some implicit tagging rules, plus adding verification code in the matching classes in your favorite implementation language, and you are all set for as strong (or as weak) a verification as you want. This is what most people do.
If you are looking for a generic "validate a data file with a schema file" ability. There's no such thing for YAML, perhaps because it isn't that useful in practice after all...
The same arguments can be made about JSON, except there you are completely at the mercy of "implicit" tagging, and the whole notion of deserialization into your application classes is an afterthought rather than being an inherent part of the spec.
Binary encoding... sorry. YAML is first and foremost about readability. Sure, you can produce pretty dense (and unreadable) YAML files using the flow styles (think JSON but with the !tag-ing, such as [!foo{bar: baz}]. That's not _too_ bad size-wise, as long as all your data is textual anyway - and you can zip it for additional shrinkage. If you have true binary data (e.g., a PNG image), you'd need to base-64 it, and zipping the result would only help so far... YAML really wasn't designed for this sort of thing.
Have fun,
Oren Ben-Kiki
On Fri, Jun 10, 2011 at 9:52 PM, Paul Lambert <p...@marvell.com<mailto:p...@marvell.com>> wrote:
So ... I've only recently joined and been tracking this list, so please excuse what might be off-topic.
I'm working on a security standard and trying to use YAML. However, I need a schema - not to enforce a schema on YAML, but primarily to be able to describe in YAML subsets of a possible schema.
The standard is focused on describing "who can do what". Who is cryptographic, what can be objects describable in YAML. Delegation requires the ability to define set of information within a schema.
I've looked at Doctrine - it's close but not quite what I need. Are there other efforts in place that could be leveraged?
I'm also looking at stronger typing and binary encoding (akin to Protobuf). The use of schemas, strong types and alternate encodings may be contrary to YAML culture and goals (of staying simple). Is YAML the right choice in this context or should I just start something new that is YAML-like?
Thanks in advance,
Paul
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net<mailto:Yaml-c...@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
Sure you could define a while new binary format, but I doubt you will get significant savings compared to a flow style (other than for stuff like embedded PNG images). After all it requires just 1 character at the start and at the end of each collection ('{' ... '}'), one character to separate collection entries (',') and so on.
I suppose it is possible to define a minor extension to the flow style - say, DLE <size of binary data> <binary data> - which would be "equivalent" to specifying a base64 string. This would require minimal tweaking of existing YAML parsers and would offer very dense encoding of the overall stream (throw in zipping if !tags are an issue).
This might be an interesting idea to keep in mind for future YAML spec versions. It wouldn't be anything we will do unless someone actually used it first, though ;-)
On Fri, Jun 10, 2011 at 11:13 PM, Paul Lambert <p...@marvell.com> wrote: > Thanks for the help …
> > Binary encoding... sorry. YAML is first and foremost about readability.
> The idea is to have both canonical human readable and binary encodings. > It’s pretty straight forward to define a TLV approach that simply takes a > tag and converts it to an enumerated integer. YAML as it stands could have > such an efficient encoding with a suitable definition of ways to do tag > mapping. A Schema could help define the tag to integer mapping (ASN.1 or > Protobuf as examples).
> I’ll sent a sample schema to the list as it gets put together …
> There hasn't been much work on YAML schemas. There is an issue here that > people mean different things when they say "schema" and "validation".
> If you are looking for validating your specific input files in your > specific application, YAML is very well suited for that. A minimal amount of > !tag-ing combined with some implicit tagging rules, plus adding verification > code in the matching classes in your favorite implementation language, and > you are all set for as strong (or as weak) a verification as you want. This > is what most people do.
> If you are looking for a generic "validate a data file with a schema file" > ability. There's no such thing for YAML, perhaps because it isn't that > useful in practice after all...
> The same arguments can be made about JSON, except there you are completely > at the mercy of "implicit" tagging, and the whole notion of deserialization > into your application classes is an afterthought rather than being an > inherent part of the spec.
> Binary encoding... sorry. YAML is first and foremost about readability. > Sure, you can produce pretty dense (and unreadable) YAML files using the > flow styles (think JSON but with the !tag-ing, such as [!foo{bar: baz}]. > That's not _too_ bad size-wise, as long as all your data is textual anyway - > and you can zip it for additional shrinkage. If you have true binary data > (e.g., a PNG image), you'd need to base-64 it, and zipping the result would > only help so far... YAML really wasn't designed for this sort of thing.
> Have fun,
> Oren Ben-Kiki
> On Fri, Jun 10, 2011 at 9:52 PM, Paul Lambert <p...@marvell.com> wrote:
> So … I’ve only recently joined and been tracking this list, so please > excuse what might be off-topic.
> I’m working on a security standard and trying to use YAML. However, I need > a schema – not to enforce a schema on YAML, but primarily to be able to > describe in YAML subsets of a possible schema.
> The standard is focused on describing “who can do what”. Who is > cryptographic, what can be objects describable in YAML. Delegation requires > the ability to define set of information within a schema.
> I’ve looked at Doctrine – it’s close but not quite what I need. Are there > other efforts in place that could be leveraged?
> I’m also looking at stronger typing and binary encoding (akin to > Protobuf). The use of schemas, strong types and alternate encodings may be > contrary to YAML culture and goals (of staying simple). Is YAML the right > choice in this context or should I just start something new that is > YAML-like?
> Thanks in advance,
> Paul
> --------------------------------------------------------------------------- --- > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On Sat, Jun 11, 2011 at 4:52 AM, Paul Lambert <p...@marvell.com> wrote: > So … I’ve only recently joined and been tracking this list, so please > excuse what might be off-topic.****
> ** **
> I’m working on a security standard and trying to use YAML. However, I need > a schema – not to enforce a schema on YAML, but primarily to be able to > describe in YAML subsets of a possible schema.****
> ** **
> The standard is focused on describing “who can do what”. Who is > cryptographic, what can be objects describable in YAML. Delegation requires > the ability to define set of information within a schema.****
> ** **
> I’ve looked at Doctrine – it’s close but not quite what I need. Are there > other efforts in place that could be leveraged?****
> ** **
> I’m also looking at stronger typing and binary encoding (akin to > Protobuf). The use of schemas, strong types and alternate encodings may be > contrary to YAML culture and goals (of staying simple). Is YAML the right > choice in this context or should I just start something new that is > YAML-like?****
> ** **
> Thanks in advance,****
> ** **
> Paul****
> ** **
> ** **
> ** **
> --------------------------------------------------------------------------- --- > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
To encode this example - I'm looking at a couple of extensions:
1) You need a mechanism to bind a integer Type to the tag (macAddress, eg. 5)
2) You need stronger typing of the values with both binary and text encoding rules for each type. For example a macAddress would be defined as 6 hex bytes for binary encoding and a dash separated hex encoding for the human readable version (17 characters).
Using these rules the examples encoding would be:
x0506123456789AB (total length 8 bytes)
So in this example you get about a factor of 4 size reduction. The notion would be that the human readable and the binary would be able to be directly converted back and forth ... as long as you have a few schema/typing hints for the conversation that include the tagging values and associated typing that goes with the tag. String oriented types would have less benefit for size reduction. Enumerations would typically have larger reductions (but are a more complicated example).
The strong typing is important. I'm doing this work for a security oriented standard and need to have clarity in the usage of the values that is different than the usually YAML simplicity. For example even though a DNS name is a string ... the semantics and valid processing for a DNS name are much more restricted that a basic string and a very different definition of containment. For delegation there needs to be a very strict interpretation of the containment of one string/name in another range. So - these concepts need to be built into the system.
I'd like to embed the hints for schema and tagging into a YAML compatible format. For example - a simple line of schema for the above could be:
macAddress: <macAddressType> # [5]
There would be a fair number of standard prescribed types for interesting objects (date/time, dnsName, ipAddress, string, int, enum, etc.) and ability to extend and add types.
I'm still futzing with the style of representing the schema information - hence my strong interest in existing YAML schema work. However, I also need to overload any schema language with the tagging information .. so some invention is still required. My intent is to do the overloading in comments so any schema is still a correct YAML document.
Paul
Paul A. Lambert | Marvell | +1 650 787 9141
From: Oren Ben-Kiki [mailto:o...@ben-kiki.org] Sent: Friday, June 10, 2011 10:42 PM To: Paul Lambert Cc: yaml-c...@lists.sourceforge.net Subject: Re: [Yaml-core] Reg: PyYAML grammar
Sure you could define a while new binary format, but I doubt you will get significant savings compared to a flow style (other than for stuff like embedded PNG images). After all it requires just 1 character at the start and at the end of each collection ('{' ... '}'), one character to separate collection entries (',') and so on.
I suppose it is possible to define a minor extension to the flow style - say, DLE <size of binary data> <binary data> - which would be "equivalent" to specifying a base64 string. This would require minimal tweaking of existing YAML parsers and would offer very dense encoding of the overall stream (throw in zipping if !tags are an issue).
This might be an interesting idea to keep in mind for future YAML spec versions. It wouldn't be anything we will do unless someone actually used it first, though ;-)
Have fun,
Oren Ben-Kiki
On Fri, Jun 10, 2011 at 11:13 PM, Paul Lambert <p...@marvell.com<mailto:p...@marvell.com>> wrote:
Thanks for the help ...
> Binary encoding... sorry. YAML is first and foremost about readability.
The idea is to have both canonical human readable and binary encodings. It's pretty straight forward to define a TLV approach that simply takes a tag and converts it to an enumerated integer. YAML as it stands could have such an efficient encoding with a suitable definition of ways to do tag mapping. A Schema could help define the tag to integer mapping (ASN.1 or Protobuf as examples).
I'll sent a sample schema to the list as it gets put together ...
Paul
From: Oren Ben-Kiki [mailto:o...@ben-kiki.org<mailto:o...@ben-kiki.org>] Sent: Friday, June 10, 2011 12:30 PM To: Paul Lambert Cc: yaml-c...@lists.sourceforge.net<mailto:yaml-c...@lists.sourceforge.net> Subject: Re: [Yaml-core] Reg: PyYAML grammar
There hasn't been much work on YAML schemas. There is an issue here that people mean different things when they say "schema" and "validation".
If you are looking for validating your specific input files in your specific application, YAML is very well suited for that. A minimal amount of !tag-ing combined with some implicit tagging rules, plus adding verification code in the matching classes in your favorite implementation language, and you are all set for as strong (or as weak) a verification as you want. This is what most people do.
If you are looking for a generic "validate a data file with a schema file" ability. There's no such thing for YAML, perhaps because it isn't that useful in practice after all...
The same arguments can be made about JSON, except there you are completely at the mercy of "implicit" tagging, and the whole notion of deserialization into your application classes is an afterthought rather than being an inherent part of the spec.
Binary encoding... sorry. YAML is first and foremost about readability. Sure, you can produce pretty dense (and unreadable) YAML files using the flow styles (think JSON but with the !tag-ing, such as [!foo{bar: baz}]. That's not _too_ bad size-wise, as long as all your data is textual anyway - and you can zip it for additional shrinkage. If you have true binary data (e.g., a PNG image), you'd need to base-64 it, and zipping the result would only help so far... YAML really wasn't designed for this sort of thing.
Have fun,
Oren Ben-Kiki
On Fri, Jun 10, 2011 at 9:52 PM, Paul Lambert <p...@marvell.com<mailto:p...@marvell.com>> wrote:
So ... I've only recently joined and been tracking this list, so please excuse what might be off-topic.
I'm working on a security standard and trying to use YAML. However, I need a schema - not to enforce a schema on YAML, but primarily to be able to describe in YAML subsets of a possible schema.
The standard is focused on describing "who can do what". Who is cryptographic, what can be objects describable in YAML. Delegation requires the ability to define set of information within a schema.
I've looked at Doctrine - it's close but not quite what I need. Are there other efforts in place that could be leveraged?
I'm also looking at stronger typing and binary encoding (akin to Protobuf). The use of schemas, strong types and alternate encodings may be contrary to YAML culture and goals (of staying simple). Is YAML the right choice in this context or should I just start something new that is YAML-like?
Thanks in advance,
Paul
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net<mailto:Yaml-c...@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On Mon, 13 Jun 2011 11:45:54 -0700, you wrote: > x0506123456789AB (total length 8 bytes)
This is precisely the kind of thing that YAML was designed to eliminate! Look at the top-priority goal of the YAML standard: "YAML is easily readable by humans." Readability trumps _everything_.
Bits are cheap. Unless you have a truly compelling reason to worry about the size of your data (e.g., you're working with an embedded CPU that has only a few kB of RAM), forget about this kind of compression. It will only come back to bite you in the butt later.
And if you really do have a reason to worry about the size of your data, you probably shouldn't be using YAML.
-Steve Schafer
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yaml-core
On Mon, Jun 13, 2011 at 10:39 PM, Steve Schafer <st...@fenestra.com> wrote: > On Mon, 13 Jun 2011 11:45:54 -0700, you wrote:
> > x0506123456789AB (total length 8 bytes)
> This is precisely the kind of thing that YAML was designed to eliminate! > Look at the top-priority goal of the YAML standard: "YAML is easily > readable by humans." Readability trumps _everything_.
Yup. This doesn't prevent people from defining all sort of strange binary formats, (which aren't YAML!)... In the example Paul gave, it seems like zip would efficiently encode what he called "tags" (actually, they are simply mapping keys). Well, at least if they are repeated more than once... Reducing the size of Mac addresses and so on would be less effective, but I still think that a simple zip (possibly combined with using a flow style) would make the vanilla YAML file "reasonably" competitive, size-wise, compared to a specially designed binary file format. By "reasonably" competitive I mean "less than twice as big" which, in this day and age, seems "good enough".
Have fun,
Oren Ben-Kiki
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
Ok - the YAML community and binary encoding do not appear to be compatible. Too bad your world view is so limited. Take a look a protobuf as an example of combining human readable and binary encodings if you want to see a worked hybrid.
The notion I was presenting was not that binary was the primary representation - but that the human readable and a binary encoding could coexist. Note that I'mm working with vendors building consumer devices, sensors, wireless protocols .. so the efficiency is quite important ... and zip is not a viable option.
> -----Original Message----- > From: Steve Schafer [mailto:st...@fenestra.com] > Sent: Monday, June 13, 2011 12:40 PM > To: Paul Lambert > Cc: yaml-c...@lists.sourceforge.net > Subject: Re: [Yaml-core] Reg: PyYAML grammar
> On Mon, 13 Jun 2011 11:45:54 -0700, you wrote:
> > x0506123456789AB (total length 8 bytes)
> This is precisely the kind of thing that YAML was designed to > eliminate! > Look at the top-priority goal of the YAML standard: "YAML is easily > readable by humans." Readability trumps _everything_.
> Bits are cheap. Unless you have a truly compelling reason to worry > about > the size of your data (e.g., you're working with an embedded CPU that > has only a few kB of RAM), forget about this kind of compression. It > will only come back to bite you in the butt later.
> And if you really do have a reason to worry about the size of your > data, > you probably shouldn't be using YAML.
> -Steve Schafer
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yaml-core
Thanks! Kwalify looks close .. as a starting point .... But I need better constraints. Paul
Paul A. Lambert | Marvell | +1 650 787 9141
From: Ingy dot Net [mailto:i...@ingy.net] Sent: Saturday, June 11, 2011 10:13 AM To: Paul Lambert Cc: yaml-c...@lists.sourceforge.net Subject: Re: [Yaml-core] Reg: PyYAML grammar
On Sat, Jun 11, 2011 at 4:52 AM, Paul Lambert <p...@marvell.com<mailto:p...@marvell.com>> wrote:
So ... I've only recently joined and been tracking this list, so please excuse what might be off-topic.
I'm working on a security standard and trying to use YAML. However, I need a schema - not to enforce a schema on YAML, but primarily to be able to describe in YAML subsets of a possible schema.
The standard is focused on describing "who can do what". Who is cryptographic, what can be objects describable in YAML. Delegation requires the ability to define set of information within a schema.
I've looked at Doctrine - it's close but not quite what I need. Are there other efforts in place that could be leveraged?
I'm also looking at stronger typing and binary encoding (akin to Protobuf). The use of schemas, strong types and alternate encodings may be contrary to YAML culture and goals (of staying simple). Is YAML the right choice in this context or should I just start something new that is YAML-like?
Thanks in advance,
Paul
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net<mailto:Yaml-c...@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
Again, I have nothing against people defining a YAML based binary format, and that may have advantages for some usages. It is just that it wouldn't be YAML :-) Protobuf is interesting and I always wondered if it were possible to define a "sensible" YAML <-> Protobuf converter.
Staying as close to YAML/JSON as possible, then using something like {dictionary:{&1 text1,&q1 text2},payload:{ *1: ...,*q1: ...}} you can reduce the size of repeated keys (and, for that matter, values) to a few characters, without resorting to zip. Doing something like the DLE extension I described you can embed binary blobs without base64 encoding. The result should be pretty compact and would have a trivial conversion to/from standard YAML.
That said, zip isn't a very expensive operation these days, even for consumer devices - of course I don't know what your usage is.
On Tue, Jun 14, 2011 at 3:20 AM, Paul Lambert <p...@marvell.com> wrote:
> Ok - the YAML community and binary encoding do not appear to be compatible. > Too bad your world view is so limited. Take a look a protobuf as an > example of combining human readable and binary encodings if you want to see > a worked hybrid.
> The notion I was presenting was not that binary was the primary > representation - but that the human readable and a binary encoding could > coexist. Note that I'mm working with vendors building consumer devices, > sensors, wireless protocols .. so the efficiency is quite important ... and > zip is not a viable option.
> Paul
> > -----Original Message----- > > From: Steve Schafer [mailto:st...@fenestra.com] > > Sent: Monday, June 13, 2011 12:40 PM > > To: Paul Lambert > > Cc: yaml-c...@lists.sourceforge.net > > Subject: Re: [Yaml-core] Reg: PyYAML grammar
> > On Mon, 13 Jun 2011 11:45:54 -0700, you wrote:
> > > x0506123456789AB (total length 8 bytes)
> > This is precisely the kind of thing that YAML was designed to > > eliminate! > > Look at the top-priority goal of the YAML standard: "YAML is easily > > readable by humans." Readability trumps _everything_.
> > Bits are cheap. Unless you have a truly compelling reason to worry > > about > > the size of your data (e.g., you're working with an embedded CPU that > > has only a few kB of RAM), forget about this kind of compression. It > > will only come back to bite you in the butt later.
> > And if you really do have a reason to worry about the size of your > > data, > > you probably shouldn't be using YAML.
> > -Steve Schafer
> --------------------------------------------------------------------------- --- > EditLive Enterprise is the world's most technically advanced content > authoring tool. Experience the power of Track Changes, Inline Image > Editing and ensure content is compliant with Accessibility Checking. > http://p.sf.net/sfu/ephox-dev2dev > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev
On Mon, 13 Jun 2011 17:20:45 -0700, you wrote: >Take a look a protobuf as an example of combining human readable and >binary encodings if you want to see a worked hybrid.
Protobuf and YAML are quite different things. They complement each other, but neither is a replacement for the other. The protobuf binary encoding is primarily an "over the wire" format; no one ever looks at it, except very rarely (during debugging). YAML is more of an endpoint format. As such, YAML -> protbuf ... protobuf -> YAML can be a perfectly reasonable thing to do.
The most successful computer/computer and computer/human protocols are heavily layered, with each layer having a well-defined set of responsibilities. Using the OSI model as the analogy, protobuf lives near the presentation/session (6/5) layer boundary, whereas YAML lives near the application/presentation (7/6) layer boundary.
-Steve Schafer
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yaml-core
Thanks Steve, I appreciate the dialog ... in the embedded forums I usually lurk YAML and human readable is a foreign concept.
Here's where the mini design stands:
Binary Encodable Schema in YAML (BESY) Binary Encoded YAML (BEY) - BESY is valid a YAML document - (YAML data + BESY schema for the data) enables conversion to/from BEY - (BEY data + BESY schema for the data) enables conversion to/from YAML - YAML data created with BESY may be subject to additional type checking and constraints - BEY encoding could be same as protobuf (exact encoding TDB or could be multiple)
One of my main goals is to allow a YAML-like syntax to be used in designing protocols for lower layer protocols and constrainedd devices. I suspect the TLV crowd will have a similar response to YAML as this forum has to binary encoding :-)
If/when the design reaches the next level I'll post a pointer to this list ...
> -----Original Message----- > From: Steve Schafer [mailto:st...@fenestra.com] > Sent: Tuesday, June 14, 2011 5:26 AM > To: Paul Lambert > Cc: yaml-c...@lists.sourceforge.net > Subject: Re: [Yaml-core] Reg: PyYAML grammar
> On Mon, 13 Jun 2011 17:20:45 -0700, you wrote:
> >Take a look a protobuf as an example of combining human readable and > >binary encodings if you want to see a worked hybrid.
> Protobuf and YAML are quite different things. They complement each > other, but neither is a replacement for the other. The protobuf binary > encoding is primarily an "over the wire" format; no one ever looks at > it, except very rarely (during debugging). YAML is more of an endpoint > format. As such, YAML -> protbuf ... protobuf -> YAML can be a > perfectly > reasonable thing to do.
> The most successful computer/computer and computer/human protocols are > heavily layered, with each layer having a well-defined set of > responsibilities. Using the OSI model as the analogy, protobuf lives > near the presentation/session (6/5) layer boundary, whereas YAML lives > near the application/presentation (7/6) layer boundary.
> -Steve Schafer
--------------------------------------------------------------------------- --- EditLive Enterprise is the world's most technically advanced content authoring tool. Experience the power of Track Changes, Inline Image Editing and ensure content is compliant with Accessibility Checking. http://p.sf.net/sfu/ephox-dev2dev _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yaml-core
On Jun 13, 8:20 pm, Paul Lambert <p...@marvell.com> wrote:
> Ok - the YAML community and binary encoding do not appear to be compatible. Too bad your world view is so limited. Take a look a protobuf as an example of combining human readable and binary encodings if you want to see a worked hybrid.
I don't understand. What's wrong with compression?
--------------------------------------------------------------------------- --- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/yaml-core
There's nothing wrong with whole-file/stream compression :-) But from what I understand, Paul is looking for something else, which is mucking with the internal structure, such as allowing for non-printable binary blobs.
BTW, YAML already provides a form of internal "compression". If you have many keys with some "long value foo", you can easily add an anchor to the 1st one and specify all the rest as "*f". Think of it as a poor man's LZW. IMO minimal-size flow-style YAML with heavy use of such anchors, plus some streaming compression of the stream, should give you a pretty compact format.
Of course I don't know Paul's application - he hinted that doing gzip may be too heavy for his use case (and I don't know what is data and BW constraints are). He's pretty much required to invent his own format which is tailored to his specific environment (or so it seems).
On Sun, Jun 26, 2011 at 5:20 AM, Trans <transf...@gmail.com> wrote:
> On Jun 13, 8:20 pm, Paul Lambert <p...@marvell.com> wrote: > > Ok - the YAML community and binary encoding do not appear to be > compatible. Too bad your world view is so limited. Take a look a protobuf > as an example of combining human readable and binary encodings if you want > to see a worked hybrid.
> I don't understand. What's wrong with compression?
> --------------------------------------------------------------------------- --- > All the data continuously generated in your IT infrastructure contains a > definitive record of customers, application performance, security > threats, fraudulent activity and more. Splunk takes this data and makes > sense of it. Business sense. IT sense. Common sense.. > http://p.sf.net/sfu/splunk-d2d-c1 > _______________________________________________ > Yaml-core mailing list > Yaml-c...@lists.sourceforge.net > https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1
> There's nothing wrong with whole-file/stream compression :-) But from what I >understand, Paul is looking for something else, which is mucking with the internal >structure, such as allowing for non-printable binary blobs.
Yes - I'm looking at a little more than just zip-like compression.
No - I'm hoping to not muck with the internal structures too much, but instead adding an alternate encoding. The main idea is that you can have both human readable (YAML text representation) and efficient binary encoding that are isomorphic. This adds a new representation type but it would be directly convertible to a readable YAML text format (assuming you have the encoding rules).
All data types would have both a canonical text and binary encoding defined. The human readable form as currently envisioned would be YAML - or at least a subset that conforms to stricter conventions, but that could be read and parsed by existing YAML libraries.
Validation would be possible using the encoding schema for both the human readable and binary versions.
The application space is small embedded devices, sensors, wireless protocols, etc. Some of the motivation comes from being tired of working in standards that are hard to read that use TLV formats and long prose to describe the protocol. I'm trying to get the most readable and efficient representation of a binary protocol.
The specific protocol I'm working on is for "privilege management" - basically defining who can do what. The expressions for these statements need to be human readable and understandable - YAMl would be ideal. Parameter validity checking is required (e.g. there are a bunch of security issues with dns names if you only treat the dns name as a string). TLV encoding and "efficient" representation of enumerated options is necessary for the embedded applications.
Paul
Paul A. Lambert | Marvell | +1 650 787 9141
From: Oren Ben-Kiki [mailto:o...@ben-kiki.org] Sent: Saturday, June 25, 2011 10:58 PM To: Trans Cc: yaml-c...@lists.sourceforge.net Subject: Re: [Yaml-core] Reg: PyYAML grammar
There's nothing wrong with whole-file/stream compression :-) But from what I understand, Paul is looking for something else, which is mucking with the internal structure, such as allowing for non-printable binary blobs.
BTW, YAML already provides a form of internal "compression". If you have many keys with some "long value foo", you can easily add an anchor to the 1st one and specify all the rest as "*f". Think of it as a poor man's LZW. IMO minimal-size flow-style YAML with heavy use of such anchors, plus some streaming compression of the stream, should give you a pretty compact format.
Of course I don't know Paul's application - he hinted that doing gzip may be too heavy for his use case (and I don't know what is data and BW constraints are). He's pretty much required to invent his own format which is tailored to his specific environment (or so it seems).
Have fun,
Oren Ben-Kiki
On Sun, Jun 26, 2011 at 5:20 AM, Trans <transf...@gmail.com<mailto:transf...@gmail.com>> wrote:
On Jun 13, 8:20 pm, Paul Lambert <p...@marvell.com<mailto:p...@marvell.com>> wrote:
> Ok - the YAML community and binary encoding do not appear to be compatible. Too bad your world view is so limited. Take a look a protobuf as an example of combining human readable and binary encodings if you want to see a worked hybrid.
I don't understand. What's wrong with compression?
--------------------------------------------------------------------------- --- All the data continuously generated in your IT infrastructure contains a definitive record of customers, application performance, security threats, fraudulent activity and more. Splunk takes this data and makes sense of it. Business sense. IT sense. Common sense.. http://p.sf.net/sfu/splunk-d2d-c1 _______________________________________________ Yaml-core mailing list Yaml-c...@lists.sourceforge.net<mailto:Yaml-c...@lists.sourceforge.net> https://lists.sourceforge.net/lists/listinfo/yaml-core
--------------------------------------------------------------------------- --- All of the data generated in your IT infrastructure is seriously valuable. Why? It contains a definitive record of application performance, security threats, fraudulent activity, and more. Splunk takes this data and makes sense of it. IT sense. And common sense. http://p.sf.net/sfu/splunk-d2d-c2
