Interoperable Security Vulnerability Checking/Tracking

[Pádraic Brady]

Hi all,

I'm separating this from the "FIG you are doing it wrong" thread to see how practical it is and to see if enough support exists to consider doing something practical, like writing actual code. You can read the less technical introduction on my blog if you haven't already.

http://blog.astrumfutura.com/2013/05/publishing-security-disclosures-in-consumable-formats-for-simpler-aggregation-and-security-checking/

So, the summary version is to try and make it easier for users to discover if their applications' dependencies have any known security vulnerabilities by querying each package's repository (using an aggregating server to map package names to relevant data files). This is more decentralised than the current SensioLabs Advisory Checking Service which relies on manual submissions to the service.

To keep things simple, I've presumed to piggy back Composer's architecture in some respects - I know there might be a strong temptation to have Composer implement more but that is not a requirement so I've tried not to imply it.

The basics are as follows:

1. Projects will publish an advisories.json file in the development branch of their git repositories, e.g. master.
2. The URI to this file should remain fixed but may be moved as workflow dictates.
3. Projects will register the URI with one or more services which maintain a mapping of URIs to Composer packages.
4. A single advisories.json URI may serve more than one package (e.g. zend-feed is a sub-package to zendframework).
5. A package should be associated with at least one advisories.json URI.
6. Aggregating services may opt to accept advisories not voluntarily published by a package's authors.
7. An advisory checking application (online or via API) should scan any submitted composer.lock file for packages and query one or more aggregating services for advisories matching the given package versions. The results will be presented to the user.
8. Aggregating services should support HTTPS and will not store queries or their results insofar as this is unnecessary to normal server operations.
9. The advisories.json file will contain an up to date list of advisories impacting the packages that it serves. It will not contain advisories in respect of its dependencies.
10. The advisories.json file will contain the following data for each advisory (at minimum): Title, ID, Summary Description, Long Description, Publication Date, Author(s), Package Name(s) and Version Range, Advisories URI, and Package Reference URI (optional).

10 points is obviously arbitrary but it's enough to sketch out the basics of how this might operate. It can be fleshed out or simplified over time as needed.

Questions for the audience:

Is this worth pursuing?
Would your project be likely to view this as a positive or an annoyance?
Would it be appropriate to propose a PSR for this?
Do you have suggestions, whether on this or on an alternative idea?
Do I need to quit drinking?

Paddy

[Evert Pot]

I think so, it's a good idea (Y)

> Would your project be likely to view this as a positive or an annoyance?

Positive, definitely.

> Would it be appropriate to propose a PSR for this?

For the format, I'd say yes.

> Do you have suggestions, whether on this or on an alternative idea?

I'd like to stop spamming the root of my project for every single tool that wants a configuration file. It would be nice if there's no restrictions on the name and path of the json file, but rather leave it to the aggregating services to provide a way to specify this path. Any publicly accessible url should work.

The idea that we're now relying on github repositories rather than reliable HTTP GET bothers me

I also feel that .json is not great format for writing. I'd love a format that allows comments, but I feel a LOT less strong about that.

> Do I need to quit drinking?

Work hard; play hard.

Evert

[Cal Evans]

Paddy!

I like the idea. I think it is the first attempt at solving a real problem.

Evert!

> Do you have suggestions, whether on this or on an alternative idea?

> I'd like to stop spamming the root of my project for every single tool that wants a configuration file. It would be nice if there's no restrictions on the name and path of the json file, but rather leave it to the aggregating services to provide a way to specify this path. Any publicly accessible url should work.

I agree with you here. How do you suggest we standardize discovery? Would the composer.json spec be updated to incluse an optional to advisories.json? 

>I also feel that .json is not great format for writing. I'd love a format that allows comments, but I feel a LOT less strong about that.

I disagree with you here. I would love to see us keep json as the defacto storage format.

My $0.02,

=C=

--
You received this message because you are subscribed to the Google Groups "PHP Framework Interoperability Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to php-fig+u...@googlegroups.com.
To post to this group, send email to php...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
Read this book before you hire a 

developer to build your website.

Avoiding a Goat Rodeo

[Evert Pot]

> > Do you have suggestions, whether on this or on an alternative idea?
>
> > I'd like to stop spamming the root of my project for every single tool that wants a configuration file. It would be nice if there's no restrictions on the name and path of the json file, but rather leave it to the aggregating services to provide a way to specify this path. Any publicly accessible url should work.
>
> I agree with you here. How do you suggest we standardize discovery? Would the composer.json spec be updated to incluse an optional to advisories.json?

Well at some point a url has to be entered somewhere.
At the moment it seems that the github url has this role more and more, but git is a terrible API.

I reckon that url should just be the full url to an advisories.json. This file can happily link back to github, or composer, or anything else..

If a 'central discovery mechanism' is desirable to allow people to just enter a project's domain, we should use:
http://tools.ietf.org/html/rfc5785 and define something like :

/.well-known/psr-advisories

That simply redirects to the actual file.

If Jordi wants to provide a link from composer.json to the advisories.json, this to me is outside the scope of php-fig.

Evert

[Zachary King]

I also feel that .json is not great format for writing. I'd love a format that allows comments, but I feel a LOT less strong about that.

A question, what would you use the comments for?

If it was related to the actual security disclosure I would say just put a 

    "comment": "blah blah blah"

element in the json. Not sure if that makes sense for this application though... Guess it depends on what you want the comments for. ;)

I am new around here so if I stick my nose somewhere I shouldn't just let me know :)

-Zach

[Evert Pot]

On May 16, 2013, at 5:52 PM, Zachary King <zchr...@gmail.com> wrote:

> I also feel that .json is not great format for writing. I'd love a format that allows comments, but I feel a LOT less strong about that.
>
> A question, what would you use the comments for?
>
> If it was related to the actual security disclosure I would say just put a
>
> "comment": "blah blah blah"
>
> element in the json. Not sure if that makes sense for this application though... Guess it depends on what you want the comments for. ;)
>
> I am new around here so if I stick my nose somewhere I shouldn't just let me know :)

No worries...

I'm a very heavy commenter in my code. Take composer.json as an example. If I have a list of PHP extensions and packages in 'require' or 'require-dev', I'm the type of guy that would want to include _why_ I need a certain dependency.

Virtually any type of configuration format meant to be written by humans, and not machines has a mechanism for this. <!-- --> for xml, # for many system configuration files, so it annoys me deeply that .json does not allow this, and for me the sole reason why I will never opt to use .json for a configuration format.

Evert

[Pádraic Brady]

Hi Evert,

On 16 May 2013 17:11, Evert Pot <ever...@gmail.com> wrote:
> On May 16, 2013, at 4:13 PM, Pádraic Brady <padrai...@gmail.com> wrote:

>> Questions for the audience:

>> Do you have suggestions, whether on this or on an alternative idea?
>
> I'd like to stop spamming the root of my project for every single tool that wants a configuration file. It would be nice if there's no restrictions on the name and path of the json file, but rather leave it to the aggregating services to provide a way to specify this path. Any publicly accessible url should work.
>
> The idea that we're now relying on github repositories rather than reliable HTTP GET bothers me
>
> I also feel that .json is not great format for writing. I'd love a format that allows comments, but I feel a LOT less strong about that.

I assumed version control, sorry, but the primary target is a URI.
Putting in a VCS is more realistic for projects without a dedicated
website (e.g. my own Mockery). You could, however, easily put the JSON
file anywhere for a HTTP GET request.

The choices for format are JSON, YAML or XML (any
suggestions/preferences from folk?). XML is very likely going too far
though I'm happy to write a brief RSS/Atom extension spec if it would
ease adoption. I know ZF already has a RSS feed for advisories, for
example. I just want to avoid the whole XML generation tool mess that
might turn something good into something truly annoying. So assuming
XML as a possible alternative - that leaves JSON or YAML to select as
the primary format. I know the SensioLabs service uses YAML at
present. I actually have no particular preference so long as its
something parseable ;).

--

--
Pádraic Brady

http://blog.astrumfutura.com
http://www.survivethedeepend.com
Zend Framework Community Review Team
Zend Framework PHP-FIG Representative

[Pádraic Brady]

Hi Cal,

On 16 May 2013 17:42, Cal Evans <c...@calevans.com> wrote:
>> I'd like to stop spamming the root of my project for every single tool
>> that wants a configuration file. It would be nice if there's no restrictions
>> on the name and path of the json file, but rather leave it to the
>> aggregating services to provide a way to specify this path. Any publicly
>> accessible url should work.
>
> I agree with you here. How do you suggest we standardize discovery? Would
> the composer.json spec be updated to incluse an optional to advisories.json?

If Packagist.org were to adopt the role of mapping packages to
advisories.json|yaml, this would make sense for them. We could define
simple conventions. For example, check if the canonical master branch
contains an advisories file or, where a website exists, check if it
contains something like a (just invented) X-Security-Advisories meta
tag and/or header. That's usually the typical discovery when it comes
to HTTP GET ops. Related, of course, is managing URI changes and
ensuring clients don't get stuck trying to request obsolete advisory
files.

>>I also feel that .json is not great format for writing. I'd love a format
>> that allows comments, but I feel a LOT less strong about that.
> I disagree with you here. I would love to see us keep json as the defacto
> storage format.
>

[Jason Judge]

On Thursday, 16 May 2013 17:52:20 UTC+1, Zachary King wrote:

...

A question, what would you use the comments for?

If it was related to the actual security disclosure I would say just put a 

    "comment": "blah blah blah"

element in the json. Not sure if that makes sense for this application though... Guess it depends on what you want the comments for. ;)

Jumping in myself here, I will have to say I am liberal with comments; I comment everything, because coming back to code the following week can be like looking at something a stranger has written. Finding that .json files are so widely used, and still do not support comments, is just weird to me.

If you have ever tried adding a "comment": "blah bling blang" to a composer file, you will find composer throwing it back in your face with a "WTH is this? Nah!" It is a technique that cannot be relied on, unfortunately. But what can be do? We are stuck with json files I think.

-- Jason

[Larry Garfield]

In concept, I like! However, there's a few important details here.

First, I agree with others who have said that a mandatory file in a repo
is the wrong approach, for various reasons. I will go a step further,
however. We don't want to be using a file at all. What we're really
talking about here is projects offering a standard-format *feed* of
security advisories, with enough information that it COULD be
cross-referenced to a package broker service such as composer.

I say feed for a couple of reasons.

1) Consider a large or older project. Drupal has been around for 13
years. We have had a lot of SAs in that time. If you include Drupal's
thousands of add-on modules that are also served by Drupal.org, there's
hundreds. (That's not a slam against Drupal; it's a good mark of how
seriously we take security SA best practices. <g>) Putting all of those
in one big file is not going to scale at all.

2) We really shouldn't be doing this via "grab a repo and open a file."
This should be done as a REST API call. What a project should publish
is simply the URI from which recent SAs can be requested; no more, no
less. (It can publish it as a composer.json key, but that's just one
option.)

3) Once you think of it in terms of a feed, you realize that this isn't
even a PHP-specific question. *Any* software package should be offering
a machine-readable SA feed. PHP's not special in that regard.

A little quick googling finds:

Drupal offers human-readable feeds with RSS versions (grouped by
core/contrib):
http://drupal.org/security
http://drupal.org/security/rss.xml

Symfony offers an RSS feed:
http://symfony.com/blog/category/security-advisories

Ubuntu offers an Atom feed with embedded HTML:
http://www.ubuntu.com/usn/atom.xml

Debian offers an RDF feed:
http://www.debian.org/security/dsa

Joomla offers an RSS feed:
http://feeds.joomla.org/JoomlaSecurityNews

Cisco offers an RSS feed:
http://tools.cisco.com/security/center/rss.x?i=44

VMWare claims to offer an RSS feed, but it appears broken at the moment:
http://www.vmware.com/security/advisories/
feed:http://vmware.simplefeed.net/rss

So it seems a lot of RSS feeds in human-readable format. Of course, I
agree that machine-parsable would be much more useful for automation
like Sensio is doing. For public feeds... XML is the dominant player.
(Sorry for the angle-bracket allergic in the audience.)

So, I would revise/counter-propose to Paddy something simpler:

1) Projects publish a known-stable URI at which security advisories will
be served.
2) That URI must return Atom, with a to-be-defined machine-readable
payload. (We could use something existing in RDF, or come up with
something, or find an IETF standard, or whatever.) Atom offers much
better machine parsing than RSS, as well as pubsub capability (which
would be very useful here).
3) That URI must use HTTPS to avoid man-in-the-middle attacks.
4) Aggregators (Composer, Sensio, or otherwise) MAY aggregate and
present that data however is logical for them. That includes caching
for a limited time.
5) Projects MAY aggregate multiple projects into a single feed if that
is documented in the feed's header.
6) There is no step 6. :-)

Of course, plenty of projects are small enough they wouldn't want to
setup their own feed service. That's fine; as long as it's a valid URI
it can be hosted by anyone, which means, for instance, Sensio could host
and serve feeds for Symfony-related projects if it wanted to, and then
individual project devs could just manually enter details into a form on
Sensio's site and it would get published.

Alternatively, if we can find an existing spec that's already doing this
(PHP or otherwise), we just piggyback on that and call it a day.

I'd have to speak to the Drupal security team to answer "beneficial or
annoyance", but I suspect beneficial as long as it's not too difficult
to implement. We already offer security notice feeds using a completely
custom XML format that installed Drupal sites pull from by default to
notify people of SAs. Switching to a standard format for that seems like
it would only be a good thing.

--Larry Garfield

> --
> You received this message because you are subscribed to the Google
> Groups "PHP Framework Interoperability Group" group.
> To unsubscribe from this group and stop receiving emails from it, send
> an email to php-fig+u...@googlegroups.com.
> To post to this group, send email to php...@googlegroups.com.

> To view this discussion on the web visit
> https://groups.google.com/d/msgid/php-fig/825332e9-e550-4581-92f6-5d350490b06c%40googlegroups.com?hl=en.

[Drak]

On 16 May 2013 18:31, Pádraic Brady <padrai...@gmail.com> wrote:

I just want to avoid the whole XML generation tool mess that
might turn something good into something truly annoying. So assuming
XML as a possible alternative - that leaves JSON or YAML to select as
the primary format. I know the SensioLabs service uses YAML at
present. I actually have no particular preference so long as its
something parseable ;).

I agree, XML does make things annoying. Need to think about the little guy here.

It's why docbook never really exploded into the mainstream for documentation and

markdown formats (like md or rst) have boomed. The easiest format is YAML

followed second by JSON. Both are machine readable, JSON would be more 

standard I suppose, but it is one step less easy for n00bs.

Regards,

Drak

[Drak]

On 17 May 2013 04:03, Larry Garfield <la...@garfieldtech.com> wrote:

1) Consider a large or older project.  Drupal has been around for 13 years.  We have had a lot of SAs in that time.  If you include Drupal's thousands of add-on modules that are also served by Drupal.org, there's hundreds.  (That's not a slam against Drupal; it's a good mark of how seriously we take security SA best practices. <g>)  Putting all of those in one big file is not going to scale at all.

I'm not sure the second part here is in cope: projects with large 3rd party module/plugin support - but they are not part of the project. They fall into the same category as a composer project with many dependencies - so it would not be up to Drupal imo, to list all the 3rd party CVEs (and it would be impossible to do accurately anyway).  

Regards,

Drak

[Larry Garfield]

We currently manage SAs for all hosted contributed modules through a central feed. 
http://drupal.org/security/contrib

There's also a per-project XML feed of releases, which are used to notify installed sites that there are updates they should install (security or otherwise).  Paddy's original post included mention of "sub components" being managed by a single feed, which is why I mentioned it.

That's beside the point, though.  The key there is that a "file in the repo" is not going to scale.  For an old project like Drupal or Joomla or Wordpress (or any project 5-6 years from now), that would be a massive file.  A feed URI will scale far better, and be more extensible, and more reusable.

--Larry Garfield

[Moisa Teodor]

Hi all,

Interesting discussion. I too believe Atom is a better suited format for this kind of task, as it is vendor/technology neutral, and besides the format itself, it also has a publishing protocol. Besides that, even if some JSON structure looks easier, Atom is a IETF standard, and lots of tools and libraries already exist for manipulating and working with Atom.

--

You received this message because you are subscribed to the Google Groups "PHP Framework Interoperability Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to php-fig+u...@googlegroups.com.
To post to this group, send email to php...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.
 
 

--
Doru Moisa
web: http;//moisadoru.wordpress.com
tel: +40 720 861 922
Bucharest, Romania

[Drak]

I think the point is, we need to make as little resistance to it as possible for the little guy. XML based formats are just complex if the developer is not familiar with it (and why would they be). It's a-lot-of-effort === resistance === less adoption. There are plenty of tools for creating XML too, but it's not the point - they are complex, XML is complex fullstop. Simple markup is plainly obvious. I think that's the reason YAML and JSON are becoming so popular as is Markdown. So I see a big part of the solution in being what is easiest for the lowest common denominator without having to learn something new.

Regards,

Drak

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/CAEJbJphsutua1XJRb9Fz8oqqX_mNXVWEwjX_ev%2BHwRhX04BWoQ%40mail.gmail.com?hl=en.

[Moisa Teodor]

Provided that tools for consumig/working with the security vulnerability tracking, 'the little guy' will not be exposed to the underlying data format (JSON or Atom). 

In fact, I believe that using an already existing standard (in this case Atom) should always be preferred instead of inventing something new. Besides that, it should also help tool developers - it is always easier to develop something against a well established standard that against something completely new, not tested in the wild and subject to changes at any time.

Not to mention the fact that using outside tools that work with Atom, you can still access the information, no need for a custom tool to do that. Or the fact that for some other party that wants to consume that information, for example users who use feed readers to be updated when something new appears, all they need to do is to just point their tool to the feed URL.

[Ryan McCue]

P�draic Brady wrote:
> The choices for format are JSON, YAML or XML (any
> suggestions/preferences from folk?). XML is very likely going too far
> though I'm happy to write a brief RSS/Atom extension spec if it would
> ease adoption. I know ZF already has a RSS feed for advisories, for
> example. I just want to avoid the whole XML generation tool mess that
> might turn something good into something truly annoying. So assuming
> XML as a possible alternative - that leaves JSON or YAML to select as
> the primary format. I know the SensioLabs service uses YAML at
> present. I actually have no particular preference so long as its
> something parseable ;).

As the developer of SimplePie, please no for XML. Having a feed with
this sort of data in it would be cool, but the canonical source should
be either JSON or YAML.

Personally, I'd prefer YAML. JSON isn't really intended to be a
human-editable representation of data (IMO), whereas YAML is
specifically design for this sort of thing.

(I'd also much prefer composer.yaml over composer.json, but it's too
late for that.)
--
Ryan McCue
<http://ryanmccue.info/>

[Pádraic Brady]

Hi Larry,

On 17 May 2013 04:03, Larry Garfield <la...@garfieldtech.com> wrote:

> In concept, I like! However, there's a few important details here.
>
> First, I agree with others who have said that a mandatory file in a repo is
> the wrong approach, for various reasons. I will go a step further, however.
> We don't want to be using a file at all. What we're really talking about
> here is projects offering a standard-format *feed* of security advisories,
> with enough information that it COULD be cross-referenced to a package
> broker service such as composer.
>
> I say feed for a couple of reasons.
>
> 1) Consider a large or older project. Drupal has been around for 13 years.
> We have had a lot of SAs in that time. If you include Drupal's thousands of
> add-on modules that are also served by Drupal.org, there's hundreds.
> (That's not a slam against Drupal; it's a good mark of how seriously we take
> security SA best practices. <g>) Putting all of those in one big file is
> not going to scale at all.

Agreed. My proposal does explicitly mention a JSON file but it should
allow for more scalable resources (as my opening blog post suggested).
I don't believe anyone should expect Drupal, Wordpress or applications
at that level to produce a single epic JSON file. However, I assume
most projects are much much smaller and we can't realistically expect
all of them to operate a service endpoint or generate an XML feed when
many of them lack a website with access to a scripting language (e.g.
relying on Github wiki or static pages).

> 2) We really shouldn't be doing this via "grab a repo and open a file."
> This should be done as a REST API call. What a project should publish is
> simply the URI from which recent SAs can be requested; no more, no less.
> (It can publish it as a composer.json key, but that's just one option.)

As above - it may sink the ship to require every project/package to
maintain a REST API. I'd prefer to maintain a limited set of
alternatives that suit projects with different resources/needs than
force everyone down the same rabbit hole.

> 3) Once you think of it in terms of a feed, you realize that this isn't even
> a PHP-specific question. *Any* software package should be offering a
> machine-readable SA feed. PHP's not special in that regard.

Far from it! There are machine readable formats out there in popular
use for distributing security information in XML. The best matching
one is probably Common Vulnerability Reporting Framework (CVRF):
http://www.icasi.org/cvrf. ICASI is a limited-member indutry group
(Oracle, Microsoft, IBM, Intel, Juniper, et al). There is also Open
Vulnerability and Assessment Language (OVAL) from MITRE (the CVE
guys): http://oval.mitre.org/about/. OVAL has been around since the
early 2000s but CVRF is practically fresh off the assembly line. If I
remember, Red Hat uses CVRF and Debian/SuSE uses OVAL.

My preference would be CVRF which was heavily influenced by Red Hat's
involvement in the WG and seems to be a tidy format (for XML).

Note: Neither of these are RSS/Atom based and I'm not aware of an Atom
container syntax. It was discussed way back but I've only ever seen
the base format published.

> A little quick googling finds:
>
> Drupal offers human-readable feeds with RSS versions (grouped by
> core/contrib):
> http://drupal.org/security
> http://drupal.org/security/rss.xml
>
> Symfony offers an RSS feed:
> http://symfony.com/blog/category/security-advisories
>
> Ubuntu offers an Atom feed with embedded HTML:
> http://www.ubuntu.com/usn/atom.xml

<snip>

The question is though, how interoperable are those RSS feeds? How can
you map generic RSS back to specific versions and package names? You
could specify an RSS extension for new elements to use but this
requires a non-standard specification when we already have a few good
non-RSS standards out there to lean on.

> So it seems a lot of RSS feeds in human-readable format. Of course, I agree
> that machine-parsable would be much more useful for automation like Sensio
> is doing. For public feeds... XML is the dominant player. (Sorry for the
> angle-bracket allergic in the audience.)

From another perspective, the dominant player is to have no feeds.

> So, I would revise/counter-propose to Paddy something simpler:
>
> 1) Projects publish a known-stable URI at which security advisories will be
> served.
> 2) That URI must return Atom, with a to-be-defined machine-readable payload.
> (We could use something existing in RDF, or come up with something, or find
> an IETF standard, or whatever.) Atom offers much better machine parsing
> than RSS, as well as pubsub capability (which would be very useful here).
> 3) That URI must use HTTPS to avoid man-in-the-middle attacks.
> 4) Aggregators (Composer, Sensio, or otherwise) MAY aggregate and present
> that data however is logical for them. That includes caching for a limited
> time.
> 5) Projects MAY aggregate multiple projects into a single feed if that is
> documented in the feed's header.
> 6) There is no step 6. :-)

If we accept XML as an alternative, which I fully support, I'd make
point 2 use CVRF and remove point 3. HTTPS is not necessary for
distributing advisories. At a stretch, an attacker could manipulate a
feed/API result to trick a user into installing an insecure dependency
but they can already try that by manipulating stacks of RSS feeds and
blog posts. HTTPS is another barrier to adoption and, in this case,
the cost seems to outweight the small benefit it would provide.

> Of course, plenty of projects are small enough they wouldn't want to setup
> their own feed service. That's fine; as long as it's a valid URI it can be
> hosted by anyone, which means, for instance, Sensio could host and serve
> feeds for Symfony-related projects if it wanted to, and then individual
> project devs could just manually enter details into a form on Sensio's site
> and it would get published.

That's the system I actually want to avoid at all costs. If the
central service goes down, we lose all of the data. Part of the
Composer/Github duo's charm, versus something like PEAR, is that
should Composer die and Packagist.org fall offline, the entire
packaging ecosystem can be resurrected in short order. The data and
the hosted packages are independent of the services and tools.

Secondly, this also ensures that implementing a checker has options -
you can aggregate advisories or just tell clients which URIs to GET
and analyse outside of the service's domain, i.e. just as Composer
maps package names to downloads from GH.

Thirdly, I suspect that filling out forms on a website imposes more of
a challenge than editing a local file (or database). I want to keep
the effort minimal so this has a better change of being successful in
the community.

> Alternatively, if we can find an existing spec that's already doing this
> (PHP or otherwise), we just piggyback on that and call it a day.
>
> I'd have to speak to the Drupal security team to answer "beneficial or
> annoyance", but I suspect beneficial as long as it's not too difficult to
> implement. We already offer security notice feeds using a completely custom
> XML format that installed Drupal sites pull from by default to notify people
> of SAs. Switching to a standard format for that seems like it would only be
> a good thing.

Is there a URI to those documented somewhere? Might check how
different/compatible it would be to a CVRF feed (anything extra to
CVRF could still be separately namespaced in the XML assuming half
decent parsers).

Paddy

[Pádraic Brady]

Hi Moisa,

On 17 May 2013 10:07, Moisa Teodor <mois...@gmail.com> wrote:
> Provided that tools for consumig/working with the security vulnerability
> tracking, 'the little guy' will not be exposed to the underlying data format
> (JSON or Atom).
>
> In fact, I believe that using an already existing standard (in this case
> Atom) should always be preferred instead of inventing something new. Besides
> that, it should also help tool developers - it is always easier to develop
> something against a well established standard that against something
> completely new, not tested in the wild and subject to changes at any time.

In this case, there are XML formats specifically geared towards
sharing security advisory data. CVRF is the newest candidate and quite
approachable as a format. When I knock out Zend Framework final
response to the Cache PSR proposal (which is WAY overdue!), I'm going
to take a stab at writing a simple parser/writer library for the
format - simply to have something concrete so it's less of an unknown
to hamper progress on this idea.

I think Larry, given Drupal's size, will make the final choice as to
the XML format. Even Zend Framework and Symfony combined, despite
being large frameworks, don't have enough advisories to make JSON/YAML
too much trouble at this time - Larry is right that this can change
over time as the molehill becomes a mountain.

> Not to mention the fact that using outside tools that work with Atom, you
> can still access the information, no need for a custom tool to do that. Or
> the fact that for some other party that wants to consume that information,
> for example users who use feed readers to be updated when something new
> appears, all they need to do is to just point their tool to the feed URL.

The main problem in using Atom is that we're targeting machine
readable data and not simply human readable data. We could use an Atom
feed, but we'd have to use it according to a custom specification and
probably a new XML namespace for any additional data that can't be
mapped 1:1 to Atom elements. This would trade the ease of using
familiar tools (e.g. Zend\Feed supports custom Atom/RSS extensions)
for the complexity of remembering all the mapping rules and the
downside of not using an industry standard format tailor made for this
domain.

[Moisa Teodor]

Hi Padraic, all,

I am biased towards Atom as the underlying communication protocol and format between services and consumers tools because besides the format itself, it also has:

 - a standardized publishing protocol, described in RFC-5023 (so there's no need to re-invent/re-implement this stuff from scratch)

 - extension points, as described in RFC-4287, section 6 (anything not already covered by the format itself can be added)

 - encryption and signing, as described in RFC-4287, section 8.5 (could be used to secure the communication or even to submit private reports, or to provide private vulnerability information to the affected project)

Of course, a service could also provide human readable web content, or other alternative machine parse-able formats (JSON, YAML etc.).

As a side note, besides CVRF, there is also VUXML - used by FreeBSD's port auditing system (example).

--
You received this message because you are subscribed to the Google Groups "PHP Framework Interoperability Group" group.
To unsubscribe from this group and stop receiving emails from it, send an email to php-fig+u...@googlegroups.com.
To post to this group, send email to php...@googlegroups.com.

To view this discussion on the web visit https://groups.google.com/d/msgid/php-fig/CALwr1GniCU5qPNiGAEP3G4QGT57VKnoCHgGSLU4FWRNg4kNhXA%40mail.gmail.com?hl=en.

For more options, visit https://groups.google.com/groups/opt_out.

[Amy Stephen]

To add to Larry's comments, what would be ideal would be to have a warning or notice of only those advisories that pertain to your release.

So, if someone has software that needs updating, they need the notification. It would be best to filter out the history that is no longer relevant. More complex to implement, but better information for the user. (If that makes sense.)

I do not believe there is a single issue more important than this to PHP apps. It's been wished for, dreamed about, spoken of in futuristic terms, for a very long time.

[Pádraic Brady]

Hi Amy,

Yep, should have made that point 11 - I don't expect the user to deal
with a massive vuln list during checking (unless they want to), just
those relevant to the versions their dependencies are locked at.

Paddy

> --
> You received this message because you are subscribed to the Google Groups
> "PHP Framework Interoperability Group" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to php-fig+u...@googlegroups.com.
> To post to this group, send email to php...@googlegroups.com.
> To view this discussion on the web visit

> https://groups.google.com/d/msgid/php-fig/15e15fca-8599-458d-81cf-fb5ea0478b59%40googlegroups.com?hl=en.

>
> For more options, visit https://groups.google.com/groups/opt_out.
>
>

--

[Larry Garfield]

Responding together up here for simplicity...

- I included HTTPS because it was in your original proposal. I'm fine
with leaving that out. :-)

- Leaving out HTTPS makes it even simpler, because if all you provide is
a URI it doesn't matter if it's a generated feed or not. A static URI
to some page in the master branch of a GitHub repo is just as valid as
an on-the-fly generated dynamic feed. So Monolog, eg, could put a
hand-edited file in its repo and provide a direct link to GitHub.
Drupal could have a generated feed coming off Drupal.org. To consumers
downstream (e.g., Composer), both are identical.

- I've not looked into CVRF or OVAL so I have no strong preference
there, but generally +1 to reusing existing standards. However, unless
they already have a listing mechanism we could still use something like
Atom as an envelope. Atom is specifically designed as an envelope
format around arbitrary XML/HTML/text; it would be fully spec-legal to
have an Atom feed where each entry is a CVRF/OVAL XML string.

- RSS feeds are not machine parsable, at least the payload isn't. The
payload in all of those cases appears to be arbitrary HTML, or RDF in
Debian's case. I wasn't suggesting we use RSS.

- I checked, and the Drupal XML format I mentioned isn't for SAs but for
project releases. (A separate but also valuable thing to offer.) It's
also entirely undocumented aside from the parsing code that uses it.
Sad panda.

- As far as Drupal adoption of such a common SA format, quote from our
current Security Team lead: "I can't imagine we'd be opposed to it
unless there were a bunch of required fields that required additional work"

--Larry Garfield

> P�draic Brady

[Pádraic Brady]

Hey Larry,

On 17 May 2013 18:20, Larry Garfield <la...@garfieldtech.com> wrote:
> Responding together up here for simplicity...
>
> - I included HTTPS because it was in your original proposal. I'm fine with
> leaving that out. :-)

Ah! The HTTPS mention there was for any central mapping/aggregation
service. It's probably prudent to ensure that folks' dependency
versions being sent out for checking are protected from eavesdropping
so as not to tip off any would be attackers.

> - Leaving out HTTPS makes it even simpler, because if all you provide is a
> URI it doesn't matter if it's a generated feed or not. A static URI to some
> page in the master branch of a GitHub repo is just as valid as an on-the-fly
> generated dynamic feed. So Monolog, eg, could put a hand-edited file in its
> repo and provide a direct link to GitHub. Drupal could have a generated feed
> coming off Drupal.org. To consumers downstream (e.g., Composer), both are
> identical.

Exactly.

> - I've not looked into CVRF or OVAL so I have no strong preference there,
> but generally +1 to reusing existing standards. However, unless they
> already have a listing mechanism we could still use something like Atom as
> an envelope. Atom is specifically designed as an envelope format around
> arbitrary XML/HTML/text; it would be fully spec-legal to have an Atom feed
> where each entry is a CVRF/OVAL XML string.

CVRF can list 1 or more vulnerabilities in the same document so
listing is supported.

> - RSS feeds are not machine parsable, at least the payload isn't. The
> payload in all of those cases appears to be arbitrary HTML, or RDF in
> Debian's case. I wasn't suggesting we use RSS.
>
> - I checked, and the Drupal XML format I mentioned isn't for SAs but for
> project releases. (A separate but also valuable thing to offer.) It's also
> entirely undocumented aside from the parsing code that uses it. Sad panda.
>
> - As far as Drupal adoption of such a common SA format, quote from our
> current Security Team lead: "I can't imagine we'd be opposed to it unless
> there were a bunch of required fields that required additional work"

I'm going to play around with a CVRF parser at the weekend - turns out
there's mainly the Python reference implementation so I'll port
something to PHP and see how much is compulsory and mappable to what
we require. Mainly so it's done regardless of the outcome of this
discussion making it less of a guessing point.

Paddy

--

--
Pádraic Brady