Re: [jfw] Validate, Filter, and Escape Package

[Andrew Eddie]

On 17 April 2014 11:44, Amy Stephen <> wrote:
> Andrew indicated he is working in this area, as well. There might be another
> option available or a way to combine work.

I've taken a slightly different approach and separated sanitisation
(filtering) from validation. So i've got a Filter package and a
Validator package.

Regards,
Andrew Eddie

[Amy Stephen]

Lots of ways to do, that's for sure.

I use an adapter for each data type (i.e., integer, or foreign key or an array).

I have a validate, filter and escape for each adapter.

There are two reasons I combined these into one package:

1. Many times, the code is the same -- often, a validate method will execute the filter method and then compare the results to the input value. If it's the same, it's valid. As a second example, filtering and "escaping" an integer is the same. Tried to keep it simple and DRY.

 2. Wanted to enable chaining. In a single call, one could validate 1) the value is an integer and 2) the value is a foreign key value. I've hooked it up to my "ORM light" so that these functions are automated on forms and during display.

Lots of ways to do it though.

Link to your code? Or, is it available publicly?

--
Framework source code: https://github.com/joomla/joomla-framework
Visit http://developer.joomla.org for more information about developing with Joomla!
---
You received this message because you are subscribed to a topic in the Google Groups "Joomla! Framework Development" group.
To unsubscribe from this topic, visit https://groups.google.com/d/topic/joomla-dev-framework/HK9p1QaX6BA/unsubscribe.
To unsubscribe from this group and all its topics, send an email to joomla-dev-frame...@googlegroups.com.
Visit this group at http://groups.google.com/group/joomla-dev-framework.

[Sven Versteegen]

Nice work Amy, but I really don't like your if else construction where the if does nothing, and the mix of strict and non-strict comparisons sometimes.
Another thing for example:

$test = filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING, $this->setFlags());

if ($test == true) {

} else {

    $this->setFieldValue(filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING));

}

filter_var returns false or the filtered value, so why check for true and do again nothing instead of checking for false
if (false === $test)
{
    $this->setFieldValue(filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING));

[Amy Stephen]

On Wednesday, April 16, 2014 10:16:15 PM UTC-5, Sven Versteegen wrote:

Nice work Amy, but I really don't like your if else construction where the if does nothing,

Yes, I think I'm the only PHP developer who is old enough to have been trained on using "positive" logic.

Definitely would "fix" that for Joomla.
 

and the mix of strict and non-strict comparisons sometimes.

You are right -- I am not completely clear on this.  If you have interest in helping on this, I would welcome the PRs.

 

Another thing for example:

$test = filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING, $this->setFlags());

if ($test == true) {

This should be === (I'll walk thru this)
 

} else {

    $this->setFieldValue(filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING));

}

filter_var returns false or the filtered value, so why check for true and do again nothing instead of checking for false
if (false === $test)
{
    $this->setFieldValue(filter_var($this->getFieldValue(), FILTER_SANITIZE_STRING));
}

Can you link to this adapter, please?

Thanks for the great feedback!

[Andrew Eddie]

On 17 April 2014 12:15, Amy Stephen <> wrote:
> 1. Many times, the code is the same -- often, a validate method will execute
> the filter method and then compare the results to the input value. If it's
> the same, it's valid. As a second example, filtering and "escaping" an
> integer is the same. Tried to keep it simple and DRY.

I know, and I did try it this way at first. But where it becomes
interesting is when you dabble into, for example, range validation,
length, etc and so on. The validation start to diverge from the
filtering.

> 2. Wanted to enable chaining. In a single call, one could validate 1) the
> value is an integer and 2) the value is a foreign key value. I've hooked it
> up to my "ORM light" so that these functions are automated on forms and
> during display.
>
> Lots of ways to do it though.

Agree.

> Link to your code? Or, is it available publicly?

Working on it.

Regards,
Andrew Eddie

[Sven Versteegen]

> Can you link to this adapter, please?

That example I took from the string adapter https://github.com/Molajo/Fieldhandler/blob/master/Source/Adapter/String.php

> This should be === (I'll walk thru this)

With this example my point was not the non-strict comparison. Like I wrote below the code filter_var returns false or the filtered value, so just check for false.

> Thanks for the great feedback!

That wasn't so great :) and I only did a very short look at the code, it's 6am here and my bed is calling me for hours^^

[Andrew Eddie]

Here's the Validator package (so far). No docs yet - still working on those.

https://github.com/eddieajau/validator

Regards,
Andrew Eddie

[piotr_cz]

Andrew > would it make sense, to use same constants for errors as in HTML5 ValidityState properties? https://developer.mozilla.org/en-US/docs/Web/API/ValidityState

I'm working on a form with both backend and frontend constraint validation and differences between these two APIs drive me crazy.

[Andrew Eddie]

On Thursday, 17 April 2014, piotr_cz <pkoni...@hotmail.com> wrote:

Andrew > would it make sense, to use same constants for errors as in HTML5 ValidityState properties? https://developer.mozilla.org/en-US/docs/Web/API/ValidityState

Yep. Shall we just create a ValidityState class just with those constants?

Regards

Andrew Eddie 

--

Regards,
Andrew Eddie

[Amy Stephen]

On Thursday, April 17, 2014 6:11:31 AM UTC-5, Andrew Eddie wrote:

On Thursday, 17 April 2014, piotr_cz <pkoni...@hotmail.com> wrote:

Andrew > would it make sense, to use same constants for errors as in HTML5 ValidityState properties? https://developer.mozilla.org/en-US/docs/Web/API/ValidityState

Yep. Shall we just create a ValidityState class just with those constants?

That is a good idea.  Good spot for a shared Interface, as well.

[Amy Stephen]

On Thursday, April 17, 2014 12:44:32 AM UTC-5, Andrew Eddie wrote:

Here's the Validator package (so far). No docs yet - still working on those.

https://github.com/eddieajau/validator

OK, took time to look thru this. Nice clean code. I see what you did with the Range inheritance. I use the Abstract Adapter for shared methods but I have been considering switching to a composition approach.

I put a spreadsheet out there and added you as an editor if you want to collaborate on definitions, plans, etc.

Couple of points:

1. Another reason I went with one package for validation, filtering and escaping was I wanted to keep things consolidated and clear -- this is where you do field stuff.

Too many times, it seems we have a little bit of filtering here or there and it's not obvious to developers where to do what. I couldn't come up with a good reason why a developer would want this validation package and this package for filtering and this package for escaping.

To me -- it's just Fieldhandling. Any time you deal with data, either by bringing it in to the application or displaying it - it needs to be handled. So, aside from the shared code, etc., above, my thinking, right or wrong, was to keep it all together.

2. In terms of keeping it simple, I like Joomla's Filter API and the Fieldhandler is modeled after the API in terms of what the developer has to do.

Right now, I have well over 50 data type adapters - some are custom for Molajo's application. For example, I have adapters for username, userid, catalog_id, category, tag, etc., some of the unique data elements I want treated a certain way.

But, the developer doesn't have to create those classes. Instead, a proxy class is injected into the relevant controller already instantiated, humming and ready to go. Then, each validation (or filter or escape) uses that instance.

There is no state to manage because the call thru the proxy creates the class instances needed (multiple in the event of chaining) and returns the results.

I think that's going to be easier to use than the approach you are using where the developer must create each validate class (know it's name then create the instance and then use it.)

Maybe your plan is to add a proxy like I described?

3. Looks like you are laying the framework and will need to then add all the data types.  Right now, you have 5 or so validations. Essentially, we are using the same approach on the validation, in a couple of cases we validate a data type slightly differently (I marked those on the spreadsheet), but other than that, the test, itself is close.

4. IMO, the PHP community needs to get-it-together on the HTML filtering. All the "experts" say to use HTML Purifier, but it *still* does not support HTML5. Personally, I'm only comfortable using a whitelist and I pulled in some very old code called kses to parse and filter the HTML. I'm thinking about how to "spread the word" and rally a little support either around HTML Purifier's update or some other solution.

As an aside, HTML Purifier had a serious security issue and fairly recent release. That's the first release since 2009. None of the plugins seem to be updated, including the one they use for their forums. Joomla's isn't even updated for new Joomla releases. (Plus, it's just used to escape the output -- not filter.)

Anyway, HTML filtering is still pretty antiquated and I'm not happy with my solution there -- but it's likely safe. I'm personally not adding blacklists or going further than that since I think it's not safe.

It'd be good to have the Framework team consider directions in this area. Do you want one package for all data element handling? Do you want to focus on the validation only? I'm happy to help once there is a good idea of where you want to lead. I don't want to get in the way -- we don't have to use my code in this form - if it helps to copy it into another format, fine.

Let me know the next steps if I can be of service.

Thanks!


 

Regards,
Andrew Eddie

[piotr_cz]

Maybe an interface that each validator woul implement

[Andrew Eddie]

On 17 April 2014 22:47, Amy Stephen <> wrote:
> I put a spreadsheet out there and added you as an editor if you want to
> collaborate on definitions, plans, etc.

I'm going to propose a crazy idea, and that is all we add to the
Framework are interfaces and possibly exception classes. So, for
example, if we define ValidatorInterface and FilterInterface, you
(Amy) can implement classes that include both while I, for example,
may implement two different packages for each interface. Let the
developers decide which implementation works for them (and we can list
them in the README files of the JF interface package).

How does that sound? Cake-and-eat-it solution?

In fact, wouldn't it be interesting if the whole Joomla Framework
ended up being nothing but interfaces :)

Regards,
Andrew Eddie

[Amy Stephen]

A framework full of interfaces? Here's mine. I have the filter, escape, and validate interfaces separated. Please feel free to use any or all of the interfaces on CommonAPI for the "Joomla Framework." 

OK, I can see this is going nowhere.

My offer to share code stands if anyone is interested in improving the antiquated framework and can take my work seriously enough to review it and collaborate on the communities needs.

 

Regards,
Andrew Eddie

[Andrew Eddie]

On 18 April 2014 12:23, Amy Stephen <> wrote:
> OK, I can see this is going nowhere.

What part of "why not just agree on an interface" don't you like? I
wasn't joking - that was a serious suggestion. That leaves the
implementation up to you (or me, or Don, or whomever), but anyone that
standardises on those interfaces is in a win-win position of having a
number of choices to work with. That would seem to be a logical first
step when there are at least two competing, but equally valid (I'm not
saying your way is wrong, it's just one option), coded solutions. I
seem to remember you advocating interfaces over implementation in the
past anyway.

And maybe I wasn't clear but it would be appropriate to list the user
libraries that implement these interfaces both as kudos to the
developers and helps the developer user find them. A nice side effect
is that you may also have more license freedom (can't wait to see what
our friend on twitter will make of that comment) but I'll play it safe
and stick to LGPL-2.1+ :)

/me is scratching his head wondering what just happened

Regards,
Andrew Eddie

[piotr_cz]

Interfaces sound good, but please don't go too abstract.

If you look at our MVC classes, these are in practice interfaces giving green developers just an advice how they could start. I have a feeling the not-opinionated framework is shifting towards theory and it's actually hard to use in real projects.

[Amy Stephen]

Of course I strive to build agreement on Interfaces, that's why I moved all of my Interfaces out of the Molajo repo and placed them into an unbranded repo and listed them as packages on Packagist. I want to encourage sharing at that level for interoperability -- without having to "give credit."  I use all of those Interfaces - there are about 150 of them - as dependencies in Composer for Molajo.

However, it is not true that I advocate Interfaces _over_ concrete implementations. I advocate Interfaces _with_ implementations. In this package alone, I am using six interfaces. In one case, it allows developers to slide in their database package for a foreign key fieldhandler without creating a dependency on the one I am using. (Joomla's).

Even in my own application, I inject interfaces for my own packages, I never name concretes in my classes, only Interfaces. This is so that a dev could use the Molajito rendering package, for example, and whatever escaping option they want. Or, they can use my Event package with their plugin environment, etc. I've worked very hard to build in those flexibilities.

(Where I do use concrete references is in the factory model classes for the IoCC, but that's the plumbing -- and that's exactly where it would be changed if another concrete was preferred. The other place you'll see me naming concretes is in the fieldhandler where the proxy class creates the adapter class. For those "internal to a package" places -- it's okay because it won't be swapped out.)

Here is my frustration. You and I are talking about replacing InputFilter and OutputFilter with something that is easier to extend and updated in terms of PHP 5.3 filtering, etc. I'm offering my code. I'm doing a little analysis in the spreadsheet to map what I have to what must be replaced in the Joomla framework and trying to think about how we can cleverly avoid BC issues and test to ensure it continues to work as expected.

No, I do not believe most developers are not going to pick a framework or package that only has Interfaces unless they are building a framework. They aren't even likely, IMO, to pick a framework that makes good use of Interfaces. What they are more likely to do is select a package that helps them validate and/or filter and/or escape their data without a lot of fuss and do so correctly.

So, to hear you say let's just offer interfaces and everyone can code it their way -- well, Andrew, that may sound fabulous to you and me -- but most developers just want the code.  And no, I don't want to do it my way and have you do it your way. The Joomla framework just needs one solid package that can go the distance the next few years.

I do not care how we do it.

I don't even care if you want to do it yourself. I know what it's like to have a vision and have no choice but to lay the code.

But, I see no reason to have less than or more than one of these in the Joomla framework and I am willing to help but this is you guys architecture and you need to make some choices. I can help provide data or analysis for review, but it's not my framework. It's yours -- and I hope you guys have a vision of where you want this to go.

Here are the Interfaces I am using -- yes, I have filter, validate and escape separated. I also added a response object given the discussion here.

FilterInterface

ValidateInterface

EscapeInterface

Response interface (similar to v1 of FIG Cache)

I don't need flexibility in licensing. I know what Joomla requires. I have signed the agreement. I'm not asking for anything different. I'm not trying to be a library -- you can have raw code -- whatever you want or need.

If any of my code is useful it can be brought into the Joomla framework under the LGPL and changed. It will be treated like any other Joomla code -- OSM's copyright -- your coding standards, your code. You can delete it - restructure it - keep it as it is. I'm still family. Like it, or not. ;-)

[Andrew Eddie]

On 18 April 2014 18:03, Amy Stephen <> wrote:
> If any of my code is useful it can be brought into the Joomla framework
> under the LGPL and changed. It will be treated like any other Joomla code --
> OSM's copyright -- your coding standards, your code. You can delete it -
> restructure it - keep it as it is. I'm still family. Like it, or not. ;-)

Ok, thanks for the offer. As I said, I'd prefer to separate Filtering
and Validation so I hope you won't be offended if when I get around to
making a PR, I do it in that fashion. I may, however, borrow some
ideas from your package if you don't mind.

Regards,
Andrew Eddie

[Amy Stephen]

I'd be lying if I said I wasn't offended. Anyone would be.

There are over 50 data type tests already available for validation, filtering, and escaping.

To use, a developer injects the $fieldhandler using the interface into the class during construction.

To Validate:

$result = $this->fieldhandler('Validate', 'Fieldname', $current_value, 'single, or, chain, of, tests');
if ($result->getReturnValue() === false) {
   $messages = $result->getErrorMessages();
}

To Filter:

$put_this_in_the_data = $this->fieldhandler('Filter', 'Fieldname', $current_value, 'single, or, chain, of, tests');

To Escape the output before rendering:

$ready_for_the_view = $this->fieldhandler('Escape', 'Fieldname', $current_value, 'html');

There are 158 tests -- all successful.

The usage is fully documented.

You didn't even test it or consider it -- nor did any of the framework team or members of the PLT -- or if they did, they didn't share any feedback.

I'm a developer -- a good one with 30+ years of experience -- and a member of this community.  Of course, I am offended. You have a prototype with five tests.

Just not one of the good 'ole boys.
 

Regards,
Andrew Eddie

[Andrew Eddie]

On 19 April 2014 11:29, Amy Stephen <> wrote:
> I'd be lying if I said I wasn't offended. Anyone would be.

Not anyone - just you. I won't put up a PR and will just maintain my
package privately. I have no interest in entering your "my package is
better than yours" debate.

Regards,
Andrew Eddie