Re: [Artefactual] Fwd: 1.0.8 evaluation by ICA

[Jesús García Crespo]

Hi all!

On Thu, Nov 26, 2009 at 2:58 PM, Peter Van Garderen
<pe...@artefactual.com> wrote:
> FYI: I added (E) and (F) to our list of Validation requirements based on the
> list below.

(E) ISAD: The software should control whether the links between
hierarchical levels are logical (for example, the software should warn
the user if he links a fonds-level description as a child-level unit
of a file-level description).

(F) ISAD: The software should control the compliance of the dates of
the low-level units with the dates of the high-level units.

I would like to discuss with Jack how this could be solved. I was
trying logical validators [1] to combine our own validators. For
example, case (F):

$validatorSchema->dates = new sfValidatorAnd(array(
new QubitValidatorCountable(...),
new QubitValidatorDate(...)));

Of course, QubitValidatorDate (any name suggestion?) should be
implemented to check if requirement (F) is being fulfilled.

Thoughts?

[1]: http://www.symfony-project.org/forms/1_2/en/02-Form-Validation#chapter_02_logical_validators

--
Jesús García Crespo

[Jack Bates]

Hi Jesus!

On Nov 26, 6:48 pm, Jesús García Crespo <cor...@sevein.com> wrote:
> Hi all!
>
> On Thu, Nov 26, 2009 at 2:58 PM, Peter Van Garderen
>
> <pe...@artefactual.com> wrote:
> > FYI: I added (E) and (F) to our list of Validation requirements based on the
> > list below.
>
> (E) ISAD: The software should control whether the links between
> hierarchical levels are logical (for example, the software should warn
> the user if he links a fonds-level description as a child-level unit
> of a file-level description).
>
> (F) ISAD: The software should control the compliance of the dates of
> the low-level units with the dates of the high-level units.
>
> I would like to discuss with Jack how this could be solved. I was
> trying logical validators [1] to combine our own validators. For
> example, case (F):
>
> $validatorSchema->dates = new sfValidatorAnd(array(
>       new QubitValidatorCountable(...),
>       new QubitValidatorDate(...)));
>
> Of course, QubitValidatorDate (any name suggestion?) should be
> implemented to check if requirement (F) is being  fulfilled.

Do you feel your work on the simple checks for mandatory elements on
the show pages is ready to commit? Maybe we haven't implemented checks
for all the mandatory elements yet, but I think most of them are
simple, and we have a pattern that works at least for those? If you
think it's a good idea, let's commit that work and move on to the less
simple checks : )

Thanks for adding (E) and (F) to the wiki page Peter,
http://qubit-toolkit.org/wiki/index.php?title=Validation

Jesus or Peter - can you refine (F), compliance of the dates of the
low-level units with the dates of the high-level units, on the wiki
page? Maybe add some detail about what you think should be checked -
or maybe some examples of noncompliant dates we should catch

Also Jesus or Peter - if there's any part of ISAD(G) which relates to
(E) or (F), please add links to the standard from the validation wiki
page. To get us started, I added links from the mandatory elements
sections of the validation page to the corresponding sections of the
standards

> Thoughts?
>
> [1]:http://www.symfony-project.org/forms/1_2/en/02-Form-Validation#chapte...
>
> --
> Jesús García Crespo

[Jesús García Crespo]

Hi Jack!

> Do you feel your work on the simple checks for mandatory elements on
> the show pages is ready to commit? Maybe we haven't implemented checks
> for all the mandatory elements yet, but I think most of them are
> simple, and we have a pattern that works at least for those? If you
> think it's a good idea, let's commit that work and move on to the less
> simple checks : )

Done!

I have some doubts:

- Validation messages, any rule? I see strange to repeat five times "This field is essential for...". Wouldn't be easyer to read for users something like "Field name: mandatory field"?

- ICA-ISDIAH :: Location and address(es)
This is a mandatory field. However, we have ContactInformation objects where other fields also belongs to ContactInformation, I mean, "Telephone, fax, email" and "Contact person". Then, what exactly we should check here? If we have a primary contact information with at least the fields related to "Location and address"?
http://ica-atom.org/docs/index.php?title=RS-3#5.2

> Jesus or Peter - can you refine (F), compliance of the dates of the
> low-level units with the dates of the high-level units, on the wiki
> page? Maybe add some detail about what you think should be checked -
> or maybe some examples of noncompliant dates we should catch
>
> Also Jesus or Peter - if there's any part of ISAD(G) which relates to
> (E) or (F), please add links to the standard from the validation wiki
> page. To get us started, I added links from the mandatory elements
> sections of the validation page to the corresponding sections of the
> standards

I will do it tomorrow.

Thank you in advance and good night,

--
Jesús García Crespo

[Jack Bates]

On Nov 27, 7:43 pm, Jesús García Crespo <cor...@sevein.com> wrote:
> Hi Jack!
>
> > Do you feel your work on the simple checks for mandatory elements on
> > the show pages is ready to commit? Maybe we haven't implemented checks
> > for all the mandatory elements yet, but I think most of them are
> > simple, and we have a pattern that works at least for those? If you
> > think it's a good idea, let's commit that work and move on to the less
> > simple checks : )
>
> Done!
>
> I have some doubts:
>
> - Validation messages, any rule? I see strange to repeat five times "This
> field is essential for...". Wouldn't be easyer to read for users something
> like "Field name: mandatory field"?

Evelyn or Peter - thoughts? So far we're discussing,

* Messages borrowed from the standard, e.g. "This archival description
is untitled. A title is considered essential for international
exchange of descriptive information." These look strange when repeated
five times, if five elements are missing
* Something easier to read, e.g. "Identifier - This is a mandatory
field."
* Other suggestions?

You can see examples of each in the latest revision of Qubit - create
a blank archival description and check out the ISAD show page

Anyone else have other suggestions we should consider? I don't feel
too strongly either way...

> - ICA-ISDIAH :: Location and address(es)
> This is a mandatory field. However, we have ContactInformation objects where
> other fields also belongs to ContactInformation, I mean, "Telephone, fax,
> email" and "Contact person". Then, what exactly we should check here? If we
> have a primary contact information with at least the fields related to
> "Location and address"?http://ica-atom.org/docs/index.php?title=RS-3#5.2

Sounds good to me - anyone else have some thoughts? Peter or Evelyn?

[Jesús García Crespo]

Hi Jack,

On Sat, Nov 28, 2009 at 7:19 PM, Jack Bates <jack....@gmail.com> wrote:

> - ICA-ISDIAH :: Location and address(es)
> This is a mandatory field. However, we have ContactInformation objects where
> other fields also belongs to ContactInformation, I mean, "Telephone, fax,
> email" and "Contact person". Then, what exactly we should check here? If we
> have a primary contact information with at least the fields related to
> "Location and address"?http://ica-atom.org/docs/index.php?title=RS-3#5.2

Sounds good to me - anyone else have some thoughts? Peter or Evelyn?

Well, I took this case for our first less simple validation, r3991.

http://code.google.com/p/qubit-toolkit/source/detail?r=3991

First check: is there a primary contact? Otherwise, validation error!

Second check: see streetAddress, city, region, countryCode and postalCode values from primary contact; if all of them are empty, validation error!

The logic is in lib/QubitValidatorIsdiahLocationAndAddress.class.php.

Sorry Jack, I have not enough imagination to solve this without producing new validators, :-).

Please, feel free to revert these changes if you have a better solution.

Have a nice weekend,

--
Jesús García Crespo

[Jack Bates]

On Nov 28, 6:14 pm, Jesús García Crespo <cor...@sevein.com> wrote:
> Well, I took this case for our first less simple validation, r3991.http://code.google.com/p/qubit-toolkit/source/detail?r=3991

>
> First check: is there a primary contact? Otherwise, validation error!
> Second check: see streetAddress, city, region, countryCode and postalCode
> values from primary contact; if all of them are empty, validation error!
>
> The logic is in lib/QubitValidatorIsdiahLocationAndAddress.class.php.
>
> Sorry Jack, I have not enough imagination to solve this without producing
> new validators, :-).

http://www.sfu.ca/~jdbates/tmp/qubit/200912010/patch

^ this was my first thought - then i realized it checks that all the
fields of the primary contact are present - not *any* of the fields :
(

how about this one? http://www.sfu.ca/~jdbates/tmp/qubit/200912011/patch

[Jesús García Crespo]

Hi Jack,

On Tue, Dec 1, 2009 at 6:15 PM, Jack Bates <jack....@gmail.com> wrote:

^ this was my first thought - then i realized it checks that all the
fields of the primary contact are present - not *any* of the fields :
(

how about this one? http://www.sfu.ca/~jdbates/tmp/qubit/200912011/patch

Can you take a look at r4049? I had to modify your one a little bit to add checking for existence of a contact information through sfValidatorAnd and QubitValidatorCountable. There is no validation for checking if a contact information was set as primary contact when is the only one, but as I saw at QubitActor class, when we don't have a primary contact, we just get one. But, when we have multiple contact informations but anyone was set as primary contact, which one are we really validating?

This problem got me stuck much time, :-(.

--
Jesús García Crespo

[Jesús García Crespo]

And... while we discuss r4049, I also would like to talk about ISAD fields 'level of description' and 'dates'.

I added some thoughts at the wiki page.

http://www.qubit-toolkit.org/wiki/index.php?title=Validation#ICA-ISAD.28G.29

These are all the difficult cases.

Regards,

--
Jesús García Crespo

[Jack Bates]

@revision 4049 - nice work using the symfony validators where
possible : )

@primary contact - according to isdiah, what are the rules we need to
check? peter or david: thoughts?

lets describe the rules we're checking and additional/alternative
rules on the validation wiki page - to get us started i described the
"location and address(es)" rules i think we're checking -
http://qubit-toolkit.org/wiki/index.php?title=Validation#Location_and_address.28es.29

^ did i get that right? should we be checking different rules - or do
you think our code isn't correctly implementing these rules?

On Dec 3, 11:29 am, Jesús García Crespo <cor...@sevein.com> wrote:
> Hi Jack,
>

> On Tue, Dec 1, 2009 at 6:15 PM, Jack Bates <jack.ba...@gmail.com> wrote:
>
> > ^ this was my first thought - then i realized it checks that all the
> > fields of the primary contact are present - not *any* of the fields :
> > (
>

> > how about this one?http://www.sfu.ca/~jdbates/tmp/qubit/200912011/patch

>
> Can you take a look at

> r4049?<http://code.google.com/p/qubit-toolkit/source/detail?r=4049>I

[Jack Bates]

cool - i gave feedback on the wiki page - is it enough?

On Dec 3, 11:50 am, Jesús García Crespo <cor...@sevein.com> wrote:
> And... while we discuss r4049, I also would like to talk about ISAD fields
> 'level of description' and 'dates'.
>
> I added some thoughts at the wiki page.
>

> http://www.qubit-toolkit.org/wiki/index.php?title=Validation#ICA-ISAD...

[Evelyn McLellan]

Hi,

Just started looking at validation. One thing I noticed is that for
lower levels of description I am getting the validation message "This
archival description requires at least one creator." However, the
lower levels inherit the creator from the parent level, so the message
shouldn't appear.

Jesús, I really like the work you've done on this.

Evelyn

[Evelyn McLellan]

I've added some comments to http://qubit-toolkit.org/wiki/index.php?title=Validation#Show_pages
and http://qubit-toolkit.org/wiki/index.php?title=Validation#Edit_pages.

Evelyn

[Jesús García Crespo]

Hi Evelyn,

On Thu, Dec 10, 2009 at 5:57 PM, Evelyn McLellan <epmcl...@gmail.com> wrote:

I've added some comments to http://qubit-toolkit.org/wiki/index.php?title=Validation#Show_pages
and http://qubit-toolkit.org/wiki/index.php?title=Validation#Edit_pages.

Thanks for your comments. I am in doubt with ISDIAH - Location and address. For us, this is a part of "Contact information". When there is not a contact information filled, I show this validation error message:

Contact information - This is a mandatory field.

Sometimes, a contact information is added but any of of the next fields were not filled: city, country, postal code, region and street address. Those are the fields related to Location and address. In this case, the message is:

Contact information - At least, you must fill one of the next fields: city, country, postal code, region or street address.

Do you think those message are correct?

Thank you in advance,

--
Jesús García Crespo

[Jesús García Crespo]

Hi,

Btw, Jack, I updated a bit the way to validate the level of description field. This is compatible with other languages. If the current information object shares the level of description with one of it ancestors, the error is raised.

http://qubit-toolkit.org/wiki/index.php?title=Validation#Level_of_description

This is the patch:

http://www.sevein.com/tmp/lodesc-patch.diff

What do you think? Should we copy sfValidatorBlacklist to our repository or should we install sfFormExtraPlugin, which includes many other validators and widgets that probably won't be used here?

Thank you!

--
Jesús García Crespo

[Jack Bates]

nice work jesus -

i would add the sfFormExtraPlugin as an svn:external, the same way you
added the sfWebBrowserPlugin

is that ok? you remember how to do that?

On Dec 16, 7:27 am, Jesús García Crespo <cor...@sevein.com> wrote:
> Hi,
>
> Btw, Jack, I updated a bit the way to validate the level of description
> field. This is compatible with other languages. If the current information
> object shares the level of description with one of it ancestors, the error
> is raised.
>

> http://qubit-toolkit.org/wiki/index.php?title=Validation#Level_of_des...

[Jesús García Crespo]

Hi Jack!

On Wed, Dec 16, 2009 at 5:38 PM, Jack Bates <jack....@gmail.com> wrote:

i would add the sfFormExtraPlugin as an svn:external, the same way you
added the sfWebBrowserPlugin

 

It's done. Tomorrow, I will try to resolve the dates validator. I think it is the last one left.

Thank you,

--
Jesús García Crespo

[Peter Van Garderen]

Hi Jes�s,

Yes, these are tough. I've added my feedback on the wiki page, copied here below
for any follow-up discussion:

The Level of Description hierarchy needs to stay flexible, allowing users to
modify it for institutional or national practices. Therefore, we can't hardcode
a Level hierarchy. The application will ship with our current default set of
Levels which the user can then revise via the standard Term edit screens. This
order should be used to validate the Level of Description hierarchy.

In an earlier release we had a 'sort' column for Terms. We got rid of that in
favour of (eventually) using the Nested set 'lft' & 'rgt' values for sorting
amongst siblings. Now that the Term object is using a nested set we have to
implement this.

The ideal way to implement that is to allow drag-and-drop in the hierarchy tree
(like we currently allow to change parentId) but then specific to the siblings
in the branch. Then we can use the 'lft' value of the siblings as the hierarchy
sort order and we can set a validation rule that ensures a particular
information object's term->lft value is not less than any of its ancestors.

--peter

[Jesús García Crespo]

Hi Peter,

On Wed, Dec 23, 2009 at 9:20 PM, Peter Van Garderen <pe...@artefactual.com> wrote:

In an earlier release we had a 'sort' column for Terms. We got rid of that in
favour of (eventually) using the Nested set 'lft' & 'rgt' values for sorting
amongst siblings. Now that the Term object is using a nested set we have to
implement this.

The ideal way to implement that is to allow drag-and-drop in the hierarchy tree
(like we currently allow to change parentId) but then specific to the siblings
in the branch. Then we can use the 'lft' value of the siblings as the hierarchy
sort order and we can set a validation rule that ensures a particular
information object's term->lft value is not less than any of its ancestors.

I was working on this. Sorting could be implemented on treeview, as you suggested (see the example [1]), or in the table list shown at the left side. I am not sure what is the best solution because treeview is being hidden now when showing the items in a taxonomy. We also can revert this and show the treeview always.

BTW, I have two problems:

1) Javascript files are not being loaded when they are required from a component. I created an issue page, http://code.google.com/p/qubit-toolkit/issues/detail?id=1185. But I wasn't able to fix it.

2) I am not sure how to move siblings in the database. It seems Propel offers some methods for NestedSets, but they are not available from our classes. Their names are "moveToNextSiblingOf(...)" and "moveToPrevSiblingOf(...)". Should I implement those methods in our class QubitTerm? I wrote the logic in moveAction and some JavaScript AJAX requests, but it needs those methods or write my owns.

Give me a hand, please!

[1] - http://www.coderfoo.com/examples/treeview-dragdrop.html

Regards,

--
Jesús García Crespo

[Peter Van Garderen]

>
> I was working on this. Sorting could be implemented on treeview, as you
> suggested (see the example [1]), or in the table list shown at the left
> side. I am not sure what is the best solution because treeview is being
> hidden now when showing the items in a taxonomy. We also can revert this
> and show the treeview always.

Yes, I think the drag-n-drop in the treeview, as per your example, is the way we
will (eventually) want to implement this. Unfortunately, it looks like we may
have to settle for something simpler in the meanwhile (see comments below).

>
> BTW, I have two problems:
>
> 1) Javascript files are not being loaded when they are required from a
> component. I created an issue page,
> http://code.google.com/p/qubit-toolkit/issues/detail?id=1185. But I
> wasn't able to fix it.

Jack is looking into this.

> 2) I am not sure how to move siblings in the database. It seems Propel
> offers some methods for NestedSets, but they are not available from our
> classes. Their names are "moveToNextSiblingOf(...)" and
> "moveToPrevSiblingOf(...)". Should I implement those methods in our
> class QubitTerm? I wrote the logic in moveAction and some JavaScript
> AJAX requests, but it needs those methods or write my owns.

Yes, we will want to have these methods available in the Qubit NestedSet.
However, adding that method will likely require some significant development and
testing time. This raises the question of how much additional development time
we want to invest in our own custom Nested Set implementation in light of the
fact that these are now more robustly supported in both Propel and Doctrine. The
future of our ORM is a discussion we will need to revisit after our current
January 4th deadline.

Regardless, the ability to sort the Level of Description taxonomy and then
enforce those ordered hierarchy levels as a Validation rule, will be a high
priority requirement for 1.1. If we can't provide that via NestedSet ordering
and treeview drag-n-drop then we may have to look at an alternative work-around
(i.e. adding a sort/weight property for the Level Of Description taxonomy)

In the meanwhile, I think we should just look for and enforce the Levels of
Description and their respective order as specifically mentioned as examples in
rule 1.3.4 of ISAD(G). Additionally, we include 'collection' as a top-level
alternative to 'fonds' as this is supported by ISAD. "It is assumed that the
same rules used to describe a fonds and its parts may be applied to the
description of a collection" (p8)

We can simply ignore any Levels that may have been custom added by users in our
validation checks for now. So for the ISAD Validation rule we just make sure
that these rules are respected:

* fonds cannot be child of collection
* collection cannot be child of fonds
* sub-fonds cannot be child of collection
* all levels in both hierarchy option 1 or option 2 are optional. The only logic
we are validating is that an ancestor level is not added as a descendant (e.g. a
fonds cannot be a child of a file, a series cannot be a child of an item, a
collection cannot be a child of a series, etc)

hierarchy option 1
===================
Fonds
^
Sub-fonds
^
Series
^
Sub-series
^
File
^
Item

hierarchy option 2
==================
Collection
^
Series
^
Sub-series
^
File
^
Item

>
> Regards,
>
> --

> Jes�s Garc�a Crespo
>

> --
>
> You received this message because you are subscribed to the Google
> Groups "Qubit Toolkit Developers" group.
> To post to this group, send email to qubi...@googlegroups.com.
> To unsubscribe from this group, send email to
> qubit-dev+...@googlegroups.com.
> For more options, visit this group at
> http://groups.google.com/group/qubit-dev?hl=en.

[Jesús García Crespo]

Hi Peter,

First of all, thank you for all the details you wrote me!

On Tue, Dec 29, 2009 at 10:02 PM, Peter Van Garderen <pe...@artefactual.com> wrote:

Regardless, the ability to sort the Level of Description taxonomy and then
enforce those ordered hierarchy levels as a Validation rule, will be a high
priority requirement for 1.1. If we can't provide that via NestedSet ordering
and treeview drag-n-drop then we may have to look at an alternative work-around
(i.e. adding a sort/weight property for the Level Of Description taxonomy)

Ok, I created a new issue.

http://code.google.com/p/qubit-toolkit/issues/detail?id=1186

 

We can simply ignore any Levels that may have been custom added by users in our
validation checks for now. So for the ISAD Validation rule we just make sure
that these rules are respected:

* fonds cannot be child of collection
* collection cannot be child of fonds
* sub-fonds cannot be child of collection
* all levels in both hierarchy option 1 or option 2 are optional. The only logic
we are validating is that an ancestor level is not added as a descendant (e.g. a
fonds cannot be a child of a file, a series cannot be a child of an item, a
collection cannot be a child of a series, etc)

Alright, I did this in r4309. However, I wasn't sure what is more correct when checking that "an ancestor level is not added as a descendant". Do you think this should be checked against the parent level of description or against all the values collected from all the ancestors? r4309 checks it in the first way but could be improvable easily.