I am keen to expand the whois gem so that it handles the craziness of the .co.za TLD a little better. Currently the situation is as follows:
Uniforum, the organisation that administers the .co.za TLD has two systems in place - a legacy system and a new system using EPP. If I try and query a domain still administered under the legacy system or the new EPP system, I get this:
1) Have the whois gem follow the referral url and get the content hosted there and then correctly parse it (for a legacy domain). 2) Use the new coza EPP whois (as per http://registry.net.za/noticeboard.php--> sorry, no permalink ) to get the EPP whois details for the domain (domains administered by EPP only).
I have had a look though the code, and I would like to make my changes where they fit in the best with the architecture, so I have these questions:
- I am thinking I should add a new adapter class for co.za - one which would first fetch the referral url and then either fetch the contents from that referral url or fetch the EPP contents (currently, I think the referral URL is the only way to determine if the domain is an EPP domain or not). - Once the contents have been passed through, I am thinking that I would need to extend the current Parser::WhoisCozaNetZa class to provide better support and perhaps add a Parser::WhoisCozaEPP class?
Am I at least in the right place or should I be looking somewhere else
Hi Rory, thanks for your interest in the whois library.
Let me recap the details to make sure I understood correctly.
- .zadna delegates the management of .co.za domains to http://co.za/ - http://co.za/ currently exposes two whois interfaces - whois.coza.net.za (legacy) for old domains - a web based interface for new domains - the only way to determine the interface is performing the whois request to the legacy system and inspect the response
On Sunday, June 24, 2012 10:04:37 PM UTC+2, Rory McKinley wrote:
> Hello List
> I am keen to expand the whois gem so that it handles the craziness of the . > co.za TLD a little better. Currently the situation is as follows:
> Uniforum, the organisation that administers the .co.za TLD has two > systems in place - a legacy system and a new system using EPP. If I try and > query a domain still administered under the legacy system or the new EPP > system, I get this:
> I would like to extend the behaviour as follows:
> 1) Have the whois gem follow the referral url and get the content hosted > there and then correctly parse it (for a legacy domain). > 2) Use the new coza EPP whois (as per > http://registry.net.za/noticeboard.php --> sorry, no permalink ) to get > the EPP whois details for the domain (domains administered by EPP only).
> I have had a look though the code, and I would like to make my changes > where they fit in the best with the architecture, so I have these > questions:
> - I am thinking I should add a new adapter class for co.za - one which > would first fetch the referral url and then either fetch the contents from > that referral url or fetch the EPP contents (currently, I think the > referral URL is the only way to determine if the domain is an EPP domain or > not). > - Once the contents have been passed through, I am thinking that I would > need to extend the current Parser::WhoisCozaNetZa class to provide better > support and perhaps add a Parser::WhoisCozaEPP class?
> Am I at least in the right place or should I be looking somewhere else
Although, thinking about it, the approach would probably be better to
query the new (EPP) interface first. It always returns data - and if
the domain in question is administered by EPP, then you would not need
to do a second request. If the domain is under legacy admin, you
should be able to determine this by looking at the response from EPP
and then you could make additional requests to get the adiditonal
legacy data.
You would end up with a slower time for requests on "legacy" domains
but your requests to EPP domains would be more efficient.
Am I on the right track in terms of where I would like to make the changes?
On Sun, Jun 24, 2012 at 10:15 PM, Simone Carletti <wep...@gmail.com> wrote:
> Hi Rory,
> thanks for your interest in the whois library.
> Let me recap the details to make sure I understood correctly.
> .zadna delegates the management of .co.za domains to http://co.za/ > http://co.za/ currently exposes two whois interfaces
> whois.coza.net.za (legacy) for old domains
> a web based interface for new domains
> the only way to determine the interface is performing the whois request to
> the legacy system and inspect the response
> Am I right?
> -- Simone
> On Sunday, June 24, 2012 10:04:37 PM UTC+2, Rory McKinley wrote:
>> Hello List
>> I am keen to expand the whois gem so that it handles the craziness of the
>> .co.za TLD a little better. Currently the situation is as follows:
>> Uniforum, the organisation that administers the .co.za TLD has two systems
>> in place - a legacy system and a new system using EPP. If I try and query a
>> domain still administered under the legacy system or the new EPP system, I
>> get this:
>> I would like to extend the behaviour as follows:
>> 1) Have the whois gem follow the referral url and get the content hosted
>> there and then correctly parse it (for a legacy domain).
>> 2) Use the new coza EPP whois (as per
>> http://registry.net.za/noticeboard.php --> sorry, no permalink ) to get the
>> EPP whois details for the domain (domains administered by EPP only).
>> I have had a look though the code, and I would like to make my changes
>> where they fit in the best with the architecture, so I have these questions:
>> - I am thinking I should add a new adapter class for co.za - one which
>> would first fetch the referral url and then either fetch the contents from
>> that referral url or fetch the EPP contents (currently, I think the referral
>> URL is the only way to determine if the domain is an EPP domain or not).
>> - Once the contents have been passed through, I am thinking that I would
>> need to extend the current Parser::WhoisCozaNetZa class to provide better
>> support and perhaps add a Parser::WhoisCozaEPP class?
>> Am I at least in the right place or should I be looking somewhere else
The legacy interface is web-based: A call to the legacy whois provides you with a URL that then provides the detail for the lookup.
The new (EPP) interface can be accessed directly - e.g. from the commandline:
whois -h whois.registry.net.za mckinley.co.za
Apologies - should read the mail more carefully before responding.
Rory
On 06/25/2012 01:14 PM, rorymckin...@gmail.com wrote:
> Although, thinking about it, the approach would probably be better to
> query the new (EPP) interface first. It always returns data - and if
> the domain in question is administered by EPP, then you would not need
> to do a second request. If the domain is under legacy admin, you
> should be able to determine this by looking at the response from EPP
> and then you could make additional requests to get the adiditonal
> legacy data.
> You would end up with a slower time for requests on "legacy" domains
> but your requests to EPP domains would be more efficient.
> Am I on the right track in terms of where I would like to make the changes?
> Regards
> Rory
> On Sun, Jun 24, 2012 at 10:15 PM, Simone Carletti <wep...@gmail.com> wrote:
>> Hi Rory,
>> thanks for your interest in the whois library.
>> Let me recap the details to make sure I understood correctly.
>> .zadna delegates the management of .co.za domains to http://co.za/ >> http://co.za/ currently exposes two whois interfaces
>> whois.coza.net.za (legacy) for old domains
>> a web based interface for new domains
>> the only way to determine the interface is performing the whois request to
>> the legacy system and inspect the response
>> Am I right?
>> -- Simone
>> On Sunday, June 24, 2012 10:04:37 PM UTC+2, Rory McKinley wrote:
>>> Hello List
>>> I am keen to expand the whois gem so that it handles the craziness of the
>>> .co.za TLD a little better. Currently the situation is as follows:
>>> Uniforum, the organisation that administers the .co.za TLD has two systems
>>> in place - a legacy system and a new system using EPP. If I try and query a
>>> domain still administered under the legacy system or the new EPP system, I
>>> get this:
>>> I would like to extend the behaviour as follows:
>>> 1) Have the whois gem follow the referral url and get the content hosted
>>> there and then correctly parse it (for a legacy domain).
>>> 2) Use the new coza EPP whois (as per
>>> http://registry.net.za/noticeboard.php --> sorry, no permalink ) to get the
>>> EPP whois details for the domain (domains administered by EPP only).
>>> I have had a look though the code, and I would like to make my changes
>>> where they fit in the best with the architecture, so I have these questions:
>>> - I am thinking I should add a new adapter class for co.za - one which
>>> would first fetch the referral url and then either fetch the contents from
>>> that referral url or fetch the EPP contents (currently, I think the referral
>>> URL is the only way to determine if the domain is an EPP domain or not).
>>> - Once the contents have been passed through, I am thinking that I would
>>> need to extend the current Parser::WhoisCozaNetZa class to provide better
>>> support and perhaps add a Parser::WhoisCozaEPP class?
>>> Am I at least in the right place or should I be looking somewhere else
That's a great news. It means we just need to update the definition for the
.co.za SLD to use whois.registry.net.za in the tld list and change the
parser accordingly.
-- Simone
On Mon, Jun 25, 2012 at 7:36 PM, Rory McKinley <rorymckin...@gmail.com>wrote:
> Oops. Sorry, missed one thing:
> The legacy interface is web-based: A call to the legacy whois provides you
> with a URL that then provides the detail for the lookup.
> The new (EPP) interface can be accessed directly - e.g. from the
> commandline:
> whois -h whois.registry.net.za mckinley.co.za
> Apologies - should read the mail more carefully before responding.
The only catch with the new interface is that it does not show all the
data that the legacy interface would show.
Given the way that the migration is taking place I would still need
access to the legacy whois interface as well, until the domains have
been migrated. How ugly would it be to provide access to both and have
the code use the interface most pertinent? (And where would be the
best place for me to put the ugliness?)
On Tue, Jun 26, 2012 at 2:28 PM, Simone Carletti <wep...@weppos.net> wrote:
> That's a great news. It means we just need to update the definition for the
> .co.za SLD to use whois.registry.net.za in the tld list and change the
> parser accordingly.
> -- Simone
> On Mon, Jun 25, 2012 at 7:36 PM, Rory McKinley <rorymckin...@gmail.com>
> wrote:
>> Oops. Sorry, missed one thing:
>> The legacy interface is web-based: A call to the legacy whois provides you
>> with a URL that then provides the detail for the lookup.
>> The new (EPP) interface can be accessed directly - e.g. from the
>> commandline:
>> whois -h whois.registry.net.za mckinley.co.za
>> Apologies - should read the mail more carefully before responding.
I tried to run 10 different queries and I haven't been able to spot any
domain using the old interface.
All the queries returned the reference to the new implementation.
Is there any document that explains the migration and how long is going to
take?
rorymckin...@gmail.com> wrote:
> The only catch with the new interface is that it does not show all the
> data that the legacy interface would show.
> Given the way that the migration is taking place I would still need
> access to the legacy whois interface as well, until the domains have
> been migrated. How ugly would it be to provide access to both and have
> the code use the interface most pertinent? (And where would be the
> best place for me to put the ugliness?)
> Also - what is an SLD? :)
> Rory
> On Tue, Jun 26, 2012 at 2:28 PM, Simone Carletti <wep...@weppos.net>
> wrote:
> > That's a great news. It means we just need to update the definition for
> the
> > .co.za SLD to use whois.registry.net.za in the tld list and change the
> > parser accordingly.
> > -- Simone
> > On Mon, Jun 25, 2012 at 7:36 PM, Rory McKinley <rorymckin...@gmail.com>
> > wrote:
> >> Oops. Sorry, missed one thing:
> >> The legacy interface is web-based: A call to the legacy whois provides
> you
> >> with a URL that then provides the detail for the lookup.
> >> The new (EPP) interface can be accessed directly - e.g. from the
> >> commandline:
D'oh! Should have picked up the similarity with TLD :)
To the best of my knowledge, the migration timeline is set by the organisations and individuals who have domains administered by the legacy system - as the process is laborious (each domain needs to be individually transferred), it may take several months for the migration to complete.
While there are still domains that are administered by the "legacy" system, there is unfortunately still some value provided by the old interface. As demonstration, if you compare the output of the new interface ( whois -h whois.registry.net.za <http://whois.registry.net.za> mckinley.co.z <http://mckinley.co.za>a ) with the old interface (http://co.za/cgi-bin/whois.sh?Domain=mckinley&Enter=Enter) - the old interface provides additional information which is not required for EPP (and hence does not show up in the response from the new interface), e.g. admin and technical contact details.
I am in a position where I need to have an interim phase where both are supported, with the legacy domains becoming less and less until only the new interface is in use. Given the freedom to choose, I would prefer just going with the solution you proposed but, sadly, external constraints prevent me from doing this. :(
So, if it is workable, I would prefer to have a solution as follows:
1) Check new whois interface.
2) If domain is an EPP domain - use new interface response.
3) If domain is a legacy domain - get link for the crusty interface - scrape the response and return that.
I am more than happy to write the required grossness for the legacy as well as new system.
PS The example used above (mckinley.co.za) is a "legacy domain". For an example of an EPP domain:
> I tried to run 10 different queries and I haven't been able to spot > any domain using the old interface.
> All the queries returned the reference to the new implementation.
> Is there any document that explains the migration and how long is > going to take?
So - having had a quick look at the code - here is what I propose for my chnage:
1) Create a new adapter: CrappyCozaAdapter (CCA)
2) CCA#request makes a call to legacy co.za whois.
For a legacy domain, the response would be as follows:
"Registered\n%referral
url:http://www.coza.net.za/cgi-bin/whois.sh?Domain=mckinley"
For an EPP domain, the response would be as follows:
"Registered\nNot the default registrar\n%referral
url:http://www.coza.net.za/cgi-bin/whois.sh?Domain=roryisawesome"
3) CCA parses the response and decides which of the two *actual* whois
interfaces it needs to use (it keys off 'Not the default registrar')
For a legacy domain it will scrape
http://www.coza.net.za/cgi-bin/whois.sh?Domain=mckinley For an EPP domain it will go to the new interface and get the data
from there
4) I would create two new parsers for the response
5) If the call in CCA#request (no (2) above) fails - CCA defaults to
using the EPP interface
I have consciously used the legacy whois interface as the first port
of call as it reduces the amount of "parsing" logic that needs to live
in the adapter.
I'm still trying to recover from all this confusion. There is one thing
that still doesn't convince me. Why don't we go straight ahead with a whois
query to whois.registry.net.za?
I tried to perform a query using two domains that, apparently, should
belong to two different interfaces. Both returned a correct value using the
new interface.
> I'm still trying to recover from all this confusion. There is one > thing that still doesn't convince me. Why don't we go straight ahead > with a whois query to whois.registry.net.za > <http://whois.registry.net.za>?
> I tried to perform a query using two domains that, apparently, should > belong to two different interfaces. Both returned a correct value > using the new interface.
The difference between the two is in the information provided.
While a domain (mckinley.co.za) is still on the legacy system, any interaction that I have with it must be done through the legacy system. For the interaction with the legacy system to be successful, I need data that is only available from following the URL provided by the legacy whois interface. I have the legacy output versus the new interface output here: https://gist.github.com/3010274
The bottom section (legacy output) has information such as admin and technical details which are not available through the new (EPP) whois interface (parts of section 2 in the output as well as sections 4 and 5 in their entirety). In an ideal world, I would transfer legacy domains to the EPP system and then the new whois output would be sufficient - but that is not an option.
As far as I know, the data that is *not* provided by the new whois is still required to perform update/deletion operations through the legacy interface - and thus, I am stuck in a hole where I need to support both new and legacy whois :(
I am not sure if I have managed to make the mess any clearer :)
On Thu, Jun 28, 2012 at 11:56 AM, Rory McKinley <rorymckin...@gmail.com>wrote:
> The bottom section (legacy output) has information such as admin and
> technical details which are not available through the new (EPP) whois
> interface (parts of section 2 in the output as well as sections 4 and 5 in
> their entirety). In an ideal world, I would transfer legacy domains to the
> EPP system and then the new whois output would be sufficient - but that is
> not an option.
> As far as I know, the data that is *not* provided by the new whois is
> still required to perform update/deletion operations through the legacy
> interface - and thus, I am stuck in a hole where I need to support both new
> and legacy whois :(
Here we are! Now it starts to make more sense. Thanks and sorry for the
long email exchange.
Speaking from a maintainer point of view, I would say that the output from
the new system would be enough to justify the change. There are parsers
with way less details of this.
However, I can feel your pain and your desire to have the full code merged
into the library. I'm just thinking whether it may be a good idea,
considering the fact that so far I decided to keep scraping and http
interactions outside the scope of this library.
I'm concerned because, once the code will be merged into master, it means
we'll have to officially support this.
I have a proposal for you, let me know what do you think.
Let's package everything into a plugin!
Here's my idea.
1. We can update the current master of whois in order to use the output
from whois.registry.net.za. In most cases, this is enough for the
majority of users.
2. The whois.coza.net.za parser will be removed and a new parser for
whois.registry.net.za will be created to handle the standard response
That's it for the main whois library. But here it comes the whois-coza
plugin. This gem will provide all the features you described before:
1. hook into .co.za definition in order to use a custom adapter
2. the adapter will perform the request, check the response and extract
the best response
3. a new parser will be added to handle the HTTP response
4. in case the response needs to be performed to the new interface, then
the existing parser (bundled in the whois gem) will be used
Users whose want take advantage of this feature, will simply have to
require the whois gem + the whois-coza gem.
> 1. We can update the current master of whois in order to use the
> output from whois.registry.net.za <http://whois.registry.net.za>.
> In most cases, this is enough for the majority of users.
> 2. The whois.coza.net.za <http://whois.coza.net.za> parser will be
> removed and a new parser for whois.registry.net.za
> <http://whois.registry.net.za> will be created to handle the
> standard response
> That's it for the main whois library. But here it comes the whois-coza > plugin. This gem will provide all the features you described before:
> 1. hook into .co.za <http://co.za> definition in order to use a
> custom adapter
> 2. the adapter will perform the request, check the response and
> extract the best response
> 3. a new parser will be added to handle the HTTP response
> 4. in case the response needs to be performed to the new interface,
> then the existing parser (bundled in the whois gem) will be used
> Users whose want take advantage of this feature, will simply have to > require the whois gem + the whois-coza gem.
> Does it work for you?
Yes! :) I am happy with that approach - it makes it easy to throw away the legacy stuff as soon it is no longer required. I can get started on the changes for whois.registry.net.za (the stuff that will form part of the whois gem) - and send you a pull request - if that works for you? :0
> 1. We can update the current master of whois in order to use the
> output from whois.registry.net.za. In most cases, this is enough for
> the majority of users.
> 2. The whois.coza.net.za parser will be removed and a new parser for
> whois.registry.net.za will be created to handle the standard response
> That's it for the main whois library. But here it comes the whois-coza
> plugin. This gem will provide all the features you described before:
> 1. hook into .co.za definition in order to use a custom adapter
> 2. the adapter will perform the request, check the response and
> extract the best response
> 3. a new parser will be added to handle the HTTP response
> 4. in case the response needs to be performed to the new interface,
> then the existing parser (bundled in the whois gem) will be used
> Users whose want take advantage of this feature, will simply have to
> require the whois gem + the whois-coza gem.
> Does it work for you?
> Yes! :) I am happy with that approach - it makes it easy to throw away the
> legacy stuff as soon it is no longer required. I can get started on the
> changes for whois.registry.net.za (the stuff that will form part of the
> whois gem) - and send you a pull request - if that works for you? :0
