[info object methods -all] doesn't return all superclass methods

[nurd...@googlemail.com]

Executive summary:

After creating a selenium webdriver object - either phantomjs or chrome - [info object methods $obj] doesn't include find_element_by_xpath.

Long story:

After a couple of years messing about with other ways to scrape web sites, I have finally got to grips with selenium-tcl. (I probably downloaded it from sourceforge as advertised on the wiki, at wiki.tcl.tk/selenium, but I can't be quite sure.)

I have a cunning plan to wrap the selenium drivers in a class of my own to provide some extra bits and pieces, and have code something like

oo::class create myDriver {
variable m_driver

constructor {} {
set m_driver [selenium::webdrivers::phantomjs::PhantomJSdriver new]

foreach methName [info object methods $m_driver -all] {
oo::objdefine [self object] forward $methName $m_driver $methName
}
.
.
.
}
.
.
.
}

Sadly, however, when I subsequently to

set d [myDriver new]
$d get $someUrl
$d find_element_by_spath $someXPath

My script throws

unknown method "find_element_by_xpath": must be accept_alert, ...

But, selenium::webdrivers::phantomjs::PhantomJSdriver has selenium::WebDriver as a superclass, and selenium::WebDriver has a find_element_by_xpath method.

% info class superclasses selenium::webdrivers::phantomjs::PhantomJSdriver
::selenium::WebDriver
% info class methods selenium::webdrivers::phantomjs::PhantomJSdriver
stop_client start_client service_url
% info class methods selenium::webdrivers::phantomjs::PhantomJSdriver -all
accept_alert active_webelement add_cookie alert_text available_log_types back clear_text close create_container_of_webelements create_webelement css_property current_capabilities current_url current_window_handle delete_all_cookies delete_cookie destroy dismiss_alert execute execute_and_get_value execute_javascript execute_script forward freeze get get_app_cache_status get_attribute get_base64_image get_cookie get_cookies get_location get_log get_rect get_screen_orientation get_screenshot_as_base64 get_screenshot_as_file get_screenshot_as_png get_text get_visible_text get_window_position get_window_size implicitly_wait is_displayed is_enabled is_select_element is_selected location_once_scrolled_into_view maximize_window move_mouse_to name network_connection page_source quit refresh remove_webelement_from_DOM save_screenshot select_element send_keys send_keys_to_alert service_url session_ID set_network_connection set_page_load_timeout set_screen_orientation set_script_timeout set_window_position set_window_size size start_client start_session stop_client submit_form switch_to_default_frame switch_to_frame switch_to_parent_frame switch_to_window tag_name title typewrite wait_until_connectable wait_until_not_connectable window_handles
% info class methods selenium::WebDriver -all
accept_alert active_webelement add_cookie alert_text available_log_types back clear_text click click_and_hold close create_container_of_webelements create_webelement css_property current_capabilities current_url current_window_handle delete_all_cookies delete_cookie destroy dismiss_alert double_click execute execute_and_get_value execute_javascript execute_script find_element find_element_by_class_name find_element_by_css_selector find_element_by_id find_element_by_link_text find_element_by_name find_element_by_partial_link_text find_element_by_tag_name find_element_by_xpath find_elements find_elements_by_class_name find_elements_by_css_selector find_elements_by_id find_elements_by_link_text find_elements_by_name find_elements_by_partial_link_text find_elements_by_tag_name find_elements_by_xpath forward freeze get get_app_cache_status get_attribute get_base64_image get_cookie get_cookies get_location get_log get_rect get_screen_orientation get_screenshot_as_base64 get_screenshot_as_file get_screenshot_as_png get_text get_visible_text get_window_position get_window_size implicitly_wait is_displayed is_enabled is_select_element is_selected location_once_scrolled_into_view maximize_window mouse_down mouse_up move_mouse move_mouse_to move_mouse_to_element name network_connection page_source quit refresh remove_webelement_from_DOM save_screenshot scroll_into_view scroll_to_bottom scroll_to_bottom_infinitely scrolling_position select_element send_keys send_keys_to_alert session_ID set_network_connection set_page_load_timeout set_screen_orientation set_script_timeout set_window_position set_window_size size start_client start_session stop_client submit_form switch_to_default_frame switch_to_frame switch_to_parent_frame switch_to_window tag_name title typewrite wait_until_connectable wait_until_not_connectable window_handles
% info object class oo::Obj66
::selenium::webdrivers::phantomjs::PhantomJSdriver
% info object methods oo::Obj66 -all
accept_alert active_webelement add_cookie alert_text available_log_types back clear_text close create_container_of_webelements create_webelement css_property current_capabilities current_url current_window_handle delete_all_cookies delete_cookie destroy dismiss_alert execute execute_and_get_value execute_javascript execute_script forward freeze get get_app_cache_status get_attribute get_base64_image get_cookie get_cookies get_location get_log get_rect get_screen_orientation get_screenshot_as_base64 get_screenshot_as_file get_screenshot_as_png get_text get_visible_text get_window_position get_window_size implicitly_wait is_displayed is_enabled is_select_element is_selected location_once_scrolled_into_view maximize_window move_mouse_to name network_connection page_source quit refresh remove_webelement_from_DOM save_screenshot select_element send_keys send_keys_to_alert service_url session_ID set_network_connection set_page_load_timeout set_screen_orientation set_script_timeout set_window_position set_window_size size start_client start_session stop_client submit_form switch_to_default_frame switch_to_frame switch_to_parent_frame switch_to_window tag_name title typewrite wait_until_connectable wait_until_not_connectable window_handles

Sorry about the length of this post; the problem doesn't show up with the simple subclass/superclass setup I tried - I hope knowledgeable people may be able to identify what's going on here maybe with reference to the selenium-tcl sources. I'll happily try to construct a simpler example if pointed in the direction of ow to simplify. Similarly, any comments on whether I would do better to wrap the object differently are welcome.

[stefan]

> Sorry about the length of this post; the problem doesn't show up with the simple subclass/superclass setup I tried

Your conclusion seems correct, but your assumptions are wrong. The sought method is provided by a mixin, not a superclass. If it was, it would show up.

This is the structure you are looking at (reduced to the bare minimum):

oo::class create Mixin_For_Element_Retrieval {
method find_element_by_spath {args} {;}
}

oo::class create WebDriver {
method foo {args} {;}
mixin ::Mixin_For_Element_Retrieval
}

oo::class create PhantomJSdriver {
superclass ::WebDriver
}

set obj [PhantomJSdriver new]
$obj find_element_by_spath; # ok
puts [info object methods $obj -all]; # superclass method 'foo' is included, mixin method "find_element_by_spath" is missing.

The doc clearly states that -all should include mixin-provided methods. So, this is worth a bug report to Donal.

For the time being, you might want to create:

* a custom info object mymethods to collect all the method-providing classes of obj and produce your own list.

* use a filter to wrap around calls to the driver object

* consider using a subclass for your custom driver.

design-wise, it is hard to tell what would be best without knowing the actual purpose of wrapping the given driver.

Stefan

[nurd...@googlemail.com]

Thanks for the response, Stefan.

Are you looking at selenium-tcl sources? I'm using selenium 2.3.2, and while a class called Mixin_For_Element_Retrieval certainly suggests that it is intended to be used as a mixin, in my sources it's used as a superclass and I'm pretty sure I haven't changed them. As far as I can see, the documentation says that [info object methods -all] should include superclass methods as well as mixin-provided methods so it's moot.

How do you recommend I report a bug to Donal? Contact him first - e.g. via a StackOverflow posting - or just bash straight in and create a ticket at core.tcl.tk? I hadn't realised that the Tcl 2017 conference was now on, so will certainly leave it a week or so before doing anything.

As to design...

I'd like to run my code both on my Linux laptop and my Windows desktop - hence the attraction of Tcl!

I have observed that the phantomjs webdriver (unclear if that's the right term) process doesn't exit when the Tcl script that created it terminates (or when the selenium object is destroyed), so my application kills it explicitly when appropriate. I can't do that on Windows (via simple [exec ps] and [exec kill] such as I'm using on Linux) without getting into Twapi or similar and I'm happier not doing that, at least for now.

I also _think_ that my code takes significantly longer to execute on Windows than on Linux, and am experimenting with using ChromeDriver on Windows, which at least gives me something to look at while I wait (it also works around the difficulty automatically killing off the webdriver process, since I'll have an obvious window to close).

Furthermore, experiment suggests strongly that I need some kind of wait mechanism after [driver get] commands to ensure that the data has arrived before attempting to process it. In the single scraping application I have so far written to use selenium, I implemented a control structure that I'm fairly proud of that executes a test script, executes a wait script a number of times, backing off a while between tries and finally executes an error script if the number of repetitions is exceeded.

I'd like to lump together the webdriver process culling, the driver selection and the wait mechanism into a re-usable class.

At present, I'm intending to loop through the selenium driver class and its superclasses building a list of methods that need to be forwarded - your first option. I'm not sure how I could use a filter, and unfamiliar with extending supplied classes using suclassing.

Thanks again for you interest - I'd be very grateful for any more suggestions on the design front.

--
Alan

[stefan]

> Thanks for the response, Stefan.
>
> Are you looking at selenium-tcl sources? I'm using selenium 2.3.2, and while a class called Mixin_For_Element_Retrieval certainly suggests that it is intended to be used as a mixin, in my sources it's used as a superclass and I'm pretty sure I haven't changed them.

I got the 2.3.2 zip from SF and voilà:

% package req selenium::phantomjs
0.1
% package req selenium
2.3.2
% info class mixin ::selenium::WebDriver
::selenium::Mixin_For_Element_Retrieval ::selenium::Mixin_For_Scrolling ::selenium::Mixin_For_Mouse_Interaction

% info class superclasses selenium::webdrivers::phantomjs::PhantomJSdriver
::selenium::WebDriver

> How do you recommend I report a bug to Donal? Contact him first - e.g. via a StackOverflow posting - or just bash straight in and create a ticket at core.tcl.tk? I hadn't realised that the Tcl 2017 conference was now on, so will certainly leave it a week or so before doing anything.

He is hanging out on IRC (conf just started), but a ticket on core.tcl.tk is certainly the way to go.

> I'd be very grateful for any more suggestions on the design front.

Once you got a draft, open a thread, and I will be happy to comment.

Cheers,
Stefan

[Donal K. Fellows]

On 18/10/2017 04:04, stefan wrote:
> Your conclusion seems correct, but your assumptions are wrong. The
> sought method is provided by a mixin, not a superclass. If it was, it
> would show up.

That's definitely a bug. :-( Here's a simple test case:

oo::class create C {
method c {} {}
}

oo::class create D {
mixin C
method d {} {}
unexport destroy
}

oo::class create E {
superclass D
method e {} {}
}

puts [info class methods E -all]
E create e1
puts [info object methods e1 -all]

The problem? Neither of the introspectors find the ‘c’ method. Bug is
present in all versions of 8.6 that I've checked with, and I've no
reason to expect anything different elsewhere.

The good thing is that we've now got a reasonable test case.

Donal.
--
Donal Fellows — Tcl user, Tcl maintainer, TIP editor.

[Donal K. Fellows]

On 18/10/2017 10:41, Donal K. Fellows wrote:
> The good thing is that we've now got a reasonable test case.

And this is now an issue.

https://core.tcl.tk/tcl/tktview/1a56550e96

[nurd...@googlemail.com]

Ahh! The method is provided by a mixin to a superclass. I hand't spotted that.

effects: Deafening sound of penny dropping.

Thanks Stefan, Donal.

Would you like me to raise a ticket, Donal?

Alan

[nurd...@googlemail.com]

Oops. Thanks again Donal.

I've worked around the problem by forwarding methods defined by the superclasses, [info class methods D] does pick up method c in Donal's example.

--
Alan

[Donal K. Fellows]

On 18/10/2017 11:00, nurd...@googlemail.com wrote:
> Oops. Thanks again Donal.

And it's now a fixed bug; I'd slipped up in a performance enhancing
rearrangement of the code and the result just didn't do what I'd
intended. Fix will go in the next Tcl releases when they come out.

[nurd...@googlemail.com]

Thank you again

--
Alan