System Invalid Cast Exception C#

[Jan de Lange]

Starting out with Selenium. Trying to retrieve data from a table. It's almost a straight copy of C# example code on the Selenium documents page http://docs.seleniumhq.org/docs/03_webdriver.jsp. First I search for labels by class name and collect results in IList<IWebElements> then I use the IJavaScriptExecution code example to get the values at these class labels and store those also in an IList<IWebELements> container. This is where I get the Invalid Cast Exception. Additional information: Unable to cast object of type 'System.Collections.ObjectModel.ReadOnlyCollection`1[System.Object]' to type 'System.Collections.Generic.IList`1[OpenQA.Selenium.IWebElement]'.

My code is as follows:
 

          // Find elements by class name
            IList<IWebElement> labels = driver.FindElements(By.ClassName("text"));
 
            // get all input elements for every class label
            IList<IWebElement> labVals = (IList<IWebElement>)((IJavaScriptExecutor)driver).ExecuteScript(
                "var labels = arguments[0], labVals = []; for (var i=0; i < labels.length; i++){" +
                "labVals.push(document.getElementById(labels[i].getAttribute('for'))); } return labVals;", labels);

Why does this not work in my case?

 
 

[Serguei Kouzmine]

Hello Jan

The IJavaScriptExecutor.ExecuteScript returns the string and the calling code needs to explicitly find the elements that the javascript provides information.

A standard example:

1 find element in some way
2 run  javascript to build the xpath to the element just found
3 find the element by xpath returned by step 2

There may be some additional activities between the steps 2 and 3 this is why step 2 is called:

[string]$script = @"
function getPathTo(element) {
    if (element.id!=='')
        return '*[@id="'+element.id+'"]';
    if (element===document.body)
        return element.tagName;

    var ix= 0;
    var siblings= element.parentNode.childNodes;
    for (var i= 0; i<siblings.length; i++) {
        var sibling= siblings[i];
        if (sibling===element)
            return getPathTo(element.parentNode)+'/'+element.tagName+'['+(ix+1)+']';
        if (sibling.nodeType===1 && sibling.tagName===element.tagName)
            ix++;
    }
}
return getPathTo(arguments[0]);
"@
$result = (([OpenQA.Selenium.IJavaScriptExecutor]$selenium).ExecuteScript($script,$element,'')).ToString()

....

  $xpath = ('//{0}' -f $result)

  [void]$wait.Until([OpenQA.Selenium.Support.UI.ExpectedConditions]::ElementExists([OpenQA.Selenium.By]::XPath($xpath)))

Hope that helps - you may use the above code to return xpaths of the labels
Thanks
Serguei Kouzmine

[Jan de Lange]

Hi Serguei,

Thanks for helping me out. Unfortunately, I find it difficult to relate your answer to my question. I should have mentioned I am using C#, so my code is a C# implementation of Selenium calls.

1. Find the element (there are actually about 300 on the page I am calling WSJ. I need the "text" classes and the "num" classes to capture the data for further processing), this is what is done in the first bit of my code: find the elements and put them in a list. This is a copy from the Selenium website (C#) and actually seems to work. It finds 111 text classes and 183 num classes.
2. Then according to the Selenium example, the next call involving the IJavaScriptExecutor should retrieve the dynamically generated data at the location of the elements found in the previous step and store these in a List of IWebElements, which I have called labVals. This code however does not work.

I think what would help me (as a first time user) is an alternative C# code for the second step. It is now all pressed into one single function call which is difficult to debug or change. I think it would be easier if this was programmed out in discrete steps allowing format checking (possibly casting) and debugging.

Here is my full C# code, if you want to try it out yourself. Not sure if the output to console works, I haven't gotten that far due to the cast error.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
// selenium includes
using OpenQA.Selenium;
using OpenQA.Selenium.Firefox;
using OpenQA.Selenium.Support.UI; // Requires reference to WebDriver.Support.dll (separate NUGet package)

namespace SeleniumTest
{
    class Program
    {
        static void Main(string[] args)
        {
            // Create a new instance of the Firefox driver.
            IWebDriver driver = new FirefoxDriver();

            driver.Navigate().GoToUrl("http://online.wsj.com/mdc/public/page/2_3022-govtbonds.html?mod=topnav_2_3004");

            //WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));

            // Find elements by class name
            IList<IWebElement> labels = driver.FindElements(By.ClassName("text"));

            IList<IWebElement> numbers = driver.FindElements(By.ClassName("num"));

            // get all input elements for every class label
            IList<IWebElement> labVals = (IList<IWebElement>)((IJavaScriptExecutor)driver).ExecuteScript(
                "var labels = arguments[0], labVals = []; for (var i=0; i < labels.length; i++){" +
                "labVals.push(document.getElementById(labels[i].getAttribute('for'))); } return labVals;", labels);

             IList<IWebElement> numVals = (IList<IWebElement>)((IJavaScriptExecutor)driver).ExecuteScript(
                "var numbers = arguments[0], numVals = []; for (var i=0; i < numbers.length; i++){" +
                "numVals.push(document.getElementById(numbers[i].getAttribute('for'))); } return numVals;", numbers);

            // Wait for the page to load, timeout after 10 seconds
            //WebDriverWait wait = new WebDriverWait(driver, TimeSpan.FromSeconds(10));

            Console.WriteLine("num labels read " + labels.Count() + " num values read " + numbers.Count());
            int nRows = labels.Count() / 2;
            int i = 0;
            while (i < nRows)
            {
                Console.Write(labVals[i].ToString() + "," + labVals[i + 1].ToString());
                for (int j = i/2 * 5; j < i/2 * 5 + 5; j++)
                    Console.Write("," + numVals[j].ToString());
                Console.WriteLine();
                i += 2;
            }

            //Close the browser
            // driver.Quit();
            Console.ReadKey();
        }
    }
}

[Jim Evans]

You're returning back an array of strings from your JavaScript call, and trying to cast it to IList<IWebElement>. It's highly unsurprising that you're getting a cast exception. The 'for' attribute of an HTML element holds IDs, not element references.

[Jan de Lange]

@Jim Evans. You are correct. Now that I look at it again it seems obvious, but I didn't get it before.

The cast error occurs because the IJavascriptExecutor outputs the general System.Object class MSDN which I then try to cast to an IWebElement. This may work in some cases, but inmy case it does not. Changing the receiving IList to IList<Object> solves the cast exception. With this the code runs, and then I found out with the debugger that all the data is captured with the first part of the code in the Labels list. The IJavaScriptExecutor returns null items only. So the second step is not required in my case.

[Serguei Kouzmine]

Hello
Here is a quick c# extension class, that casts things.

I use it not for elements but for timings, but it sure may unblock you.


-- snip
using OpenQA.Selenium;
using OpenQA.Selenium.Chrome;
using OpenQA.Selenium.Remote;

namespace WebTester
{

    public static class Extensions
    {
        static int cnt = 0;

        public static T Execute<T>(this IWebDriver driver, string script)
        {
            return (T)((IJavaScriptExecutor)driver).ExecuteScript(script);
        }

        public static List<Dictionary<String, String>> Performance(this IWebDriver driver)
        {
            // NOTE: performance.getEntries is only with Chrome
            // performance.timing is available for FF and PhantomJS

            string performance_script = @"
var ua = window.navigator.userAgent;

if (ua.match(/PhantomJS/)) {
    return 'Cannot measure on ' + ua;
} else {
    var performance =
        window.performance ||
        window.mozPerformance ||
        window.msPerformance ||
        window.webkitPerformance || {};

    // var timings = performance.timing || {};
    // return timings;
    var network = performance.getEntries() || {};
    return network;
}
";
            List<Dictionary<String, String>> result = new List<Dictionary<string, string>>();
            IEnumerable<Object> raw_data = driver.Execute<IEnumerable<Object>>(performance_script);

            foreach (var element in (IEnumerable<Object>)raw_data)
            {
                Dictionary<String, String> row = new Dictionary<String, String>();
                Dictionary<String, Object> dic = (Dictionary<String, Object>)element;
                foreach (object key in dic.Keys)
                {
                    Object val = null;
                    if (!dic.TryGetValue(key.ToString(), out val)) { val = ""; }
                    row.Add(key.ToString(), val.ToString());
                }
                result.Add(row);
            }
            return result;
        }

        public static void WaitDocumentReadyState(this IWebDriver driver, string expected_state, int max_cnt = 10)
        {
            cnt = 0;
            var wait = new OpenQA.Selenium.Support.UI.WebDriverWait(driver, TimeSpan.FromSeconds(30.00));
            wait.PollingInterval = TimeSpan.FromSeconds(0.50);
            wait.Until(dummy =>
            {
                string result = driver.Execute<String>("return document.readyState").ToString();
                Console.Error.WriteLine(String.Format("result = {0}", result));
                Console.WriteLine(String.Format("cnt = {0}", cnt));
                cnt++;
                // TODO: match
                return ((result.Equals(expected_state) || cnt > max_cnt));
            });
        }

        public static void WaitDocumentReadyState(this IWebDriver driver, string[] expected_states, int max_cnt = 10)
        {
            cnt = 0;
            Regex state_regex = new Regex(String.Join("","(?:", String.Join("|",expected_states),  ")"),
                                          RegexOptions.IgnoreCase    | RegexOptions.IgnorePatternWhitespace    | RegexOptions.Compiled);
            var wait = new OpenQA.Selenium.Support.UI.WebDriverWait(driver, TimeSpan.FromSeconds(30.00));
            wait.PollingInterval = TimeSpan.FromSeconds(0.50);
            wait.Until(dummy =>
            {
                string result = driver.Execute<String>("return document.readyState").ToString();
                Console.Error.WriteLine(String.Format("result = {0}", result));
                Console.WriteLine(String.Format("cnt = {0}", cnt));
                cnt++;
                return ((state_regex.IsMatch(result) || cnt > max_cnt));
            });
        }
    }

    [TestClass]
    public class Monitor
    {
        private static string hub_url = "http://localhost:4444/wd/hub";
        private static IWebDriver selenium_driver;
        private static string step_url = "http://www.carnival.com/";
        private static string[] expected_states = { "interactive", "complete" };
        private static int max_cnt = 10;
        private static string tableName = "";
        private static string dataFolderPath;
        private static string database;
        private static string dataSource;

        public static void Main(string[] args)
        {

            dataFolderPath = Directory.GetCurrentDirectory();
            database = String.Format("{0}\\data.db", dataFolderPath);
            dataSource = "data source=" + database;
            tableName = "product";
            // driver = new ChromeDriver();
            Console.WriteLine("Starting..");
            // TestConnection();
            createTable();
            // ActiveState Remote::Selenium:Driver
            // selenium_driver = new RemoteWebDriver(new Uri(hub_url), DesiredCapabilities.Firefox());
            selenium_driver = new RemoteWebDriver(new Uri(hub_url), DesiredCapabilities.Chrome());

            selenium_driver.Navigate().GoToUrl(step_url);
            selenium_driver.WaitDocumentReadyState(expected_states);
            List<Dictionary<String, String>> result = selenium_driver.Performance();
            var dic = new Dictionary<string, object>();

            foreach (var row in result)
            {
                dic["caption"] = "dummy";
                foreach (string key in row.Keys)
                {


                    if (Regex.IsMatch(key, "(name|duration)"))
                    {

                        Console.Error.WriteLine(key + " " + row[key]);

                        if (key.IndexOf("duration") > -1)
                        {
                            dic[key] = (Double)Double.Parse(row[key]);
                        }
                        else
                        {
                            dic[key] = (String)row[key];
                        }
                    }
                }
                insert(dic);
               
                foreach (string key in dic.Keys.ToArray())
                {
                    dic[key] = null;
                }
                Console.Error.WriteLine("");
            }
            if (selenium_driver != null)
                selenium_driver.Close();
        }



-- snip
Ping me if you need github ref..

On Saturday, May 23, 2015 at 8:36:39 AM UTC-4, Jan de Lange wrote:

[Serguei Kouzmine]

Hello Jan

I refer you to github
https://github.com/sergueik/powershell_selenium/tree/master/csharp/chrome_performance
I am working on this project after hours, it is not commercial, you are free to fork. Some past commits may be broken, head is good as far as i know.

Let me know if you have questions..

Serguei Kouzmine

On Saturday, May 23, 2015 at 8:36:39 AM UTC-4, Jan de Lange wrote: