How can I iterate through a list of items and extract a specific part using Selenium and Python

[develop...@gmail.com]

Please someone help me. I have been working on this for more than a week with no luck!

From this web page "https://meshb.nlm.nih.gov/treeView", I want to iterate through each node of the tree and if I see the word "Cardiovascular..." in their items, I want to create a dictionary that lists the top level node along with all of cardiovascular associated items. For example, in the above page you can see that if you expand "Anatomy [A]", you will see cardiovascular. Now, I want this part along with whatever included in cardiovascular if you expand it. I have attached 2 screenshots of a part of html.  Here is my Python code:

from selenium import webdriver
from selenium import webdriver
chrome_path=r"G:\My Drive\chrome_driver\chromedriver_win32\chromedriver.exe"
driver=webdriver.Chrome(chrome_path)
driver.get('https://meshb.nlm.nih.gov/treeView')
sidebar = driver.find_element_by_xpath("/html/body/div[2]/div")
i=1
for i in range(16):  # since I have 16 div(s)
   sidebar.find_elements_by_xpath("/html/body/div[2]/div/div[i]")       
   element = driver.find_element_by_xpath("/html/body/div[2]/div").find_element_by_xpath("/html/body/div[2]/div/div[i]").find_element_by_xpath("//*[@class='ng-scope']/span")
   element.click()

But I keep getting this error:no such element: Unable to locate element: {"method":"xpath","selector":"/html/body/div[2]/div/div[i]"}

Any help is appreciated!

[develop...@gmail.com]

I  updated my code with this line: element = driver.find_element_by_xpath("/html/body/div[2]/div").find_element_by_xpath("/html/body/div[2]/div/div[" + str(i) + "]").find_element_by_xpath("/html/body/div[2]/div/div[" + str(i) + "]/div/span")

When I print it, it shows me the first level nodes like Anatomy,.... But it does not open all plus buttons!

[Selenium007]

Did you try findElements? This returns the list of Web Elements and you can iterate through the list.

Also, relative xpath might be better than dynamic xpath

[develop...@gmail.com]

I did. Here is the updated code with the help of a friend. But the problem is that it says the <i> is not clickable and when I changed it to span only, it got stuck on the first node! Any help?

while True:
    elements = driver.find_elements_by_xpath("//a[@class='ng-scope']/following-sibling::span[@ng-if = 'node.HasChildren']/i")
  if (len(elements) > 0):
    elements[0].click() # click on the first in the list
    driver.implicitly_wait(10) # wait until list will be updated
    continue
  break
spans = driver.find_elements_by_xpath("//span[contains(., 'Cardiovascular')]/parent::*/parent::*/descendant-or-self::node()/a/span")
for span in spans:
  print (span.text)

[Selenium007]

How about "//a[@class="ng-scope"]/following-sibling::span[@ng-if = 'node.HasChildren']/i[contains(@class, "fa fa-plus-circle")]" since i has plus and minus based on if its expanded

[develop...@gmail.com]

Thanks for your comment, but , still does not work!

[Shiva Prasad Adirala]

Hi Mate,

As per my understanding - I wrote snippet for your problem

1. Get List of Node's

2. Expand Each Node One After other

3. Get List Of Sub-Nodes

4. Compare each sub-node and expand sub-node if text==Cardiovascular

5. Get List of Sub-Sub-Node text's (Need to store in list)

from selenium import webdriver

from selenium import webdriver

from selenium.webdriver.common.by import By

import time

chrome_path=r"E:\Selenium\drivers\Chrome64\chromedriver.exe"

driver=webdriver.Chrome(chrome_path)

#navigate to website

driver.get('https://meshb.nlm.nih.gov/treeView')

# Get list of nodes

main_list = driver.find_elements_by_xpath("//div[@ng-repeat='node in trees']/div/span")

# Iterate for list of nodes

for i in range(1, len(main_list)+1):

    # Expand each node

    node_expand_icon = driver.find_element_by_xpath("(//div[@ng-repeat='node in trees']/div/span)[" + str(i) + "]")

    node_expand_icon.click();

    # Wait for 5 seconds

    time.sleep(5)

    # Get list of sub-nodes

    sub_nodes_list=driver.find_elements_by_xpath("(//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li")

    

    # Iterate for list of sub-nodes    

    for j in range(1, len(sub_nodes_list)+1):

        sub_node_text=driver.find_element_by_xpath("((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "]").text

        # If sub-node contains "Cardiovascular" then expand

        if "Cardiovascular" in sub_node_text:

            sub_node_expand_icon=driver.find_element_by_xpath("((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li/span)[" + str(j) + "]")

            sub_node_expand_icon.click();

            # Wait for 5 seconds

            time.sleep(5)

            # Get sub-sub-node lists

            sub_sub_nodes_list=driver.find_elements_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li")

            # Iterate for list of sub-sub-nodes            

            for k in range(1, len(sub_sub_nodes_list)+1):

                sub_sub_node_text=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]").text

                print    sub_sub_node_text

I have tested from my end.Please let me know if you need any help on this

[develop...@gmail.com]

Hi Shiva, thanks for taking your time answering my question.  However, the code does not do my intended goal. Please see the attached picture, As you can see in the picture, if "Cardiovascular" is found, I want that item to be expanded and if it has more buttons to expand, I want all of them to be expanded as well. But, using your code, this does not happen. Also, the print function does not work either. I want for example,  have Cardiovascular Disease as the header and all of  the items that falls under it.  If any of these nodes has a sublist, I want them as well...

[Shiva Prasad Adirala]

Dude,

Following snippet is similar to your requirement, Do some modifications in loop to get to meet requirement. Following snippet will give you the idea for expand each element. Hope you get the code and you develop code based on this.

This approach is not suggested, since it takes lot of time to get result. Will keep post u on this

from selenium import webdriver

from selenium import webdriver

from selenium.webdriver.common.by import By

import time

chrome_path=r"E:\Selenium\drivers\Chrome64\chromedriver.exe"

driver=webdriver.Chrome(chrome_path)

#navigate to website

driver.get('https://meshb.nlm.nih.gov/treeView')

# Get list of nodes

main_list = driver.find_elements_by_xpath("//div[@ng-repeat='node in trees']/div/span")

# Iterate for list of nodes

for i in range(1, len(main_list)+1):

    # Expand each node

    node_expand_icon = driver.find_element_by_xpath("(//div[@ng-repeat='node in trees']/div/span)[" + str(i) + "]")

    node_expand_icon.click();

    # Wait for 5 seconds

    time.sleep(2)

    # Get list of sub-nodes

    sub_nodes_list=driver.find_elements_by_xpath("(//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li")

    

    # Iterate for list of sub-nodes    

    for j in range(1, len(sub_nodes_list)+1):

        sub_node_text=driver.find_element_by_xpath("((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "]").text

        # If sub-node contains "Cardiovascular" then expand

        if "Cardiovascular" in sub_node_text:

            sub_node_expand_icon=driver.find_element_by_xpath("((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li/span)[" + str(j) + "]")

            sub_node_expand_icon.click();

            # Wait for 5 seconds

            time.sleep(2)

            # Get sub-sub-node lists

            sub_sub_nodes_list=driver.find_elements_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li")

            # Iterate for list of sub-sub-nodes            

            for k in range(1, len(sub_sub_nodes_list)+1):

                child_sub_node=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]")

                # Expand if it has child element's

                try:

                    has_child_element = driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/span")

                    has_child_element.click();

                    # Wait for 5 seconds

                    time.sleep(2)

                    

                    print child_sub_node.text

                    

                    # Get Super Sub node

                    super_sub_node_list=driver.find_elements_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li")

                    # Iterate for list of sub-sub-nodes            

                    for l in range(1, len(super_sub_node_list)+1):

                        super_child_sub_node=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]")

                        # Expand super childs if

                        try:

                            has_super_child_element = driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]/span")

                            has_super_child_element.click();

                            # Wait for 5 seconds

                            time.sleep(2)

                            print super_child_sub_node.text

                            super_super_sub_node=driver.find_elements_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li")

                            # Iterate for list of sub-sub-nodes            

                            for m in range(1, len(super_super_sub_node)+1):

                                super_super_child_sub_node=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]/ul/li[" + str(m) + "]")

                                # Expand super childs if

                                try:

                                    has_super_child_element = driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]/ul/li[" + str(m) + "]/span")

                                    has_super_child_element.click();

                                    # Wait for 5 seconds

                                    time.sleep(2)

                                    print super_super_child_sub_node.text

                                    a_list=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]/ul/li[" + str(m) + "]/ul/li")

                                    # Iterate for list of sub-sub-nodes            

                                    for n in range(1, len(a_list)+1):

                                        super_super_super_child_sub_node=driver.find_element_by_xpath("(((//div[@ng-repeat='node in trees']/div)[" + str(i) + "]/ul/li)[" + str(j) + "])/ul/li[" + str(k) + "]/ul/li[" + str(l) + "]/ul/li[" + str(m) + "]/ul/li[" + str(n) + "]")

                                        print super_super_super_child_sub_node.text

                                       

                                except Exception:

                                    print super_super_child_sub_node.text

                        except Exception:

                            print super_child_sub_node.text

                except Exception:

                    #print Exception

                    print child_sub_node.text

                #print    child_sub_node.text