Ajax / nav bar problem: Test runs, scrape does not

[steven saurbier]

I am getting the error:

element wait time exceeded for selection4 in template homepage

The failure occurs in a nested navigation bar, which is the kind of thing i've scraped many times. 

I have tried using CSS and Xpath selectors; parsehub knows the object is in there (see picture 2 from test run), but according to the test run, it can't "click" into the object to expose the elements i'm trying to interact with.

Pic #1 is expected behavior (in test), pic #2 is logged screenshot from production, where there is no click action, and parsehub seems unable to click through the nav bar to the data.

I would like to continue working through the nav bar, but I am stuck here. 

Thanks!

[Andrew11]

Screenshot #2 has a message about enabling Javascript; is it possible that your template has it disabled?

[steven saurbier]

Unfortunately not. JS is enabled in each template, as well as the overall project.

However, good eye and I believe you are right - thank you. I get inconsistent failures on previous pages too, which is confounding, but when the script fails, the server snapshot says, "You need to enable JavaScript to run this app."

So what is the issue here? JS is enabled in the script, so what else can I do? Or perhaps I am selecting something incorrectly? I am providing 30 seconds after the previous (successful) click, and waiting for 30 seconds for elements to appear. 

[Andrew11]

Maybe IP rotation is enabled and the external (to the web page) JS is missing its mark? Or maybe it's starting actions before JS is downloaded? You can Select a script element and wait for it to appear on the page if so. If that doesn't work try posting the URL here.

[steven saurbier]

Unfortunately i've checked these avenues:

- IP rotation is NOT enabled - i did try to workaround with that though.

- Wait times are carefully managed - i have solved plenty of problems that way, but not this one. Specifically, the selection before (root selection in nav, loads with page) is loading, so the JS is there. I wait 30 seconds after clicking, and then an additional (? or concurrent) 30 seconds on select for the sub level selection.

- I have "ignore disabled elements" disabled

The full error text from the site in the server snapshot is the following, which is a tracking feature:  

//

 <iframe src="https://www.googletagmanager.com/ns.html?id=GTM-NPH4W8W&gtm_auth=PJKCZ08WDcz86PPj6LnR-w&gtm_preview=env-2&gtm_cookies_win=x" height="0" width="0" style="display:none;visibility:hidden" id="tag-manager"></iframe>You need to enable JavaScript to run this app.

//

I notice in the server screenshot that the "arrow" button doesn't appear in the nav bar, and the commensurate dynamic nav bar action (open branch) doesn't occur on click of the div. 

This aligns with an odd Xpath behavior i noticed - when trying to manually link the elements with Xpath after the auto parsehub didn't work, I noticed that I cannot select the "arrow" element in parsehub with the correct Xpath from the site. I can select the div and span of the text. 

URL is here but behind authentication: https://www.myunfi.com/shopping/

[Andrew11]

Since I can't get in there through the login, one last guess: try setting max workers to 1. Sometimes when there's more workers than 1 each worker has to log in before things start to work inside it.