code is not working for long script

[Twinkal Paralkar]

Here , I need to check title of very link by using bs4.this code was checking 380 link ae per excel. this was working last week. but from 26 feb 2024, this code is just runing for  5 link as shown above. it is just waiting. Please help me to solve this. our work is affected by this. code :

def extract_source(url):

    #  print("calling 1")

     agent = {"User-Agent":"Mozilla/5.0"}

     source=requests.get(url, headers=agent).text

    #  print(agent)

     return source

def extract_data(source):

     soup=bs4.BeautifulSoup(source, 'lxml')

     names=soup.findAll('title')

    #  for i in names:

    #     print(i)

     return names[0].text

##################################################

Thank you,

Twinkal

 

[leonardr]

Twinkal,

It's difficult to say what's going on without knowing the URL or markup that is affected, but the most likely explanation is that your requests.get() call is hanging due to a change in whatever service is being accessed, and the problem is happening before any Beautiful Soup code runs.

Since the code works for five other URLs, and stopped working on a particular date with no changes to your code, I would look at the URL that's causing the code to hang. For example, see what is happening when you open that URL in a web browser.

On the other hand, if you've found an HTML page that makes Beautiful Soup+lxml hang during parsing or searching, I would very much like to see that HTML page.

Leonard