Hi CDAP Team,
I am facing an issue with HTTP Plugin “Pagination” property, when I set the format of the output to “text”.
I am using a custom python code to get the URL of next page that needs to be called from the HTTP Plugin.
When I configured the custom code (with format of the output as “json”) and tried running in “Preview” mode, I was able to see the pagination happening.
But the same code when executed with format of the output as “text”, throws me error.
Below is the custom code that I have used.
import json
def get_next_page_url(url, page, headers):
"""
Based on previous page data generates next page url, when "Custom pagination" is enabled.
Args:
url (string): previous page url
page (string): a body of previous page
headers (dict): a dictionary of headers from previous page
"""
page_json = json.loads(page)
try:
next_page_link = page_json['paging']['next']['link']
except:
next_page_link = None
# stop the iteration
if next_page_link != None:
return next_page_link
else:
return None
Attached is the error log.
Is there something else that I would need to change in the Custom code, if the format of the output is set to “text” ?
Thanks,
Kethar