CDAP HTTP Plugin - Pagination issue

55 views
Skip to first unread message

Kethareeswaran Krishnan

unread,
Oct 27, 2022, 5:28:12 PM10/27/22
to cdap...@googlegroups.com

Hi CDAP Team,

 

I am facing an issue with HTTP Plugin “Pagination” property, when I set the format of the output to “text”.

 

I am using a custom python code to get the URL of next page that needs to be called from the HTTP Plugin.

When I configured the custom code (with format of the output as “json”) and tried running in “Preview” mode, I was able to see the pagination happening.

But the same code when executed with format of the output as “text”, throws me error.

 

Below is the custom code that I have used.

 

import json

def get_next_page_url(url, page, headers):

    """

    Based on previous page data generates next page url, when "Custom pagination" is enabled.

 

    Args:

        url (string): previous page url

        page (string): a body of previous page

        headers (dict): a dictionary of headers from previous page

    """

    page_json = json.loads(page)

    try:

      next_page_link = page_json['paging']['next']['link']

    except:

      next_page_link = None

    # stop the iteration

    if next_page_link != None:

      return next_page_link

    else:

      return None

 

Attached is the error log.

 

Is there something else that I would need to change in the Custom code, if the format of the output is set to “text” ?

 

 

Thanks,

Kethar

default-11d8ad9b-563d-11ed-959b-1aa6323a3fe9.log

Kethareeswaran Krishnan

unread,
Oct 31, 2022, 3:11:38 AM10/31/22
to CDAP User
Hi CDAP team,

Any ideas or solution for this issue raised in below email.


Thanks,
Kethareeswaran

René Klemm

unread,
May 11, 2023, 3:32:22 AM5/11/23
to CDAP User
Hi,

I faced the same issue. For everyone wondering, this was a bug in the HTTP plugin which was fixed in version 1.2.4: https://cdap.atlassian.net/wiki/spaces/DOCS/pages/1280901131/CDAP+Hub+Release+Log#December-12%2C-2022

So, if you face this issue, please update the HTTP source plugin to version 1.2.4 or higher.

Regards,
René
Reply all
Reply to author
Forward
0 new messages