REST API response pagination problem

Mikko Kuusirati

unread,

Mar 12, 2021, 10:16:17 AM3/12/21

to COZYROC

Hi,

I have a problem with API response that uses offset and batch size mode. Batch size seems to be working but offset value is growing by 200 every iteration. I would like it to grow by 50 but I’m not able to figure out what setting controls this.

My REST configuration for the paging is: {{=parseInt(parameters.iterator || 1) + parameters.batchSize}}

And I have two parameters: ‘batchSize’ with value: {{=50}} and ‘offset’ with value: {{=parameters.iterator}}

What I’m doing wrong here?

BR,

Mikko

Ivan Peev

unread,

Mar 12, 2021, 10:16:43 AM3/12/21

to COZYROC

Hi Mikko,

You set the batch size in the REST Source component.

Mikko Kuusirati

unread,

Mar 12, 2021, 10:32:50 AM3/12/21

to COZYROC

Yes, both params are in the REST source component. Is that wrong place?

Ivan Peev

unread,

Mar 12, 2021, 5:00:07 PM3/12/21

to COZYROC

Mikko,

What is the question?

Mikko Kuusirati

unread,

Mar 13, 2021, 7:36:19 AM3/13/21

to COZYROC

The question is; why my offset value is growing by 200 on every iteration round. It should grow 50 as my service (NetBox) is feeding 50 results per page. At the moment I getting only 1/3 of the results because paging params are going like 50-0, 50-201, 50-401, 50-601 etc. It should go 50-0, 50-51, 50-101, 50-151...

This is my POC project for Cozyroc REST API source. Must say it looks very promising and I like use it widely if this problem is solved. Specially loved the feature that takes in sample of json reply and automatically parse all the fields

Pagination setup:

offset parameter:

batchSize parameter:

Captured response from the fiddler:

-Mikko-

Ivan Peev

unread,

Mar 13, 2021, 4:56:53 PM3/13/21

to COZYROC

Mikko,

What did you set for the batch size parameter in the REST Source component?

Mikko Kuusirati

unread,

Mar 14, 2021, 7:28:37 AM3/14/21

to COZYROC

Ok, now it works. I didn't realize that the batch size value comes over there. I was wrestling with the connection manager whole time and didn't check what values are submitted from the REST source component.

Thanks for the help!

-Mikko-

Suresh Kasu

unread,

Mar 22, 2021, 7:56:39 PM3/22/21

to COZYROC

Hi Mikko,

(Posted the same message before but it seems lost. I am posting again).

Appreciate your help. I am also having problem with pagination using CozyRoc Rest Source. my job is getting the data from API but it runs continuously and it doesn't stop until I have to force stop the job. Its pulling the data continuously creating duplicate records getting same data gain and again. I used the same configuration as you. is there a way to stop the job once the valid data is returned. Screenshots attached. Thanks.

Best Regards

Suresh

Mikko Kuusirati

unread,

Mar 23, 2021, 6:57:06 AM3/23/21

to COZYROC

I think I ran same kind of problem once. If I remember correctly, I was missing one of the parameter and that trigger infinite loop.

-Mikko-

Reply all

Reply to author

Forward