504 Gateway Time-out errors

[Sören Zschoche]

Hi,

since yesterday been experiencing a very high number of: "504 Gateway Time-out"

while trying to import outbrain campagin data in our system.

https://api.outbrain.com/amplify/v0.1/reports/marketers/00e514c51bc7f38a5fd163170bf08e58da/campaigns/sections?from=2022-08-01&to=2022-08-01&limit=500&offset=0&includeArchivedCampaigns=false

The strange thing is, it doesn't work on 2 marketers, but it works for the other 20 marketers perfect.

It also worked some days ago. (The marketer in the url doesn't work)

Are you already working on this problem?

Thanks a lot!

Kind regards,

Sören

[Sören Zschoche]

And also 502 errors, 504 and 502 errors.

[mfa...@redgobo.com]

We have also been experiencing a notable percentage of 502 and 504 errors for a while now. 

[ssha...@outbrain.com]

Hey and thanks for reaching out and sharing. 

Can you please share some more details? when its started, exact time when it happens (logs including timestamp will be great) and the URLs.

[Sören Zschoche]

Hey,

unfortunately we don't got any logs.

It started on sunday and affects only a few of our accounts.

For example when we try to fetch this:

https://api.outbrain.com/amplify/v0.1/reports/marketers/00e514c51bc7f38a5fd163170bf08e58da/campaigns/sections?from=2022-08-01&to=2022-08-01&limit=500&offset=0&includeArchivedCampaigns=false

We always just get a response:

504 Gateway Time-out

Thanks a lot!

Sören

[mfa...@redgobo.com]

We've seen this problem for a few months, and I had made another thread for it back in June: https://groups.google.com/u/2/g/outbrain-amplifyapi/c/NNF5PlMgCMA/m/iQ1bi8TKAAAJ.

We do have logs though there is no AMPLIFY-REQUEST-ID on 500-level errors. Below is info on 2 separate errors with UNIX timstamps -- first is 502 and second is 504. The 504 errors always have a request length of 60s so I'm assuming you have some kind of max request length set for the API?

502 error:
    {
        "end_time": "12:10:18",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Tue, 02 Aug 2022 16:10:18 GMT"],
            "Content-Type": ["text/html"],
            "Content-Length": ["122"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",
        "response_time": 16.73,
        "start_time": "12:10:02",
        "status": 502,
        "timestamp": 1659456602,
        "url": "https://api.outbrain.com/amplify/v0.1/marketers"
    }

504 error:

    {
        "end_time": "02:06:03",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Tue, 02 Aug 2022 06:06:03 GMT"],
            "Content-Type": ["text/html"],
            "Content-Length": ["132"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",
        "response_time": 60.15,
        "start_time": "02:05:03",
        "status": 504,
        "timestamp": 1659420303,
        "url": "https://api.outbrain.com/amplify/v0.1/marketers"
    }

[ssha...@outbrain.com]

We're investigating it,

Can you please provide some more URLs / marketer IDs for example? 

[Julian Weber]

Hi,

I'm the colleague of Sören. I'm currently checking the imports today. But it seem to work again.

[Julian Weber]

It works today. I can also import the last failed days.

[mfa...@redgobo.com]

Here are a few more 502 errors from us:

[
    {
        "end_time": "01:10:09",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Wed, 03 Aug 2022 05:10:09 GMT"],

            "Content-Type": ["text/html"],
            "Content-Length": ["122"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",

        "response_time": 7.71,
        "start_time": "01:10:02",
        "status": 502,
        "timestamp": 1659503402,
        "url": "https://api.outbrain.com/amplify/v0.1/marketers"
    },
    {
        "end_time": "21:21:05",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Wed, 03 Aug 2022 01:21:05 GMT"],

            "Content-Type": ["text/html"],
            "Content-Length": ["122"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",

        "response_time": 1.04,
        "start_time": "21:21:04",
        "status": 502,
        "timestamp": 1659489664,
        "url": "https://api.outbrain.com/amplify/v0.1/realtime/marketers/0023fe0e8d7443259724668fe395c102a3/campaignsHourly?hours=5"
    },
    {
        "end_time": "21:21:04",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Wed, 03 Aug 2022 01:21:04 GMT"],

            "Content-Type": ["text/html"],
            "Content-Length": ["122"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",

        "response_time": 0.34,
        "start_time": "21:21:04",
        "status": 502,
        "timestamp": 1659489664,
        "url": "https://api.outbrain.com/amplify/v0.1/realtime/marketers/0047ad7cc51fe97d2b5e1de0abf75ba5e7/campaignsHourly?hours=5"
    },
    {
        "end_time": "20:40:20",
        "headers":
        {
            "Server": ["awselb/2.0"],
            "Date": ["Wed, 03 Aug 2022 00:40:20 GMT"],

            "Content-Type": ["text/html"],
            "Content-Length": ["122"],
            "Connection": ["keep-alive"]
        },
        "method": "GET",

        "response_time": 0.21,
        "start_time": "20:40:20",
        "status": 502,
        "timestamp": 1659487220,
        "url": "https://api.outbrain.com/amplify/v0.1/realtime/marketers/002116226c0507d0a98b30ee7725ef2f98/campaignsHourly?hours=5"
    }
]

[ssha...@outbrain.com]

Regards the exceptions from the realtime API endpoints at 03 Aug - it was a temporary issue that was resolved.

For the other endpoints we will need the exact time and URL when it happened. 

Please let us know when it happens again with the details and will dig into it further again.