Sudden increase in packages executed against Sharepoint/Office 365

[John Nickell]

Beginning yesterday afternoon (12/9 Central US time) we began seeing a marked increase in SSIS packages execution failures. These packages have been in operation for several months and experienced no failures on 12/8.  Initially I brushed it off as temporary, but now it seems as if "none" of them are working.  Several of these packages run hourly, with the first failure around 10:30 on 12/9. Between 10:30 and 1500 'most' succeeded, but after 1500 on 12/9 most failed

Error messages that I'm getting are not helping me pinpoint the cause. I'm getting errors and failures in the production instance (where no changes have been made yet) and on my 'development' computer which I upgraded from 2.1.2.9999 to 2.1.100.2882 as a test.

I'm testing with a relatively simple dataflow package.  I have two sources (SQL and Sharepoint). From the sources, I compare the two and then update the Sharepoint list with any changes that have been made (SQL query is the authoritative record). The Source Sharepoint list is the same list that is being updated.  As a further step, I removed all steps except for querying the Sharepoint list and sorting it. The initial query still fails

Errors are happening inconsistently within the dataflow package. For example since I've been testing this morning, I had one (and only once) that made it through the package to the point it was trying to update the Sharepoint list only to fail there. Most of them (and all recent attempts) are failing when the dataflow queries the Sharepoint list initially.

I'm connecting to Sharepoint using MS Graph. Testing the connection (Connection Manager) within VS 2019 has succeeded every time. 'Usually' if I attempt to preview the Sharepoint source that is successful, but not always. Even if it is successful attempting to debug and run the package fails. I'm not seeing any alerts on Microsoft or Azure that would provide any indication that the problem's there, though I feel like something must have changed there.

I have opened a support ticket with CozyRoc, but I'm uncertain if they have anyone monitoring this over the weekend and was hopeful someone in the group could point me in a direction or confirm they're seeing similar issues?

Thanks for any thoughts or pointers.

[John Nickell]

At suggestion of Ivan in support ticket I ran the SSIS package that seems to "most often" error locally while running Fiddler on my computer.

The first time I ran it, it ran successfully. I reran the exact same package (about a minute  or two later) and this time it failed while trying to update the Sharepoint list

The error Fiddler is showing that I got back is a 502 Bad Gateway: 

HTTP/1.1 502 Bad Gateway
Transfer-Encoding: chunked
Content-Type: application/json
Content-Encoding: gzip
Vary: Accept-Encoding
Strict-Transport-Security: max-age=31536000
request-id: 3586f320-a8e0-4ff9-b9b8-7085d34bbffd
client-request-id: 3586f320-a8e0-4ff9-b9b8-7085d34bbffd
x-ms-ags-diagnostic: {"ServerInfo":{"DataCenter":"North Central US","Slice":"E","Ring":"3","ScaleUnit":"000","RoleInstance":"CH01EPF0000C550"}}
Date: Tue, 13 Dec 2022 14:39:12 GMT
Connection: close

Still trying to determine how I can "report" this to Microsoft, or if we're at fault.

[John Nickell]

Resolution was found by waiting. After about 1 week of having significant failure rates, we've had 3 days of no failures. No changes were done to SSIS packages that are leveraging CozyRoc.