Tasks not being delivered

[Jeremy D]

We're having problems with Task requests not being delivered. In Logs, I see "Status: INTERNAL" before it triggers another retry of the task. The instance its trying to reach has no logs it ever even attempted to make the request. Nothing has changed on our end, this is being triggered from an Appengine Flexible instance and being targeted back to the same instance.

  {

    "insertId": "1ckf9kkb2h",

    "jsonPayload": {

      "@type": "type.googleapis.com/google.cloud.tasks.logging.v1.TaskActivityLog",

      "task": "projects/redacted/locations/us-central1/queues/default/tasks/58101560599978915711",

      "attemptResponseLog": {

        "attemptDuration": "0.003367s",

        "targetType": "APP_ENGINE_HTTP",

        "scheduleTime": "2021-10-06T19:26:21.290817Z",

        "maxAttempts": 3,

        "responseCount": "0",

        "targetAddress": "POST /api/ping/tasks/goals/trigger_goal_emails",

        "status": "INTERNAL",

        "dispatchCount": "4"

      }

    },

    "resource": {

      "type": "cloud_tasks_queue",

      "labels": {

        "target_type": "APP_ENGINE_HTTP",

        "location": "us-central1",

        "queue_id": "default",

        "project_id": "redacted"

      }

    },

    "timestamp": "2021-10-06T19:26:21.296110014Z",

    "severity": "ERROR",

    "logName": "projects/redacted/logs/cloudtasks.googleapis.com%2Ftask_operations_log",

    "receiveTimestamp": "2021-10-06T19:26:22.631910850Z"

  }

[Jeremy D]

I should also mention, some of these requests do reach the endpoint, most (probably 90%) of them do not. No indication the instance is overloaded or too busy.