API regression: Why does a label value need to be RFC1035-compliant?

[Alex Van Boxel]

This nights run got a braking API change:

status: {

code: 2 

message: "Invalid value for field 'resource.properties.labels': '"timeout" : "60"'. Label value must be an RFC1035-compliant name when non-empty" 

}

If I look at the spec I don't see that as a requirement:

https://cloud.google.com/dataproc/docs/concepts/labels

https://cloud.google.com/dataproc/docs/reference/rest/Shared.Types/ClusterOperationMetadata

Me not happy :-(

[Alex Van Boxel]

created a support ticket: Case 12222303

Note: just tried it in the UI (Added a label timeout with value 60) got the same error.

[Alex Van Boxel]

Here is a reproducible via the UI

Go to "DataProc", open " Preemptible workers, bucket, network, version, initialization, & access options "

Press "Add Label":

put in key: "timeout"

put in value: "60"

[Dennis Huo]

Sorry! We should have communicated this better. This is part of changing labels from beta until GA; up until the most recent changes, labels were considered beta (screenshot of last cached page)

Though we didn't anticipate breaking changes, part of GA is needing to propagate labels to GCE VMs, and ultimately for billing rollup.

We observed that GCE's label requirements were more strict, and trying to propagate labels starting with a number, for example, unfortunately broke deployment because of that.

Normally we run a scan of what existing usage might break with new constraints, but looks like this one fell through the cracks; we'll be sure to follow up internally to make sure we detect and announce these kinds of things even when changing beta APIs.2

[Dennis Huo]

This is also an interesting case that we hadn't considered in-depth; most cases of numeric values are placed in "metadata" instead of "labels" due to retrieval from inside the VM itself, but it makes sense that one might use labels if performing some management/orchestration from outside of the cluster.

Depending on the scope of this issue we'll assess whether we can add a workaround for compatibility in the short term.

[Alex Van Boxel]

ok, thanks. with this we can implement a workaround.

[Alex Van Boxel]

Sorry, pressed to fast on the POST button, we will change the label to comply with the new spec.

[Dennis Huo]

Thanks, sorry again for the break! We've completed a complete assessment based on all available historical logs of labels usage, and confirmed that there was a minimal number of projects using non-compliant label values; I agree this wasn't intended constraint on Dataproc's initial implementation of labels though, and we'll plan to better reach out to affected projects for these kinds of things in the future.

We've also investigated further and it appears the constraint in itself on the GCE side might be unintentional for label values; even though label keys are indeed required to conform to the RFC, label values are possibly intended to support a more relaxed syntax, including fully numeric values.

So, incidentally, in the future such values might be allowed once again, but it's still unconfirmed officially. If GCE does relax their label validation in the future, and we confirm there's no possibility of reverting that relaxation, Dataproc will be able to also relax its constraints within ~2 weeks of GCE's update.

[Alex Van Boxel]

We already fix at our end the label (we just put a t before the numerical value). FYI: we have a daemon running that looks at labels on our cluster that shuts it down at none activity.

I prefer a "consistent" set of rules and constraints over all the services. (less confusion)

[Alex Van Boxel]

A note of things we also use labels for (not in DataProc but on k8s) is versions. In general versions start with numbers. (just saying...)