My opinion:
- if the verb isn't allowed: 405 Method Not Allowed
- if the request isn't of an acceptable mime-type: 415 Unsupported Media
- if the request is invalid or not well-formed xml (or the like): 400 Bad Request
- if the request doesn't contain all required info: 400 Bad Request or 417 Expectation Failed
I also believe it is imperative to return details of the error beyond the status code, unless its completely obvious what the error was.
For example, say you allow application/xml only and receive an application/json request body, and you return a 415; in this case, I think its optional to include error details (whether in an additional response header or in the response itself) that says 'hey, i only accept application/xml'. Thats what API documentation is for, and including it at runtime is purely a developer convenience, since the STATUS CODE is the actionable item (and is static across all applications and implementations), and additional error details may or may not be used for programmatic handling of errors (every application will have a different way of describing an error in text and doesn't make for a good standardization point).
On the other hand, say you have invalid xml in the request and return a 400. In this case I believe its required by a good api to include 'invalid or not well-formed xml' in the error details. Otherwise, there would be no way to distinguish that from 'you didn't send me all required information', assuming you do a 400 for that case as well.
Ultimately, my view is this: the response code is the signal to the consumer about the state of their request and whether they need to do something additional in order for it to be successful. Any additional details, whether sent in a response header or in the response body, are for the developer's convenience and shouldn't be relied upon to be consistent across applications, therefore making it a poor candidate to simply be echoed back to the user. One other thought on this is that I am an advocate (and have thought of using it in my own api but haven't as of yet) of a 'standardized' response header containing 'sub-status codes', which could be used to attempt to create a more meaningful, standardized way of returning additional info using additional numbers; e.g. 400.1 Missing Required Info, 400.3 Invalid XML, etc. Since it would be part of the api, it could arguably be termed a proprietary standard and would be documented.
I fully agree that it would be great if there was some definitive implementation guideline stating which code to use for what scenario using real-world conditions that everyone would follow. From what I've seen, its really just opinions out there at this time (a decade later, bleh).
Kevin