6

One company (who I will refer to as Company NNN from here forth), has a REST API. At my company, we use Company NNN's REST API to provide some content to our end users.

If I send the request with the User-Agent header field filled in, the REST API returns valid JSON data. If I send the request without the User-Agent header field filled in, the REST API returns 500 Internal Server Error. Upon notifying Company NNN of this, their response to me was that specifying a User-Agent is best practice.

Shouldn't this API be able to return valid JSON even without a User-Agent request header?

an12
  • 171
  • 1
    This happened to me when I tried direct (non-browser) access on a site. I guess it deters bots? But easy to work around, just add in a reasonable User-Agent. (I copied and pasted what Chrome sends, which looked like semi-gibberish). Not sure how typical is this behavior. – user949300 Aug 15 '17 at 00:48
  • I'm sure it can return JSON without a UA specified, but they aren't obligated to make their service do so. – Andy Aug 15 '17 at 01:18
  • Is it the company's application itself that is throwing the 500? Sometimes there are network nodes in between (e.g. if the site is Akamaized or behind a proxy) that require the agent string too. – John Wu Jun 08 '18 at 16:06

3 Answers3

15

REST is not a specification therefore there are no hard and fast rules for REST API. Company NNN is allowed to make any decisions they want when designing their REST API. It's not a defect if they intentionally want to enforce clients specify a user agent.

I think it was a good idea to report it because it is a bit unusual for a REST API to enforce a user agent because REST API are consumed by programmatic clients that typically don't set a user agent by default.

Secondly 500 is the incorrect status for an error of this kind. It likely signifies there is a bug in their system. If it were intentional that user agent must be specified I would expect a 400 Bad Request response with a meaningful message.

User agent could be useful for them to monitor the source of unauthenticated requests and group them in a meaningful way but the fact that they're returning a 500 internal server error (and I assume no helpful message) gives me the impression this is actually a bug. Their response is classic, "it's not a bug it's a feature".

Samuel
  • 9,187
7

Whether or not it's REST, it's still HTTP, and subject to the HTTP specification.

Per RFC 7231:

A user agent SHOULD send a User-Agent field in each request unless specifically configured not to do so.

Now you ask, what does SHOULD mean?

SHOULD This word, or the adjective "RECOMMENDED", mean that there may exist valid reasons in particular circumstances to ignore a particular item, but the full implications must be understood and carefully weighed before choosing a different course.

So it is recommended ("best practice," per NNN) and unless you have carefully weighed the alternatives you should make every effort to include it.

John Wu
  • 26,462
0

All the libraries I have seen set the User-Agent string, usually with their definition. The usual cases for an missing User-Agent string is a telnet session or a bot. Crawlers have a convention for their User-Agent so you can easily identify them. Knowing the user agent can be very useful in debugging edge cases where some clients fail and other succeed.

While it may be possible communicate without the User-Agent it is abnormal. I don't see a particular issue with failing when it is missing. Use appropriate testing tools and the User-Agent will be set.

You could use a simple user agent header like User-Agent: telnet. This should make it easy to find your tests in the access log.

BillThor
  • 6,270