7

When recieving data from clients how much data that you provided to them should you require back from them?

For example when clients order a product from a webservice should they just need to provide a product code? The other alternative would be to require the product code product name and product price ect.

I have alway built webservices the former way with the minimum of data returned back to me, however I have just been consuming another web service that requires the vast majority of the data returned back to them. What are the advantages of the second way?

Tom Squires
  • 17,755
  • 11
  • 67
  • 88

8 Answers8

13

The Zen masters of Web Services would say:

Require the absolute minimum amount of data required to fill out the request.

Reply with as much data as might be useful to a requester.

The reason being that extra data requires extra (and unnecessary) work on the client side an introduces more overhead and more bugs. Sending "too much" data in the reply is generally only a little extra work for the server, it requires no effort for the client to ignore any fields it does not currently need, and, it reduces the number of change requests over the long term. In this way your API will be easier to use and useful to as wide an audience as possible.

Community
  • 1
James Anderson
  • 18,147
  • 1
  • 43
  • 72
  • I have used -no suffered- services where huge amounts of data were requested. you could get a list of products and it would return them with a lot yet always missing some key information (like the price of the item for example). then you'd have to query each and get detailed information. Query could not just be done with the product code but also had to contain the name, package type, quantity in package etc any error in any field would invalidate the whole query... Ho the pain... why as thou forsaken me such to the 7th level of www hell ! – Newtopian Oct 27 '11 at 03:56
  • @Newtopian. Its always amazed me how many WSDLs don't make obviously optional fields optional! Its fine having the option of limiting your request to a particular package size, but, making the client supply it every time is just plain dumb. – James Anderson Oct 27 '11 at 04:01
  • now add a paranoid level of security on top of this requiring each messages to be cryptographically signed... twice.... you end up with a single request (for the user) generating 300+ requests to the server in turn generating 600+ signature requests. Thank the gods the USB key API we were forced to use caches the pin so the end user do not have to enter it every times !!! Seems to me LDS usage is flourishing among a certain cast of self proclaimed "Architects". – Newtopian Oct 27 '11 at 06:26
1

Off the top of my head?

  • Requiring redundant information can reduce (though certianly not eliminate) bugs involving spurious or 'off-by-one' type errors. Submitting an order with a product code and product description is less likely to be accidentally the wrong product code.
  • You can force the interaction to use your full interface. Web scrapers will have to maintain session state to have enough information to run your service. This means you can rate limit them more effectively, if that's an issue for you.
  • You don't have to look up anything; its all there in the request: add_to_order(Product(**json.loads(request.data)))
SingleNegationElimination
  • 4,717
  • thanks for your answer but remember this is a webservice. Things such as typing errors and web scrapers arent really relevent. – Tom Squires Oct 11 '11 at 15:51
  • No, I don't think so either; I prefer just using primary keys too. – SingleNegationElimination Oct 11 '11 at 15:53
  • 2
    You don't even get to avoid looking up the extra data passed to you, because now you have to validate it instead :P The only such service I've seen do this (and it was a pretty dubious use) was a case where an object could be looked up using either of two keys, one of which was itself composed of two values. – shambulator Oct 11 '11 at 16:57
  • There is one thing though - if you're processing orders for an online store, the price in your database may change between filling the cart and posting the actual order. If you let the client post the price alongside the product code, you can catch this scenario and return a warning instead of silently processing the order at the new price. – tdammers Oct 11 '11 at 17:45
  • Not to mention any other number of problems with trusting the user-submitted price for a product. – Bryan Boettcher Oct 11 '11 at 22:00
  • 1
    The "correct" way to deal with price is to keep track of the price offered to the user in the user's server session, and to reuse that price when they confirm their order. Many countries have laws that require you to fulfill a sale at the price you advertise. – Joeri Sebrechts Oct 27 '11 at 07:31
0

It seems logical that if you send data to the client, then you only need a smal amount of data back rather than the full population. However, there may be cases where an interface is treated as a save operation that can do both updates and inserts. An insert will usually require the entire population is nothing is sent to the client and there are no pre-defined default values.

Some developers try to reduce the number of interfaces because fewer interfaces means fewer points of failure and reduced maintenance cost. Although it does make things a little more difficult on the client side.

However, with that said, if performance or bandwith is an issue then you should try to minimize the amount of information passed between client and server.

This is a typical performance/simplicity trade-off that developers struggle with.

k rey
  • 571
  • "Fewer interfaces means fewer points of failure and reduced maintenance cost" - do they now? Perhaps you should stop to consider that fewer interfaces means larger and more complex interfaces that are each more likely to fail than smaller, simpler interfaces. In a typical web service app, most endpoints run under the same host or hosting environment anyway, so you're not adding more points of failure. – Aaronaught Oct 30 '11 at 21:09
  • @Aaronaught, while you make a valid point that simpler interfaces are less likely to fail, the point remians that there are additional interfaces. From a pure testing perspective, 100 interfaces requires 100 tests to be written. Whereas, one interface only requires one test to be written. As I stated, if it makes sense, then additional interfaces should be developed. However, it may also make sense to have fewer interfaces from a testing perspective. One interface is also easier to modify and less buggy than 100 if an interface change is required. – k rey Nov 01 '11 at 20:42
  • What exactly are you calling an "interface" here? If you're writing tests for each feature/code path then consolidation doesn't change this at all. For example, if you combine and insert and update into a SaveOrUpdate (or just Save, if that's your bag), you still need to write tests to cover both the insert and update paths. If you're not going to test for all conditions then why bother writing tests at all? – Aaronaught Nov 01 '11 at 21:23
0

In case we are just talking about Data Services for retrieving and storing entity objects as little information as possible is nice to have. This allows you to work stateless and gives greater control over the integrity of the data (because it's all managed in the data tier).

If however we are talkin about integration services e.g which support long running transactions, the requirement for the amount of required post data might shift towards a bit more.

Situations where one might offer a client a certain price for a specific order could vary over time and per sales person but the acceptance of such an order might take a few days. In that case you want to be able to retrieve more than just the order but some other criteria as well. In that case you end up with extra data that provides enough context to reconstruct the situation.

So I guess it all comes down to being able to having enough information available to reconstruct the original situation and derive the required actions to perform.

Carlo Kuip
  • 560
  • 2
  • 6
  • 1
    Those are exactly those situations where you can't rely on the client giving you correct data anyway. – Jan Hudec Oct 25 '11 at 13:11
  • @Jan Hudec Never trust user input for sure, completely agree on that. On the other hand you will have to trust some incoming data otherwise it's useless to have a service in the first place. Adding checksums etc. is no guarantee but will allow you to check the information integrity. It's by no means bullet proof but adding meta data will help you overcome the obvious tampering. – Carlo Kuip Oct 25 '11 at 13:23
  • It's often reasonable to pack your state, encrypt it and send it to the client to provide it back later. In fact it's the most reasonable way to implement sessions in web applications. But it's magic cookies, not the raw data. – Jan Hudec Oct 25 '11 at 14:08
0

When the data change, you may need to know whether the client requests the operation based on the current version. Than you can either request all the relevant data back for verification or provide a version/timestamp and request that.

If the data change in big transactions, version/timestamp is usually easier, but if various bits change independenty, than you'd have to provide many versions, so just asking the data back is easier.

Jan Hudec
  • 18,340
0

Request should contain essential information only in addition to filters (if applicable). Result should contain most important data with which a user can use the information to take a decision. The returned data quantity should be minimal for systems with large number of users.

If you system is well designed to serve the business, it is possible to do this.

GUI validation is required to fall in line with business nature so that the amount of data returned is not excessive. For example, search by name in a bank should not allow single letter search. Get me all names starting with "D" is an odd request that GUI should prevent so that you don't get 100000 names back.

The database and queries should be designed for incremental browsing to serve this purpose.

NoChance
  • 12,462
0

You should carefully evaluate what kind of information does the client need to send to the server. In my opinion you should provide the full set of information needed to operate only to a stateless service, on the other side I prefer to send the minimum set of data needed if the service is state-full.

In an order filling web service (as you describe) there are a set of relevant information which should be kept in a session by the server and not provided by the client. Think of the price. What if a malicious client changes the price from 100$ to 1$?

Requesting redundant information to the client is a general security threat. That said you should not ask a client to send that kind of data. The other good side effects of this practice are that a client is easier to implement and that the payload you transfer is smaller but I consider them side effects as a good security policy on the transferred data trumps them all.

Terenzio Berni
  • 101
-1

Web service messages should be chunky, not chatty. Message-oriented, not call-oriented.

A request or command message should contain, at a bare minimum, enough information to:

  1. Validate the caller (authentication/authorization).
  2. Validate the transaction (for example, an update request in an optimistic concurrency environment might require the original version number)
  3. Execute the transaction (identifiers, search parameters, etc.)

In addition, there will usually be several parameters/elements that you will want to have as optional but not required:

  • Lists of sub-requests, with a client-supplied correlation ID. For any given operation, don't require the client to make requests one by one; instead, allow them to cram it all into a single message, and make sure you repeat their correlation IDs in the response. This is critically important in high-latency environments. (Correlation IDs should of course be optional, like the list itself).

  • The data type (for REST especially) - allow clients to specify XML, JSON, etc.

  • Control over the size and shape of the response message, especially if the response will be very large and/or contain many elements. At a minimum, provide a way to throttle the maximum number of results. Sorting and paging options are better. Some services - for example Salesforce - also provide a "query ID" that can be used to quickly retrieve pages of a result.

    If it's likely to make a difference in performance, you might also consider allowing clients to indicate the level of nesting and/or which relationships to load, using a sensible default (generally all or none).

  • A return address (for one-way messages).

  • Timeouts, error-handling strategies, log settings, or anything else that might be of particular importance for a long-running transaction. (Again, use sensible defaults, and validate the inputs!)

  • Some services will supply clients with an area to just stuff in whatever user data they want, either as a string or as an XML element. Think of this as the "memo" line on a cheque. It's especially useful in one-way messaging scenarios.

I can't emphasize strongly enough that this second list of options needs to be optional and only used for messages that actually benefit from it. You don't want to confuse clients with a bewildering array of seemingly pointless options.

Finally, try to keep information that is common to all messages (especially credentials) in the headers, not the body. If you have information that's common to many messages but not necessarily all (sorting/paging being one common example), then considering abstracting it into its own data type (parameter object). That way the client can reuse the same settings over and over again if it wants to.

Please don't require all sorts of checks and balances like names that have to match IDs, or session IDs, or checksums, or control totals, or whatever. You have to trust your clients with the privileges they've been given. If you don't trust them to send correct data (according to their own requirements) then you have a business problem to solve, not a technical one.

Aaronaught
  • 44,275
  • You stated, "Please don't require all sorts of checks and balances like...". This practice and advice is very dangerous as it opens systems up to Hackers. A good programmer will defensively validate at all levels (Interface, UI, Business Layer, Domain Layer, Data Layer) to ensure data integrity and prevent hack attacks. – k rey Nov 01 '11 at 20:58
  • @krey: Uh, "validate the caller" and "validate the transaction" were the first two items on this list. Did you actually read the answer before downvoting/quipping? – Aaronaught Nov 01 '11 at 21:24
  • You are missing my point. Validating a caller/transaction is not enough. A SQL injection attack is a valid SQL statement and it may come from a valid source. Passing wrong ID's (or email addresses) in transactions to gain account access information could also appear valid. Finaly, some users want to know if the control totals do not match so they can identify bugs in thier respective system. I say, 'do required additional checks and balances to mitigate hacking and improve stability'. – k rey Nov 03 '11 at 15:50