Coarse-grained views on server vs fine-grained views assembled on client vs fine-grained views with batching

Question

I've currently got a browser talking via an "API" to the server. API is scare quoted because the API really wouldn't support another user interface very well, because it consists of every call our UI happens to need, in the form it happens to need it, and nothing else. Which could be fine. However, we are starting to have calls along the lines of "GetUserSummaryData", "GetUserHisFriendsAndHisPurchases", "GetUserAndHisPurchases", etc.

Note that on the server a more reasonable fine-grained API is written, (and the current API calls just cause the server to wire those together) it's just that what is exposed to the UI matches exactly what it needs at that particular moment.

The justification for the current approach is that going from "I need those 6 things at the moment" to "I need to make 6 calls and put the data together" has to happen somewhere, and if you do it on the server then the browser can make 1 course-grained call, vs. the alternative of either making 6 calls and taking a (significant) performance hit, or else figuring out some system of batching the 6 calls (including dealing with the issue of dependencies, possibly by having the call batching mechanism smart enough to say "the 3rd parameter of call 2 should be this part of the JSON response object from call 1" and having the server understand that).

Doing things on the server as we are does make it pretty much impossible to write another UI without making more customized views on the server to support it. But we have no reasonable current expectation that someone without access to the server would want to write another UI.

Is continuing to make really customized views on the server as the only "API" the right approach, and if not what would be better?

Answer 1

Way-too-specific "query APIs" are typically what you get when your application architecture and sometimes the development process itself fails to keep up with expansion of requirements scope.

Well-designed APIs aren't specific to a client. Those aren't really APIs at all, they're basically a "business layer" or middleware application tier. APIs should either encapsulate business operations or, in the ever-more-popular REST style, application resources.

Building highly-maintainable, high-performance applications is hard, and bloated not-really-APIs evolve because - and I'm speaking from experience here - development teams lack the experience, mandate, and/or skills to do everything that is necessary to achieve that goal. In a system where you actually want to have an API, one that is meant for external as well as internal consumption, you should expect to see several qualities:

1. Clear separation of responsibilities. As in, X API deals with X resource, Y API deals with Y resource. Note that I say "resource" and not "data"; resources are not tables, you could maybe consider them to be similar to an aggregate root.

2. Versatility. Names like GetOrdersByCustomer imply an extremely myopic API design. What you want to see is a GetOrders API (or an /orders/ resource) that lets you query by example, or specify several different search/filter/paging criteria in one go. Really great APIs will have the foresight to allow plurals in filter criteria, especially when the ID is a potential criterion but even for other criteria (like, say, orders owned by any of 5 or 10 different customers). You're not trying to go nuts here and reinvent SQL; the de facto standard (in my experience) is that "fields" are combined with AND, field values are combined with OR. This completely natural to do in an HTTP URL: /orders?date=today&customerIds=6,7,8.

3. Asynchrony. This goes hand in hand with #2. The key here is to understand how users actually use your app. Users want a fast response time but don't process all of the information on the screen instantaneously. Say you're looking at a search results page; it's OK if the filters load a second or two earlier or later, and if there are annotations (like say, products you've already ordered, or reviews from other users, etc.) then those annotations can finish loading a few seconds later, as long as the most important information loads quickly. Or, on a dashboard page, you don't need each fragment to appear at exactly the same time. Note that this does not mean you make every request in sequence, because that would be insanely slow; typically a "page load" happens in 2 or 3 stages (first to get the data, second to decorate the data and load first-level relationships, subsequent steps to decorate relationships or load second-level relationships, such as information about the members who wrote the product reviews). During each step you make several parallel requests, because your API is clean and they are all independent.

4. A powerful composition and data-binding system on the client side. My areas of specialization are .NET and JavaScript, so I think of rich-client frameworks with observable support (WPF, Silverlight, etc.) or MV-* web frameworks (AngularJS, KnockoutJS, etc.). The reason for this is, let's say you're trying to populate something like a grid or table but aren't going to have all columns available on the first pass. These types of frameworks will automatically update the grid when the information does become available, which makes the "put all the data together" step largely unnecessary. On the other hand, Forms-style frameworks, while very convenient for slamming together quick-and-dirty apps, are absolutely miserable at composition, and you have to do a lot of extra work in order to maintain loose coupling, not to mention synchronization with the GUI thread.

5. Tests, tests, and more tests. APIs demand not only unit tests but also integration/acceptance tests, because they encapsulate both data and business rules. I'd be extremely surprised if an operation like GetUserHisFriendsAndHisPurchases actually has tests. When you have hundreds of these ultra-specific operations, the cost of maintaining such tests becomes prohibitively high. It's far better to have many different small and simple services then one or two mammoth services with no clearly defined scope.

Like I said, this approach is hard. It pays dividends in the end, especially on a largish dev team where different developers need to be able work on the different APIs and functional areas in general without stepping on each others' toes. But it takes experience and time to do well. That's why many if not most teams don't do it, especially not teams in a corporate setting where software is treated as a cost center and everybody would rather have fast and cheap than good.

I don't know if this is the right solution for you. But I do know that bloated APIs which try to do all the composition for you are a design smell and should generally be avoided when possible.

Answer 2

If you're following REST principles, then the definition of resources is pretty loose. So you can define granular resources, and collections of granular resources. But, over time, it might make sense to start exposing course resources that bundle the granular resources.

IMHO it's fantastic that you've tailored your API closely to the client's requirements. But if the client requirements are consistently met by bundled data, then that's fair game too. Sounds like you're getting into an optimization phase.

It's a design / implementation decision if you want to generalize the bundling logic, or just have code to bundle specific resources.

EDIT: After reading @Aaronaught 's answer I re-read the description. I agree with him, with api resources like "GetUserHisFriendsAndHisPurchases", "GetUserAndHisPurchases", you're kind of going off in the weeds. More on that in a second.

Also, the resources shouldn't involve the verb: "GetUserSummaryData" should just be "UserSummary" (I hope it's data!) and you use GET to get it.

Sounds to me like you need to step back and redefine your broad resources. Also, remember you can use the query string to broaden or narrow the resource. So "GetUserHisFriendsAndHisPurchases" would be much better as "user/{id}?includefriends=true&includepurchases=true" and the User resource can include the bundled dependent resources if requested.