Universal Standards for Data Interchange - Do they exist and do you follow them?

Question

Mostly fact and maybe a little bit of opinion:
One of my pet peeves in programming is data interchange. I work exclusively with small business software (as opposed to working with corporate ERP systems) and I see that many small businesses store contacts two or three times in different types of software where there is little or no interchange. Time clock systems usually don't integrate with calendaring and project management systems. Task management may be in a system separate from project management, or more commonly, tasks are managed on paper or mentally. These are just a few examples of the data interchange problems I see every day in almost every small business.

It seems every software company has a different idea about how to store it's data and whether or not to expose that data for interchange and integration. If they do expose some of their data there is really no common standard to facilitate interchange or synchronization. You're always forced to read through the SDK/API documentation if you do want to program some type of integration and then it's almost never a painless experience to get it working and keep it working.

It would be really nice to receive an "electronic document" from our vendors that would allow us to quickly enter bills into our accounting system. It would be really nice if we hire a new employee that we could make one single entry somewhere, enter his roles and then he would be entered into all the correct systems even if his information would need to be flagged for review in systems such as accounting. It would be so nice if all our contacts were seamlessly shared between systems so that updating an address or phone number in one of them would make the changes in all of them.

Questions:

1. Do data interchange standards for common entities such as people, purchase orders, sales orders/invoices, shipping documents, projects, tasks, appointments, etc. already exist?

2. Assuming some do exist, are they commonly accepted, ratified standards?

3. Would you program a business system to use or follow these standards if the initial requirements of the project did not call for it? In other words, how common or accepted are these standards, enough to be followed every time?

Answer 1

Do data interchange standards for common entities such as people, purchase orders, sales orders/invoices, shipping documents, projects, tasks, appointments, etc. already exist?

Yes. That's what EDI is all about.

Start here: http://en.wikipedia.org/wiki/Electronic_Data_Interchange

Assuming some do exist, are they commonly accepted, ratified standards?

Absolutely.

Would you program a business system to use or follow these standards if the initial requirements of the project did not call for it?

No.

In other words, how common or accepted are these standards, enough to be followed every time?

The issue isn't "common" or "accepted" it's "cost" and "value". Sometimes EDI doesn't create enough value. Sometimes it's absolutely essential for working in the given industry.

Answer 2

Realistically, no, there probably are no interchange standards and there probably never will be-- the entities you're talking about are likely too complicated to create a standard that everyone can embrace that are not also so hideously complicated that everyone hates using it.

In various domains, there have been attempts to define interchange standards. In health care, for example, HL7 defines standards for defining various common entities like patients. These are widely used in software but everyone that deals with them tends to hate doing so because the standard is always wildly more complicated than the individual system needs (or really wants) to handle. The problem always arises that while every system only needs a relative handful of patient-related information, in order to become a standard, you have to embrace the requirements of everyone that uses the data. That means that the standard PERSON record, for example, would likely need to have about a dozen different addresses and would need to encode rules for determining things like your "current mailing address for a particular type of document" because some systems let you define multiple mailing addresses with various rules for interpreting them (i.e. Person A lives in Rhode Island from March to September and goes to Florida from October to February but financial statements should be directed to her accountant in Massachusetts). And for the 99% of systems that don't actually need to have that level of information, writing the import process is a royal pain because you have to figure out what scenarios the standard actually allows and figure out how to map all those scenarios into your system automatically. That's one of the reasons that no one ever speaks fondly of dealing with HL7 data.

Moving out into general purpose software with much smaller companies, the cost and complexity of writing that sort of import code isn't worthwhile. If you're a large medical software company and you know that your system needs to interoperate with lots of other systems, it's financially worthwhile to use something like the HL7 standard even if you spend a ridiculous number of hours talking about corner cases because some system you're interested in might produce a record with that format because you then don't have to write custom import and export routines for every other medical system in the world. If you're selling to a small business, however, it's almost certainly much less costly to write 3 or 4 import & export routines for the 3 or 4 most common types of products that a small business would have (i.e. import & export accounting stuff to & from QuickBooks, exchange contacts with Outlook) than to try to hit some massively overcomplicated spec even if you could get the momentum to produce one.

Update: What about EDI?

EDI is a very generic term. There are multiple, competing standards for what elements compromise a person or a purchase order. And, like HL7, each of the EDI standards is likely to have far more requirements than any given piece of small business software really wants to deal with. For example, there are separate ANSI EDI standards for a Purchase Order and a Grocery Products Purchase Order. If QuickBooks wanted to generate a "standard" EDI Purchase Order, it would need to know whether grocery products were involved, potentially on a line-by-line basis so that separate grocery- and non-grocery purchase orders could be created. So there would be a number of data elements QuickBooks would need to gather that would annoy 99% of their users in order to be able to meet the ANSI standard for generating a Purchase Order message. Of course, there are also SAP, Oracle, and dozens of other purchase order EDI message format standards.

And even if you can assemble the message, then you have to deal with exchanging it with someone else. Your vendor would need to know how to deliver the EDI message to you. That would generally require that your small business runs a server on the internet that can accept EDI messages and relay them to QuickBooks which can generate and send the acknowledgements. That introduces a number of moving pieces that QuickBooks has to code and deploy, that your business has to maintain, and that your vendors have to deal with. What if your vendor processes invoices while your EDI server is down? How do you handle ensuring that the Purchase Order Change that was sent last night to QuickBooks gets turned into some notification to you that the customer has changed their order from 100 red widgets to 200 blue ones?

Answer 3

We do have a few: http, xml, json, tcp/ip.

I lived through the XML schema wars of the early part of this century. You know, that few year period where we thought we could use this magic XML stuff to create a data definition for everything and then use it everywhere.

The object lesson was the problem with universal "standards" is that they tend to be designed by committee and a try and cover every permutation. Leading to typical designed by committee problems such as over-engineering, over-emphasis on [INSERT BIG IMPORTANT ENTITY]'s marketing objectives for that quarter. They rarely work.

The standards that do work are born in the field and crafted under stress. They tend to do what is needed and not more and be very flexible and succinct. Things you'll never get with an architecture committee.

Answer 4

There is for example an Open Catalog Interface designed by SAP. It's primarily used in the SRM (Supplier Relationship Management) domain to integrate external vendor catalogs int existing systems.

Answer 5

One that I have used is here: https://schema.org/docs/full.html. They have definitions for lots of things, from recipes, to various types of businesses.

It is used in many places too. For instance, Amazon orders etc contain Schema from here in their emails, and when you open up their email in Gmail or Inbox, you see a summary of what you ordered, the price, and others.

Answer 6

1) Do data interchange standards for common entities such as people, purchase orders, sales orders/invoices, shipping documents, projects, tasks, appointments, etc. already exist?

Yes and no. The glib response is "the nice thing about standards is that there are so many to choose from".

There are some standards like vCard which is meant for contact management that will work with email applications, but typically not much more than that. Purchase orders and sales orders/invoices are going to be proprietary to the system--a different format for Quickbooks Pro vs what you would use for SAP. You can use iCal for appointments, which seems to work for calendar tools but are a pain to consume in your tool. You might be able to extend iCal to handle tasks (it has a data type for to-do items). As to projects, again those are proprietary.

Part of the problem is that there is no compelling reason to standardize for the people who make these products. When the data interchange is standardized, then that means it is easier to swap out any part of the system. That's a scary proposition for large cumbersome products that want to get you hooked because it would be too expensive and time consuming to change.

2) Assuming some do exist, are they commonly accepted, ratified standards?

vCard and iCal standards are commonly accepted in some settings. Some project management tools will export data using these standards so that a user can have the information on their calendar. However they are rarely used to exchange appointments and contacts system to system. Not sure if there are technical reasons for that, or if it is just something not enough people thought about.

3) Would you program a business system to use or follow these standards if the initial requirements of the project did not call for it? In other words, how common or accepted are these standards, enough to be followed every time?

It really depends on the audience. Every project needs to evaluate whether using the standards are appropriate, make good business sense, and satisfy a real need. In some cases it would be a slam dunk to say "thou shalt use vCard and iCal for exchanging information, thus sayeth the almighty contractor!" In other cases the tool is just too small to warrant the overhead--or the client will never use the feature anyway.

Other standards like RSS can be consumed in creative ways, so they also make a lot of sense provided the data supports it. It depends on the data and the need.

Answer 7

1 Do data interchange standards for common entities such as people,
  purchase orders, sales orders/invoices, shipping documents, projects, 
  tasks, appointments, etc. already exist?

Yes there are for some of these. Sometimes there are competing standards. Usually there are good reasons for these different standards. If you are designing a system, the existing standards are a good place to look for information on attributes you might want to capture and store.

• For people standards include LDAP schemas, X.500, and vcard. There is an international postal standard for the layout of addresses.
• For business transactions there are EDI standards. Unfortunately, industries have specific and differing requirements.
• For appointments the only standard I have used is ical.

There are other data standards I am not aware of or have not listed. I have dealt withe the above standards to some level.

Beyond these, there are a number of underlying standards we use such as ASCII, UTF-8, FTP, SCP, SMTP and others. These are building blocks that make the interchange of data possible.

2 Assuming some do exist, are they commonly accepted, ratified standards?

Lower level standards are all ratified and commonly accepted. When you get into actual data formats, many are ratified or commonly accepted.

3 Would you program a business system to use or follow these standards
 if the initial requirements of the project did not call for it? 
 In other words, how common or accepted are these standards, enough 
 to be followed every time?

The simple answer is they apply to data interchange, and not to business requirements for systems. As such they really aren't relevant to most systems. The standards are most useful when a system needs to exchange data with a large number of systems in other organizations. In that case I would follow the appropriate standards.

When programming a business system, I would refer to the appropriate standards to validate the data model. I would not add any attributes because they were required by an interchange standard. Most systems have little need to interchange data with external systems, and the interchange standards are not the best method to do internal data transfer.

If and when a system was required to communicate with another system, then I would consider how best to build the interface module. An important consideration at that time is how to select and secure the data being transferred. It would be critical to ensure that only the correct subset of data is transferred.

My primary concern in build a business system is what data is required by the business. Data recorded in the system needs to reflect the needs of the business. This may include recording data not reflected in any standards, or omitting data that is required by a standard.

Most of the data interchange systems I have worked with have involved transfer of data from one system to another. While standards are useful in such cases, they may require far more effort than is necessary.

It would be wonderful if I could update my contact data in the one true place and have everyone who needed it get the appropriate parts. However, the one true place would also need to ensure my privacy, and prevent unauthorized access to data. Instead we have a piecemeal process where different systems have different data. Many stores have obsolete data about me, but I am fine with that. I will update the data when I consider it important. However, I may not give all the data they might like.

Answer 8

It would be nice if there were standards. But there aren't. It really depends on the requirements when the software was written. However, if you're always converting data from the same source, there is software that can help with that process. Personally, I have used Data Junction in the past. It saves time from programming your own.

Tom

Answer 9

1. yes. several. some try to do it all, and fail. some do too little. variety is the spice of life, and the bane of standards. the real world says: use the standard that works, ignore the standards that don't. if no standard exists and you really need it, create one.

2. some are, some aren't. some are ancient, complex, and clumsy (EDI, for example). some are vendor-specific and often client-mandated.

3. no, not if the scope of work did not include data interchange using a specific standard.

In many systems, there is no advantage or value in adding data interchange standards and services, especially when there is no obvious/pressing need for them.