2

I have a proprietary file format that is a compressed database file. This database file has a few dozen tables. Each of these tables only have a few records, many of them don't have any records at all. A few of these tables contain fields that are stored as blobs of hex data. These blobs account for 99% of the disk space of the overall database file.

As far as I can tell, these blobs are not compressed data (by using unix 'file' command). I have tried finding known values in these blobs by exporting values from the proprietary software, converting to hex and searching for that value in the database file. So far I haven't been able to find any matches. The problem is that the software can export in a myriad of formats and I'm not sure which one (if any) the data would be stored in.

Most of the tables contain checksum fields, which I believe, are responsible for my inability to edit the blobs and see what changes in the proprietary software. This combined with the fact that I cannot directly change the values that I wish to extract from the proprietary files leaves me in a difficult position.

Does anybody know any tricks for trying to tease out time series data from binary data?

Edit This zip file contains 2 hex blobs (index and value) from the decompressed database and the same data as it is exported from the program.

horriblyUnpythonic
  • 133
  • 1
  • 6
  • If you change one single byte (character, number?) in a copy of your database and then compare it byte for byte with the original file, do you see a comparable change? (I.e., on only one or two positions, plus -- optionally -- the checksum.) If the file size changes, the database may be compressed. If the size is the same but lots of other data changes, it may be encrypted. – Jongware Nov 16 '13 at 03:32
  • Wait, you said "I cannot directly change the values". So this is not a database that is actively read/written, correct? – Jongware Nov 16 '13 at 03:33
  • Yeah, the software is such that it can only get the values I care about from an actual run with real hardware. I haven't been able to find anything that will allow me to directly change those values. – horriblyUnpythonic Nov 18 '13 at 18:51
  • Can you share any sample file? At the very least I could check if it contains anything I can recognize by eye. – Jongware Nov 18 '13 at 20:14
  • I have added a link to the question. – horriblyUnpythonic Nov 18 '13 at 21:08
  • ... Sorry to be a bother, but can you use an upload service that doesn't require signing up? – Jongware Nov 18 '13 at 22:20
  • I didn't think this one did, sorry about that. I have updated the link. – horriblyUnpythonic Nov 18 '13 at 23:43

2 Answers2

3

After seeing the test files (thanks!):

... too easy :)

In value.hex, the first dword appears to be a total file length; the second dword the data length. The third and fourth dwords may be flags of some kind and do not appear to point to data. This lops off the first 16 bytes, hence my guess the 2nd dword is 'data length'.

Right after this header comes that familiar pair 78 9C again, so I brought out my zlib decoder wrapper. Unpacking value.hex, starting at offset 0x10, and using TINFL_FLAG_PARSE_ZLIB_HEADER (as I am using miniz.c for eaze) gave me a correct unpacking result and a data file of 33,208 bytes long.

Inspecting this with 0xED shows this file consists entirely out of double values (8 bytes each); the first few are

0.991932
0.991931
0.991932
0.991932
0.991932
0.991933

(okay, there appears to be a pattern here -- the devil is in the last few digits which 0xED doesn't show, they are not all the same values).

The second file, index.dat, also unpacks correctly and gives another long list of double values, this time clearly going up:

0.0000
0.0082
0.0163833
0.0245667

I didn't cross-reference these values against the XLS file you provided, I assume you can work that out from here.

I only unpacked until I got a positive result back, I did not check if there are more data packets (compressed or otherwise) following the first one and you should verify using the end result of your own favourite decompression routine.

Just as I was heading to bed, it struck me that the 3rd and/or 4th dwords in the header (weren't they the same anyway?) may be the 'unpacked' length.

Jongware
  • 2,364
  • 2
  • 16
  • 30
  • Great! It took me a while to figure out what you meant in that second paragraph, but I think I've got it now. I've never seen/used the 78 9C magic number, so that was a lifesaver that you noticed it right off. I'm assuming that TINFL_FLAG_PARSE_ZLIB_HEADER is just a way to skip the first 16 DWORDs. Is that right? I'm using python so I got it from zlib.decompress(hexData[16:]) and then reading in little endian. – horriblyUnpythonic Nov 19 '13 at 02:23
  • No, that flag was there to inform my unpacking routine it should expect a zlib header, and you indeed need to skip the first 16 bytes manually (4 dwords). For more info, see the comments in Where could one find a collection of mid-file binary signatures? – Jongware Nov 19 '13 at 09:39
  • Cool, good to know. BTW, as far as I can tell, you were right about 3rd and 4th header DWORDs being the unpacked length. – horriblyUnpythonic Nov 20 '13 at 00:13
2

Since you did not give any details of background, I'll attempt a wild guess. It depends greatly on your application field. For example in the medical field, most algorithms used are very well known, and therefore you can simply skipped a bunch of binary data to find well known algorithm such as zlib for meta-data and/or simple encoding such as RLE for images.

What I would try is create a string such as "super calli fredgulistic ex pe alli doschus is a cool word". And then export the file in all possible format, then simply grep for that in all exported files. Eg:

$ grep "super calli fredgulistic" *

or if your strings is in 16bits little endian:

$ strings -e l output_format | grep "super calli fredgulistic"

You'll find the format the most easy to deal with (well hopefully!).

tibar
  • 375
  • 4
  • 18
  • I tried to say as much background that was applicable in those first three paragraphs, but I think I see where I left out information. – horriblyUnpythonic Nov 18 '13 at 18:55
  • As far as I can tell, are no ways to change any raw data or strings that would get exported. The only thing that I can export is the raw data (numerical) in multiple ways, basically just changing the scales of the numbers (e.g. changing time from minutes to hours). I have been unsuccessful in finding any of these values in the uncompressed file.

    \P What do you mean by, "therefore you can simply skipped a bunch of binary data," was that a typo?

     – horriblyUnpythonic Nov 18 '13 at 19:03