68

I've just finished a basic RPG game written in C++ SFML, I've put a lot of effort into the game and I'd want to distribute it, however I've ran into a small issue.

Problem is, I have well over 200 images and map files (they're .txt files which hold map codes) all in the same folder as the executable, when I look in the folder, it makes me want to cry a little bit seeing so many resources, I've never seen a game which shows you all the resources directly, instead I believe they pack the resources in a certain file.

Well, that's what I'm trying to achieve: I'm hoping to pack all the images in 1 file (Maybe the .txt files as well) then being able to read from that file or easily add to it.

Ambo100
  • 79
  • 2
  • 13
Bugster
  • 2,024
  • 2
  • 25
  • 32
  • 8
    See also: http://gamedev.stackexchange.com/questions/4118/custom-content-package-for-files http://gamedev.stackexchange.com/questions/24316/is-there-c-library-for-packaging-and-reading-multiple-files-to-from-one-file http://gamedev.stackexchange.com/questions/25502/how-can-i-create-a-pack-file-archive-for-game-data-files-and-then-load-them – Jari Komppa Sep 25 '12 at 07:26
  • I've seen code where all the images were packed into a string in the code. But that was done due to requirements. (It was a 64KB Java competition) – Omar Kooheji Sep 26 '12 at 14:44
  • Sorry it was a 4Kb java competition. Not a 64Kb one. – Omar Kooheji Sep 28 '12 at 08:57

9 Answers9

64

Game programmers have relied on one of two main methods of data storage:

  • store each data file as a separate file
  • store each data file in a custom archive format

The drawback to the first solution is the wasted disk space problem, as well as the problem of slower installations.

The second solution provides it's own pitfalls, first is that you must write all your own image/sound/etc. loading routines which use a custom API for accessing the archived data. A further drawback is that you have to write your own archive utility to build the archives in the first place.

Unless you will always load all files from the archive, TAR/GZ might not be a very good idea, because you cannot extract specific files as you need them. This is the reason many games use ZIP archives, which do allow you to extract individual files as required (a good example is Quake 3,​​​​ whose PK3 files are nothing but ZIP files with a different extension).

EDIT : "Hide" the game folder structure and "Keep" only the executables

Another solution is often used to "hide" the game files in folder structure. Keep only your executables and maybe a readme file in the main directory and move the game files into a sub folder named "data" or other related.

EDIT : Gamedev Tuts Plus have a nice resource

EDIT : libarchive

One potential solution libarchive, which is an archiving library that will handle extracting files from an archive such as a ZIP file. It even allows you to assign the extracted file to a standard FILE pointer, which would make interfacing with any other libraries potentially more straightforward.

EDIT : ZIP Format files with alternative extension

Here, I like @Joachim Sauer's Comment

Using ZIP-format files with alternative extensions has a great tradition outside of games as well: Java does it, the OpenDocument Format (a.k.a. the OpenOffice/LibreOffice format) does it, even Office Open XML (a.k.a. the "new" Microsoft Office format) does it.

Md Mahbubur Rahman
  • 2,867
  • 20
  • 30
  • 2
    Yeah, Thief/SystemShock2 used .zip files renamed something else too. In Java you can use something like TrueZip to access files in zips as if they were files in folders, I imagine there are similar libraries for C++. – Nick Sep 25 '12 at 10:46
  • 18
    Using ZIP-format files with alternative extensions has a great tradition outside of games as well: Java does it, the OpenDocument Format (a.k.a. the OpenOffice/LibreOffice format) does it, even Office Open XML (a.k.a. the "new" Microsoft Office format) does it, ... – Joachim Sauer Sep 25 '12 at 12:46
  • Why would the first solution waste space? I mean, media files are often already compressed aren't they, so why would sticking them all in one archive save a lot of space? – Svish Sep 25 '12 at 16:29
  • 5
    @Svish. Depending on the platform, the amount of disk space taken up by a lot of small files can be considerably greater than the amount taken up by one large file, even if that large file is not compressed at all (.tar, for example). It's to do with the way data is physically stored on the disk. – TRiG Sep 25 '12 at 16:33
  • 1
    Ah, that's true. Shouldn't be much though, unless there are a lot, but in that case it can indeed build up. I can imagine installation and uninstallation also takes a hit with lots of small files. – Svish Sep 25 '12 at 17:01
  • 1
    The 'disk space' negative should be thought of as a deficiency in the filesystem implementation. I really hate random data formats that are just a whole ton of things that should be separate files shoved into one thing. That's what a directory is for, and the filesystem should be making that use-case efficient, and in fact can make that use-case far more efficient than any archive format. That being said, you have to code for reality, not an ideal. But I think keeping the ideal in mind is always a good idea. – Omnifarious Sep 25 '12 at 18:04
  • 1
    Unfortunately a general purpose filesystem knows nothing about what order your application accesses that data and how frequently it does it, and that can be a factor in choosing to consolidate the files together. – Kylotan Sep 25 '12 at 18:15
  • 2
    Slightly related to Kylotan's comment, here is a post by Jonathan Blow on The Witness' dev-blog about using archives and how they interact with Steam's patching mechanism. "The Witness packs most of its data into a single 2-gigabyte file stored in the .zip format. Before uploading the initial version to Steam, I took a safeguard to minimize patch sizes, or so I thought [...] (Imagine that files are stored in random order: probably every patch involves every player re-downloading the entire game!)" – Eric Sep 25 '12 at 18:25
  • 1
    IIRC, in the past, quake 3 had loading problems because certain AV programs realized that the pak files were ZIP and were apparently re-re-re-scanning at map load time. (Often an AV update would trigger the behavior, masking the cause). Am I correct in my recollection, and are there any strategies to avoid this? – horatio Sep 25 '12 at 19:18
  • @Kylotan: Actually, a really good filesystem can track those things and figure it out. Seriously. btrfs for Linux is working on a way to automatically move things to and from SSD storage based on usage patterns. – Omnifarious Sep 26 '12 at 06:20
  • I would say the biggest drawback of using a archive format would actually be that it adds a additional step and tool to your workflow and that you are making the access to the data harder for the modders. – API-Beast Sep 26 '12 at 10:07
  • @Omnifarious - you keep talking about the filesystem as if it's something the app developer can just ask to have changed. They can't, and from their perspective it's barely any more practical than asking CPUs to have a better instruction set. What someone is working on is irrelevant - what is in place now is what matters. – Kylotan Sep 26 '12 at 12:19
  • 1
    @Mr Beast: on the contrary, usually when you use an archive format you can deploy files to the filesystem as well which override the ones in the archive. This actually makes it easier for modders because they can change the game data without having to alter the original data. – Kylotan Sep 26 '12 at 12:20
  • @Kylotan This has nothing to do with having a archive or not, you can just as well have multiple directories, one for the the base data and one for each mod. – API-Beast Sep 26 '12 at 12:48
  • @Kylotan: I think app developers should demand more from filesystems. I'm really tired of having people contort their app designs in sad ways in order to handle the restrictions imposed by an OS vendor who refuses to actually do decent development on their filesystem. But yes, it's true, ultimately you have to code for reality. – Omnifarious Sep 26 '12 at 12:51
  • @Mr.Beast - but the point is that you can have the benefits of the archive system without the drawbacks you mention. – Kylotan Sep 26 '12 at 13:01
  • @Kylotan Can the modder find out how the data of your game is structured without first finding out what archive format you are using? No, he can't. – API-Beast Sep 26 '12 at 13:25
  • If you actively want people to mod your game, it takes less than a minute to put a readme.txt in the data directory telling them the format of the files. – Kylotan Sep 26 '12 at 14:41
  • @Kylotan Still they need a additional tool in their workflow. – API-Beast Sep 26 '12 at 15:01
  • @Mr.Beast Renamed .zip file is a common enough technique that I suspect it would be hte first attempt of any half-serious modder. Also, format matching via file header is becoming fairly common in 3rd party tools. Unless you use a propriety format trying it with an archive tool should figure it out automatically. – Dan Is Fiddling By Firelight Sep 26 '12 at 17:12
  • Please use the chatroom (or a chatroom) for extended discussion. – Jesse Dorsey Sep 27 '12 at 00:14
  • 1
    I think all these comments are relevant to the answer. – Kylotan Sep 27 '12 at 11:19
  • 1
    I picked the ZipLoader option from this answer, and for whoever came here from google DO NOT use it. After 12 hours I'm having some errors which, after over a year of programmer are stunning me, not to mention the framework link didn't even come with the DLLs, or even instructions on how to use. NOT recommended. :( – Bugster Sep 27 '12 at 17:15
  • 1
    Also tried PhysFS, the result was obvious: ||Edit the max errors limit in compiler options...|

    :(

     – Bugster Sep 27 '12 at 17:56
  • 1
    @ThePlan, Yes, You are right. I have tried ZipLoader and PhysFS. Not better. For this reason, i have removed ZipLoader and PhysFS from my Answer. – Md Mahbubur Rahman Sep 28 '12 at 03:28
39

Another solution often used to "hide" the game files is folder structure. Keep only your executables and maybe a readme in the main directory and move the game files into a sub folder "data". I don't think that it is very uncommon to do so. Many games I know store their content in such a way.

tom van green
  • 1,622
  • 12
  • 13
  • 8
    +1 If disk space or protection isn't an issue, that's the easiest fix. – Laurent Couvidou Sep 25 '12 at 08:14
  • 1
    +1 because the other guy said +1. Can't be wrong, eh? (Joking aside, great answer.) – jcora Sep 25 '12 at 14:20
  • This, in my opinion, is the ideal choice. If the filesystem can't handle that efficiently the filesystem is broken and you should be leaning on the OS maker to improve it. – Omnifarious Sep 25 '12 at 18:05
  • 3
    @Omnifarious That's not always true, e.g. when reading from an optical disk, even with a perfect filesystem, it's common to pack files in loading order to prevent excessive seeking (the speed up can actually be quite surprising). But this is a totally different story for hard disks, and I tend to think the OP is in this case. – Laurent Couvidou Sep 25 '12 at 19:17
  • @LaurentCouvidou Though single files can be just as fragmented as multiple files. It's really the file system that should take of it, not the game programmer. – API-Beast Sep 26 '12 at 10:10
  • 3
    @Mr.Beast You missed my point. I'm talking about optical disks. I've worked for a studio where it was a single programmer's full time job to make sure files would be packed as efficiently as possible on a DVD, to get the best loading time possible. I'm sure he'd be glad to learn he could just have relied on the DVD file system ;) – Laurent Couvidou Sep 26 '12 at 10:14
22

I really like PhysFS for this. It allows you to access either folders or zip archives with the same code. It works well for all stages of a Games lifetime.

  1. During development: access the resources directly from a folder hierarchy. This way compressed archives are not in the way and you can rapidly iterate.
  2. Initial deployment: zip up your resources for easy deployment / quick installs. No code needs to change. Just point PhysFS at the zip instead of the folder.
  3. Updates / Modding: PhysFS can load multiple zip archives where the resources from one override the other. Updates can be as simple as a new zip with only the changed files. Similarly for modding.

update:

I used the tutorial included with the source and the doxygen documentation to learn PhysFS. The API really is fairly simple, you'll spend more time learning how to load images into SFML via memory buffers instead of by file path.

Community
  • 1
deft_code
  • 7,602
  • 5
  • 37
  • 54
  • 4
    I'm using PhysFS for my current project and I love it. I don't always need to regenerate my content archive file; I can just drop an updated version in the search path and BAM! It works. – Zack The Human Sep 25 '12 at 17:37
  • After some answers I decided I'll go with physFS. Would you care to recommend a nice tutorial? :) – Bugster Sep 25 '12 at 17:51
12

I'd suggest using a ZIP file. It's an ubiquitous format and you'll find ready-made libraries that allow you to load files from within a ZIP file. A quick Google search even revealed a zip loader for sfml.

bummzack
  • 22,646
  • 5
  • 61
  • 86
3

Well, you could cheat like they mentioned:) You can make a sprite-sheet from the imagery, there is no need to zip it nowadays unless it saves a ton of space. After you create a sprite-sheet or several sprite-sheets from the images, you could simply rename their extensions. Most Gamers will not bother to check and why not leave that option to Artists who may want to mod your game in the future? That way they can too create a sprite-sheet, rename it's extension and start rolling.

With the text files, I would humbly suggest you convert that data into binary form. I am pretty sure you will find a simple way to do it.. For instance, if you only use A-Z, a-z and the 0-9, you can use 6 bits to represent each character, that will somewhat protect your copyrighted material, it will also prevent others from editing maps.. You can always add a map converter if you like. Zipping is completely reasonable for text though.

AturSams
  • 10,517
  • 1
  • 32
  • 58
  • Converting to binary is good advice but not for protection - it's just going to be easier to parse and faster to load. Would suggest it as a pre-processing step, but that's going to off-topic for the question. – Maximus Minimus Sep 25 '12 at 14:04
  • Well, I am guessing, if he wants to protect it more effectively, he could try to encrypt it in RSA or a similar algorithm(there is no need). I doubt anyone will make unauthorized use of the map files from an indie game if it's inside a binary file. It just doesn't seem worth investing the time in figuring our the parsing. Text files are just completely vulnerable and as you suggested something ineffective for storing data. – AturSams Sep 25 '12 at 14:13
  • 3
    On HDDs, loading a single zip-file is faster than a lot of small files because there are less expensive random seeks to the physical disk. A binary for a small game can be mapped into memory and work "magically", almost no parsing required. – Liosan Sep 25 '12 at 14:16
  • @Liosan You could have misinterpreted what I meant or I misunderstood you.. I said binary is better for storing data and that other game designers are unlikely to want to learn how to parse your binary map code, unless it is documented and encouraged for modding purposes. Therefore it offers some rudimentary protection over text files. – AturSams Sep 25 '12 at 14:23
2

It's not only about the number of resources or disk space.
With a lot of different texture files, as you are using SFML, which renders with OpenGL, each time you are going to render a texture, OpenGL needs to bind the next texture to the video card (check glBindTexture() ), and it is a really expensive task.
With that in mind you can realize why games usually put a number of sprites in the same texture, the so called spritesheets, according to your menus/levels/maps of the game.
The result is a huge gain in performance, since you are rendering a lot of sprites with the same bind texture call.

Edmo Freitas
  • 21
  • 1
  • After getting several references to making a spritesheet I've considered some refactoring to lower the image count. Thanks – Bugster Sep 25 '12 at 19:42
2

I would personally head down the custom binary file route for this. This is something I do myself all of the time now, it isn't as difficult as it might sound, and it also provides a level of obfuscation for you game resource files. I wrote a small introductory Gamedev article about this not so long ago if you are interested:

http://gamedev.tutsplus.com/tutorials/implementation/create-custom-binary-file-formats-for-your-games-data/

Sprite-sheets are a different subject entirely but they are the norm for most games :)

Si Robertson
  • 21
  • 1
1

Another method you can use:

The free version of XnView provides a feature called "Strip of Images" (SHIFT+A), which allows you to create sprite-collections.

This minimizes file access and is particularly important for web applications/games to minimize the number of HTTP-roundtrips.

Access to individual images is then granted through coordinate maps (for instance css-definitions).

House
  • 73,224
  • 17
  • 184
  • 273
Lorenz Lo Sauer
  • 352
  • 2
  • 9
0

First you must write all your own image/sound/etc. loading routines which use a custom API for accessing the archived data. A further drawback is that you have to write your own archive utility to build the archives in the first place. Unless you will always load all files from the archive, TAR/GZ might not be a very good idea, because you cannot extract specific files as you need them. This is the reason many games use ZIP archives, which do allow you to extract individual files as required (a good example is Quake 3,​​​​ whose PK3 files are nothing but ZIP files with a different extension). http://zpp-library.sourceforge.net/