77

Diablo 3 and StarCraft 2 have a nice feature: I can start playing before the full game or update has finished downloading.

I guess Diablo 3 downloads mandatory files like like UI assets and some meshes and textures that are used across several levels first. However, when starting a game, I can choose any level I want, assuming I have appropriate saves. I never experienced any lag from such when playing—starting is always very smooth.

How does this work? How could I implement a similar feature?

DMGregory
  • 134,153
  • 22
  • 242
  • 357
nikitablack
  • 1,279
  • 2
  • 10
  • 8
  • 2
    World of Warcraft too :P – Kroltan Oct 21 '14 at 22:36
  • 3
    How are webpages viewable before you download the entire web pages? How can Google Image search present an infinite list of images without using infinite time to download? Games are more sophisticated but they are quite similar. – Lie Ryan Oct 22 '14 at 22:14
  • 3
    @LieRyan: Regarding "Games are more sophisticated", that phrase has no fundament at all. There may be games that are more sophisticated, likewise there are (and looking at app-stores, that's the vast majority of software that does not solve serious problems) games that are massively less sophisticated than a massively scalable websearch that spreads thousands of servers and handles & stores billions of user requests each day. And then, search is just one of many features for that company. So please, pass the life-phase of thinking that games are inherently more complex than "normal" software. – phresnel Oct 23 '14 at 15:50
  • 2
    @jhocking: Art thou talking to me? (if so, better use @ so I will be notified) Nah, not really bitter. When I was younger, I thought that games-dev would be the king of the hill of software engineering. I just know better now, especially after the release of a number of source code of commercial games :) It's a common misconception that repeats to pop up. Games-Dev looks like Elite really, but in reality it's just another specialisation of software development, just like Super Computing, Search Engines, Image Recognition, Email Marketing, or ERPs for large E's. – phresnel Oct 23 '14 at 18:39
  • 1
    Just a quick comment, I've got poor bandwidth at home (cost of living in a rural area), and the game remains unplayable for me even though it says, "playable". I have to wait for "optimal". Otherwise I'm stuck waiting 5 minutes for every load screen and the lag is unreal, so its definitely downloading some stuff at that time. Can probably do further testing on your own by limiting your bandwidth. – DoubleDouble Jan 28 '16 at 16:57
  • WoW is playable within minutes of starting download. I can't believe the engine, opening cinematic, starting area models, textures, music, and sounds were all delivered that quickly. – Charlie Schliesser Jun 15 '19 at 15:48

4 Answers4

72

Assets such as sound, video, models, and textures are a majority of the download and for each of these assets there are multiple versions. These multiple versions are to support various graphic options.

By sending the assets needed for a low graphics option first (which also happen to be the smallest ones). You have everything you need to play the game despite having only 15% or so of the total assets needed to support every graphics option.

You can test this by re-installing StarCraft 2. Play it as soon as it will let you then go to the graphical options screen. Many options and setting will be unavailable.

ClassicThunder
  • 8,383
  • 35
  • 49
  • 24
    You can also apply the same technique to entire maps or levels in games with appropriate linearity - download the first levels first, or something. Guild Wars 2 only shipped audio and starting areas (plus a bit extra) on the disc, for example. –  Oct 21 '14 at 15:07
  • 13
    Some games (like Blizzard games) also have the ability to download assets on-demand, based on your current location in the game. This works much like how assets are often asynchronously streamed from the local hard drive, but in this case with a greater delay before requested assets become ready. – bcrist Oct 21 '14 at 15:33
  • The same is in Kill Shot. You can download missions on demand. But I don't know how such a system implemented technically. – MathNerd Oct 21 '14 at 19:20
  • 2
    IIRC, Diablo 3 uses the method described by Josh Petrie - it becomes playable as soon as the resources (possibly only the low-quality version as described in this answer), but only Act I is initially playable. That is, if you throttle your internet connection enough, you will be forced to stop and let the download catch up after finishing Act I. – Aaron Dufour Oct 23 '14 at 14:46
  • 1
    Stated another way, the game can technically run as soon as the executable is installed. This can be just a couple of megabytes of data. How far you can get into the game depends on how many assets are available. A game that installs in 20GB has about 19.9999GB of data and 0.0001GB of executable code; as long as it can continue loading data from the network, the game can run with less than 1% of installation complete. – phyrfox Oct 23 '14 at 16:15
  • The question asks "How does this work? How could I implement a similar feature?". This is a good answer for the first question, but does not address implementation. – Brian Oct 23 '14 at 17:58
  • 1
    @Brian Look at the edit history. It asked none of those things when I wrote the answer. – ClassicThunder Oct 23 '14 at 18:21
  • @ClassicThunder Ah, I see your point. The original question did ask "I wonder how it is implemented?" though. It's not proper English, but I think the intent was there. – Brian Oct 23 '14 at 20:00
  • 1
    @Brian implementation details would depend heavily on the game engine, mechanics, etc. etc. all of which we don't know. – jwenting Oct 24 '14 at 09:26
34

I once implemented this for MMOs. One weekend while attempting to install WOW for my daughter took 48 elapsed hours, (patches, download errors, etc.) so I decided to make my own better solution.

The game usually needs say 10 GB of data before it will run. Not all files are actually needed right away, but games used to wait until all files were locally present. My solution was to have the game run locally as normal, but the EXE was tricked (using a Windows file minifilter) into thinking all files were already present. When a requested file was not local, the file system downloaded it and saved it. The game was slowly copied locally as needed. When there was available bandwidth, the other not yet needed files were trickled in in the background. This worked with ALL games without the need to recompile, because my minifilter driver got files when needed.

The worst drawback was latency. My solution to that was to create a small Markov Chain model to predict which file might be needed next, and prioritizing the background loader. This worked like a charm, and our MMO was able to run almost IMMEDIATELY after just the EXE and a few loading screen files were local (~20 MB). We were able to click a link on a webpage and run our 10+ GB game in about 30 seconds. We had a 99% hit rate, meaning when the game needed a new file for the first time it was already there!

I'd be happy to help anyone else implement this.

Chris Mayer
  • 441
  • 3
  • 3
  • 5
    +1 and It would be great if you could share this driver because I was looking for something like that and was about to create it myself like you did but if it's already done I'd gladly use it. – Furkan Omay Oct 24 '14 at 08:27
  • 7
    Interesting use of some pretty advanced engineering. You should really write a blog post about this. :) How much did the machine learning techniques actually help? How well could they predict what files were needed? – Paul Manta Oct 24 '14 at 17:18
  • 1
    @FurkanOmay such things are almost always company property of someone's employer. Don't ask people to probably break the terms of their contracts. – jwenting Oct 30 '14 at 07:29
  • 3
    @jwenting From what I understand, what he had done wasn't for a company but for his daughter and for games like WoW. I'm just politely asking, nobody is forced. He can just refuse for any reason and everything will be okay. I asked because he said "he'd be happy to help anyone else implement this" so I don't think asking for at least a starting point is not a bad idea as I already said I'm willing to implement this from the ground up. – Furkan Omay Oct 30 '14 at 09:06
3

Also in some games end-game world areas and files are delayed, only the mandatory areas are kept, and can also be installed while still playing the game.

Chantola
  • 131
  • 2
2

To summarise what several people have said, and maybe add some information:

  • initial download is small, just the startup files and download manager for the rest. Would be nice to include account creation, avatar creation, and maybe the start zones in this.
  • while that's running, the user is logging in and creating their avatars, start downloading other stuff, low level zones and common files first (like basic sounds and textures, you do reuse those as much as possible all over your game)
  • as the user progresses through the levels, start pre-downloading upcoming zones and other content
  • if the user selects options that need downloading, download low resolution variants first, then as he's playing download higher resolution versions as needed and replace the low res versions displayed with those on the fly. That way the user sees something, and gradually the full glory of the game is revealed (if he has enough bandwidth and computer power of course)
  • repeat until everything is downloaded


It will mean a higher network load while playing, until all content is cached locally, but a smaller initial burst load on your servers as well as a shorter and less frustrating wait for users while they're initially downloading the game.
Of course if the user has a slow network connection he might still get "please wait, loading content" messages (do provide those) when switching zones, or you might want to check the bandwidth when initially starting the game and if too low to download reliably on the fly have him wait there and then for at least the entire first level (and then again when entering zones that haven't been fully downloaded yet).
This to prevent the user lagging out in the middle of actual gameplay because his network connection gets bogged down.

jwenting
  • 442
  • 2
  • 5