Cocos2d-x resources - "On-the-fly" resource updating


"On-the-fly" resource updating (Android + iOS, C++)


To run your app on a device, you typically connect the device to your computer by USB cable to upload the app. When you change something, you need to upload the app again every time to check your changes. Although this works fine, I find it to be quite tedious because in most of my previous work (desktop apps, websites) I am used to being able to run and check things very quickly.

To elaborate on what I mean by tedious:
  • the entire app is uploaded every time, which for large apps can be quite slow
  • updating multiple test devices requires physically connecting and uploading to them one by one
  • sharing a change with a team member requires them to build the app again
  • pushing changes to testers requires them to download and install the app again
Depending on what you're making, you might be able to get by with the simulator. But for games that need accelerometer input you really need to test on a real device. Testing on a real device continuously throughout development is good practice in general because problems and performance issues will show up much quicker. Managing multiple touches is another thing which can often cause problems that don't show up easily with a simulator.

It occurred to me that the only part of the app that really needs to be uploaded every time is the executable file itself. All other files (images, sounds, setting files, level maps and other info loaded by the program at run time) could potentially be obtained externally, without having to re-upload the whole app.

The obvious answer is to have the data files accessible via network, and make the app download them. But simply downloading all the data files every time would be much slower than getting them through USB cable, so to do this efficiently requires that the app can somehow check which files have been changed, and only download modified ones.

It takes a little setup to do this because we'll need a webserver running PHP to help out on the serving end, but once the system is in place I think you'll find it to be a huge time-saver. Here are some of the benefits of using this method:
  • data-only changes do not require building or re-uploading the app via USB
  • data files are not uploaded, so uploading is very fast
  • changes are immediately accessible from all test devices
  • sharing changes with a team member or tester does not require them to build or install
  • changes are immediately accessible worldwide (if your server is public)
  • multiple sets of data can be switched between easily
Here is a video showing a simple example of using this method:


This system turned out to be one of the biggest improvements I have discovered for mobile development (after cocos2d-x of course :)
Let's take a look at the details of how this works.


Overview


The core of this setup is detecting which files have been changed. This is done by the OTFLayer class, which is a subclass of cocos2d::Layer. Here is how the flow progresses:
  1. When the app starts up, OTFLayer must be the first scene shown
  2. OTFLayer downloads 'otf-manifest.php', which is a list of all files available on the server along with their md5 hash
  3. Each entry in the list is checked to see if it already exists on the device, and has the same md5 hash
  4. Files that do not exist or have a different hash are downloaded
Hmm... files are downloaded... to where? Well, we can't put files into the app bundle itself because that is read-only, so we put them in the private storage folder (FileUtils::getWritablePath) of the app. But by default, the app does not look there for data files, eg. to load a sprite image etc it will look in the app bundle. Fortunately, cocos2d-x makes it very easy to set a different root folder to look for data files (FileUtils::setSearchPaths).


Server setup


To use this OTF system you will need a webserver with PHP enabled. I have only ever used Apache for this because it's free and easy to get running, but any webserver that does PHP should be ok. There is plenty of info on the net about this and only the most basic setup is necessary, so I will not go into details here. If you're using MacOSX 10.0 or later then I believe you will have both Apache and PHP already, and you just need to enable or turn them on somehow. On Fedora Linux you can type this in a terminal as root: "yum -y install httpd php ; service httpd start". There are installers for Windows around.

Once you have the server running, make a folder somewhere under the docroot to act as the root resource folder for the app, and put all the data files for your app into this folder. Alternatively, you could make a symbolic link to your working folder where you already keep your data files, to avoid keeping multiple copies. Make sure the firewall on the server is set to allow HTTP connections, and that your devices can access it over wifi.

Place this file: otf-manifest.php in the resource root folder along with your data files. To check that everything is set up correctly, access the manifest URL directly from your browser, and you should see some JSON text. As a guide, you can click here to see the output for the example in the video above.


App setup


The OTF system uses some extra file utils, and the ScreenLog class. You can get these from the other topics linked to at the top-left of this page. In AppDelegate.cpp, initialize the screenlog as shown in the screenlog topic.

For the OTF layer itself, add these files to your project: otfFiles.zip

At the top of OTFLayer.cpp are some settings you'll need to change to the appropriate values for your app:
1
2
  string otfSourceURL     = "http://www.iforce2d.net/otfdemo/";
  string otfLocalFolder   = "otf";
The source URL is the location on your server where the data files and otf-manifest.php are located. The local folder is the location within the writable folder of your app where the downloaded files will be kept.

In your AppDelegate.cpp, include OTFLayer.h and make an OTFLayer the first scene when the app starts:
1
2
3
4
  // in applicationDidFinishLaunching()
  auto scene = OTFLayer::scene();
  g_screenLog->attachToScene( scene );
  director->runWithScene( scene );
In OTFLayer.cpp, the onTouchesBegan function will trigger a scene replacement after the OTF process has finished. Change the replaceScene call here to use the layer that your app would normally have as the first scene.

Optionally, remove all data files from your project. This is not strictly necessary because data files in the app bundle will be ignored due to FileUtils::setSearchPaths being set to the writable folder. However, if your app has a lot of data files, you can significantly shorten the time it takes to upload it to the device over USB by removing them completely.

To allow updating the data files while the app is running, you will need some way of showing the OTFLayer again. Usually I do this by putting a button on the main menu screen that is removed for final release. You could also detect the 'shake' gesture, or use the soft-key back button on Android.


Reverting to normal method for release


Obviously you would only use this system during development, and for release you would revert back to including all data files in the app bundle. To revert, you just need to change AppDelegate.cpp to load your normal starting scene instead of OTFLayer, and add the data files to the project again.


Some points to note


You may have some files on the server that should be ignored by the OTF system. In otf-manifest.php you will see a section where some files are ignored, so you can modify that according to your needs.

Currently this system checks for new or modified files, but it never deletes any files from the device. If you remove a file from the server, it will still exist unchanged on the device. If you really need to have files deleted, try bugging me about it and maybe I can implement that.

Currently this system does not support filenames with spaces in them.

If you use a publicly visible server, keep in mind that anyone will be able to download your data files if they know the URL. Unlike a normal web browser, the OTFLayer has no concept of sessions, https or even basic HTTP authentication. I don't think this will be a problem for most users of this system, but if it was, I guess you could make the app send a password as a POST parameter and change the accessed URLs to be parameters to a php that checked the password and passed through the appropriate file. The server could also be configured to only answer requests from specific IP addresses etc.


Switching between multiple sets of data files


Sometimes when working in a team, other people will have slightly different data files depending on who is working on what part of the app. By changing the source URL to that of your team-mate you can easily switch to using their version of the data, and only download the subset of files that differ. (This will require a rebuild and USB upload, but if you could make the URL modifiable from within the OTFLayer via a text input if you really wanted to switch OTF sources without rebuilding.)


Further uses


As well as changing data files like images, sounds, level files etc, you can use this system to tweak all kinds of settings in your app if the program is made to load them from a file. In many apps there is often a lot of fiddling with values to get a nice feeling game. If you're uploading the entire app onto the device every time, these values might as well be hard-coded into the program itself because it needs to be built every time anyway. But now that you have a quick way to get data files onto your devices, you may like to start loading settings from files instead and vastly improve the turnaround time for tweaking and twiddling those values.

Another thing to keep in mind is that PHP can do so much more than simply spit out the manifest file. When combined with a database it can become a very handy tool for managing settings. For Downhill Supreme 2, we kept tweakable settings in a database table, and used a simple web form to edit them. When values were saved in the database, they would also be written to a text file in the resource folder of the OTF system. This allowed for anyone on the team to tweak a whole bunch of values and then playtest the behavior on a real device in a matter of seconds. When coding the app there are many cases where you know a value will need proper tweaking and testing, so it's nice to be able to add it to this system and know that it can easily be adjusted later.


Drawbacks


For small to medium sized apps, and a server on a local network, there are no drawbacks that I can think of. For larger apps though, the time taken to check for changed files can get fairly long because the md5 hash of every file needs to be calculated. Instead of making the OTFLayer the first scene to be shown every time the app starts, you may want to make it only show when called for, ie. when you know there are actually files that need updating. As a rough example, at the end of Downhill Supreme 2 development the total data size was around 70mb, and it was getting to be a bit annoying to wait for the check every time. Although we got by without changing anything, I think from around 40-50mb or so you would probably want to avoid doing the full check every time.

Another consideration is the distance between you and the server. While developing Downhill Supreme 2, we occasionally used each others OTF sources, and the distance between Japan and Latvia made it a pretty slow process, especially when a large number of files needed updating.

Update: After writing the above, I re-measured the times with my Nexus 7 (2012) and the md5 calculation time was not as bad as I remembered. Our app was 1569 files making for an 80mb .apk:
  • Time to upload app with data files normally over USB: 57s
  • Time to upload app without data files over USB: 3s (.apk is only 3mb without data files)
  • Time for OTF update after fresh install (download every file but no md5 calculation): 2m 40s (local network)
  • Time for OTF incremental update (some files changed, md5 calculated for all existing files): 5s