Fed up with watching Compiling Shaders messages in UE4-Editor? We’ve Open Sourced an Unreal Engine plugin that can help you automate filling your local DDC overnight so that it’s all done for you in the morning when you get back to your desk.
Working outside of the studio means that we don't have the same access to the infrastructure that we normally use to make day-to-day development faster. Most things can be worked around using a VPN, but some tasks are just too slow to do over this connection.
A great example of this is the system to speed up loading the editor for the first time after a build/sync. In the studio, in the background UE4 would copy some data (quickly) from the network. That’s too slow over the VPN and instead of sharing one set of results, they have to be calculated on each workstation. The upshot is that everyone gets to watch the 'Shaders Compiling' pop-up slowly count down to zero (it's worth pointing out that while the Shaders are compiling, there's actually a whole bunch of other stuff going on at the same time too - the shaders just put up a message about it). This is especially true for Play-In-Editor (PIE) where it feels like you only want a few assets, but they take what feels like forever to be available.
While we can't change the dependency on this data, if we could automate producing it then we could schedule it to happen when we're not using the computer, for example overnight.
So, we've written a UE4 (game) plugin called "Fill DDC For PIE", and we've made the source available on Github with an MIT license.
Each evening when we finish for the day, we run a script that:
When we come back in the morning, pretty much all the assets we need are ready to go and we don't have to 'wait for shaders'.
Unreal Engine's Asset System makes use of a file cache to load the bulk data for certain asset types. It stores this in the Derived Data Cache (DDC). If the editor cannot find cached data, or the cached data is out of date, it must generate it as the assets are loaded. This can take a huge amount of time and processing power. It is a very common (even usual) occurrence after syncing to the latest version in source control.
There's more information about how it all works on the Unreal Engine website.
This plugin makes use of the Automation Framework. (Further information on this can be found on the Unreal Engine website)
This performs the equivalent of you loading a level and pressing play. Because we are scheduling this to happen at a time the computer isn't being used, we get to do this 'for free'.
The plugin contains a single automated test FillDDCForPIETest. It allows the user to predefine a list of maps they wish to cache data for, then for each map it will:
1. Open the map in Editor
2. Run Play In Editor (PIE)
3. Run extra steps (These are defined in the configuration. See below.)
4. Close Play-In-Editor
The source code for the plugin, along with further information, can be found on our GitHub page.
To Setup, you just add the maps you need to your UserEngine.ini or DefaultEngine.ini if you prefer like so:
[/Script/AutomateDDC.FillDDCForPIESettings]
Maps=/Game/Levels/SampleMap1
+Maps=/Game/Levels/SampleMap2
Create a batch file that runs something like this (put this at the end of your normal Sync script, or as a post sync/build step in Unreal Game Sync):
UE4Editor.exe "T:/Path/To/Your/Project/Game.uproject" -execcmds="Automation RunTests FillDDCForPIETest" -unattended
It is also possible to add additional steps to the automation. Each 'ExtraStep' executes a console command and then waits either for a time out, or for a specific message to appear in the log.
For example, once play in editor has opened a map, you may want to also activate the Pause Menu to cause all those assets to load and be pre-cached.
To do so, you could implement a console command to "OpenPauseMenu". This would open the game's menu and once completed, write 'Opened pause menu' to the log. Make a similar command to "ClosePauseMenu". To use these as part of the automation task, you would add the following to the config file:
[/Script/AutomateDDC.FillDDCForPIESettings]
Maps="/Game/Levels/SampleMap1"
+Maps="/Game/Levels/SampleMap2"
ExtraSteps=(StepId="Open Pause Menu",ExecCmd="openpausemenu",CompleteCondition=(LogStringForAdvance="Opened pause menu",TimeoutSeconds=600.0), bWaitForSettleAfterStep=True)
+ExtraSteps=(StepId="Close Pause Menu",ExecCmd="closepausemenu",CompleteCondition=(LogStringForAdvance="Closed pause menu",TimeoutSeconds=20.0), bWaitForSettleAfterStep=True)
If you are co-located with other people working on one project, you should set up a network DDC folder. If you have build machines on the same network, you really should be doing this.
When setup correctly, the team can take advantage of a nightly process to populate the Shared DDC, so that in the morning the bulk of the Derived Data needed for their editor is already available on the network.
As each user runs the editor the derived data is copied from the network share to their local cache. Load times are mainly be limited by the speed it can copy across the network.
If you do have a shared DDC accessible, you can still use the FillDDCForPIE plugin to automate loading specific maps and priming the DDC for everyone.
This is a UE4 commandlet that iterates over all uassets, or a specified subset, and loads the assets. This causes derived data to be generated.
Depending on the scale of your project this could be a very large number of assets and may not finish in a sensible amount of time.
The other drawback is that it does not cache the slate data which is specific to running the editor.
Like the above commandlet this iterates over a large portion of the game content and loads the assets to cook. This causes derived data to be generated.
Again, this can be a very slow task and it too does not cache the slate data which is specific to running the editor. By its nature, it cooks maps and content that are intended for the game, whereas FillDDCForPIE can also cache data for WIP and Developer levels.
This will cover a significant portion of your data. The key issue is that it is only applicable to your editor start map.
If you use this your editor start times will be quicker but when you change map or run PIE, you'll encounter the more missing DDC woes.
Use this plugin if you want to speed up opening UE4 editor for the first time after a sync/update and can leave some automation running at a time you aren’t using the computer for anything else.
You can find the plugin on our Github site.
Hope you find it useful. Let us know if you're using it on your project, we'd love to hear about it.
Terry is a Senior Engineer at Studio Gobo. He has eleven years’ experience in the games industry, ten of which have been working heavily with Unreal Engine.