Dolt 3.3.5A Addon
Thaiel Trask
Here at DoltHub, we're not just making the first SQL database with git-like version control because we think it's cool. We're making the first SQL database with git-like version control because we believe that version control is incredibly useful and should be used for everything. That means part of our job is showcasing how version controlled databases can benefit everything from medical transparency to tackling income inequality.
One industry that we think is a great fit for Dolt is game development: video games often contain a large amount of structured data that's immutable in the final product, but under heavy iteration during development. Dolt is a fantastic tool for allowing multiple collaborators to collaborate on this data efficiently.
If you've been following the news, then you've also seen the recent influx of game developers migrating away from Unity and toward Unreal Engine for some reason. I wanted to understand how Dolt could be used to improve the workflow of developers using Unreal, and figure out how to reduce the cognitive costs of introducing Dolt to a game-dev workflow.
After speaking with some developer friends who use Unreal in their day-to-day and messing around with the Unreal Editor, I quickly realized some things about how Unreal handles both databases and version control:
Unreal has a first-class solution for structured data called Data Tables. Data Tables are a type of Unity Asset, which means they're managed by the runtime, have built-in editor support, and are integrated with Blueprints, Unreal's nocode scripting environment.
Unreal also offers integration with Git, Perforce, and Subversion through an interface. Even more version control options can be supported by implementing this interface, and workflows built on top of this interface can perform version control operations without needing to know which backend is being used.
And yet, every developer that I talked to told me that they didn't use Data Tables in their project. They wanted to but they couldn't, all because of a pernicious detail in the interaction between these two otherwise powerful tools, a critical flaw that made Data Tables entirely unsuited for version control: Like all Unity Assets, Data Tables are binary files, and binary files play poorly with version control.
You might learn this lesson the hard way: imagine you and a co-developer both attempt to modify the same Data Table at the same time. If you were using Perforce or another version control scheme where users "check out" write access from a central server, then you got lucky: your attempt to concurrently modify the Data Table was immediately detected and prevented. But if you were using Git, then it's possible that no one noticed until one of you had already checked in the file. Since Unreal lacks a mechanism to merge DataTables, the only remedy is to revert your changes, pull the most recent revision, and then reapply your changes. Say goodbye to your afternoon.
I've had multiple different developers give me this exact story as a reason why they don't use Data Tables in their Unreal project. It's the achilles heel of Data Tables. More generally, it's the achilles heel of using binary files in traditional source control.
Code isn't the only thing that can be version controlled: you can (and honestly, should) version control just about anything that might change over time. And sure enough, there are people on GitHub using Git for things like book drafts, rules for tabletop games, and software terms of service.
I suspect a big part of it is because version control was originally invented for code, and so people who aren't software engineers just haven't been exposed to it. But I think an equally important reason is that any decently sized project is ultimately going to result in multiple contributors modifying the same file at once. And when simultaneous edits happen to a file, they need to be merged. Merging text is easy. The algorithms and workflows that we use to diff and merge text files have been around for almost fifty years.
Game developers are obviously no stranger to version control, and understand its usefulness. The slow adoption of version control for data isn't because users don't want it, it's because current tools are ill-suited for the job. Version controlling data takes more than just chucking a binary file into Perforce: your tools need to understand the structure of the data. You need workflows for diffing, branching, and merging. Thus, file formats that are more amenable to these operations are more likely to be version controlled.
This isn't the only way to use Dolt. If you want your team to get the most out of Dolt, you probably want Dolt running as a server that each developer can connect to, and we offer both self-hosted and hosting as a service options. But running Dolt locally is a fast and easy way to get started with it immediately.
Before you can use the plugin, you need to configure the plugin via the Edit > Project Settings menu in the Editor. Here, you tell the plugin where your Dolt binary and Dolt repository are located. If you don't know where the dolt binary is installed, running which dolt on Mac/Linux and gcm dolt on Windows Powershell can come in handy here.)
The plugin creates two Dolt branches in the repo, named local and remote. local will contain tables that mirror the Data Tables in the local copy of your project. If you're using version control, remote will contain tables that mirror the most up-to-date copy of each Data Table in your version control system.
The simplest feature offered by the Dolt Plugin is one-click importing and exporting of tables between Unreal and Dolt. Whenever you right click on a Data Table in the Content Browser, there will be a new available action: Scripted Asset Actions > Dolt > Export to Dolt.
Selecting this action will copy the table into Dolt, where you can use the full power of Dolt to query, analyze, and update the table. You can even do all of this from within the Unreal Editor console. For example, if you exported a table containing monsters for your RPG, you might run something like dolt sql -q "select name from local.monsters order by attack desc;". Dolt is 100% compatible with the MySQL dialect1, so any query that you can imagine should work here.
Once you're done modifying the data, you can load it back into Unreal with the Scripted Asset Actions > Dolt > Import from Dolt context action. Note that this loads the new version of the table back into Unreal Editor, but doesn't save it to disk yet. Remember to save your changes!
If one of your Data Table columns is a UStruct, it will get exported to Dolt as its string representation, but future versions of this plugin could be configured to instead export it as a JSON object in order to leverage Dolt's extensive JSON support, or by flattening the struct, giving each field its own table column.
(Caution: some Dolt subcommands are interactive and read from standard input, such as a naked dolt sql command without a -q flag. You should not try to run such commands from within the Unreal console due to a known issue)
Consider the horror story from before. While making a large-scale change to a Data Table, a co-developer makes and pushes their own change to the same table, and now your change is blocked. You know that your changes are compatible because you each touched different table rows. But your VCS doesn't understand table rows; all it sees is that you both made changes to the same binary blob.
Before, you'd be out of luck. But now you can just run Scripted Asset Actions > Dolt > Pull Rebase for the affected table. This action fetches the changes from your VCS, imports both those changes and your changes into Dolt, and then uses Dolt to merge the changes together. If there's no conflicts, it then syncs your workspace to the most recent version of the file and pulls the combined changes back into Unreal. From there you can just submit the changes like normal.
For more detailed information, here's our guide for handling conflicts in Dolt merges. Once all the conflicts are resolved, you can run Scripted Asset Actions > Dolt > Force Sync and Import from Dolt to complete the process.
Finally, if you want to compare the local and remote changes to a Data Table without attempting a rebase, you can use the Scripted Asset Actions > Dolt > Three Way Export action. This imports both the local and remote changes into Dolt, and has them both depend on the same "ancestor" commit. From here you can still merge them if you like.
All of this works regardless of which version control backend you're using! Because the plugin uses interfaces provided by the engine, it doesn't care whether you're using Git, Perforce, or Subversion.
There's even more that the Dolt Plugin could do. Since Unreal Editor allows for invoking delegates when a Data Table row changes, we could make Dolt automatically stay in sync with changes to Data Tables. We could also mirror the entire revision history of every table in your project, making it dead simple to visualize and analyze the complete history of your data.
This plugin is designed as a proof of concept to demonstrate how Dolt can improve developer workflows in Unreal. But it's not battle-hardened, and you probably shouldn't use it with important data yet. There's a lot more that we could do, but we also want to make sure that we're focusing our efforts on the features that will be most useful for people. So if this feels like something that provides value, definitely play around with it!
Dolt, on the other hand, is battle-hardened and production-ready. We actively encourage using Dolt for video game development. If you decide to use Dolt in your workflow, drop us a line on Discord and we'll answer questions and help you get started.
And if you want to see us make a production-ready version of this plugin, or want to contribute to such a tool, then definitely drop us a line! We're always looking to understand more about how people want to use Dolt, so that we can focus our efforts on the most important features.
7fc3f7cf58