Writing scripts for automating manual tasks is something that I have been doing on almost every project I was on.
Life if Feudal
The first time I realized that I needed a better tool than just a bunch of python files was when I was working on Life is Feudal MMO. Since the game is quite complex, running the game locally was a multi-step process: you would need to build a server dispatcher, dedicated server, and client, you would need to make sure that your local DB is running, then start the server dispatcher and the client.
Having build/deployment of the project to be a multi-step process means that you have several points in time in time when you can switch context.
Building something in C++ usually means minutes to tens of minutes. You can imagine that as a developer I wouldn’t stare into the screen for 20+ minutes waiting for my script to finish. I would go to Slack, try to work on something that wouldn’t affect the compilation, or read some articles in the meantime. That means that when one of the steps of the script is finished, I need to detect that and start the next step.
That can easily add tens of minutes more to the already long process of preparing the local build.
What I did back then, is I wrote an interactive Python script, which would prompt me with actions that I may want to do, and then would execute everything needed to build and run the game in order:
- Do you want to pull the latest from git?
- Do you want to reset the DB?
- Do you want to open the IDE when done building?
That was quite good, I also made it work with git worktrees, so I could work on another feature branch of the game while all this was being built and run, and wouldn’t need to context switch every once in a while. I also believe I shared this script with QA, who were doing these steps multiple times per day.
Ubisoft
On the projects I worked on in Ubisoft, we had a lot of great tools, but one that I fell in love with was the tool for automating running scripts, it was a tool you start when you boot your PC in the morning, you would choose a preset to prepare everything you need to start working, you would go and drink some tea, and when you return you would have everything updated to the latest version, the project built and your IDE open.
Ok, I wouldn’t only drink tea for 20-40 minutes every morning, but you got the idea, it was seamless, and it was beautiful.
Of course, I could have built another Python script to do the same, but compared to my hard-coded Python script, this tool was a shared tool, with shared scripts, you could modify or add the scripts, and share them with your team. You could also see the progress of the execution, which script failed, and many more neat project-specific things.
I was taking advantage of the ability to add new scripts a lot, and I automated every part of my workflow that I could. As the craziest example, since we were working in p4, and I was a git user in the past, I wanted to work with smaller commits, being able to shuffle them around in time, discard, reapply, and split changes in the same files. For that, I’ve added a local git repo, and every time I update in my p4, it creates two commits in my git repo: one before the update, and one after, so my repos were always in sync and I could switch to using git at any time. The only thing that I didn’t figure out back then was how to notify me about finishing a build if I’m not at my desk.
Mojang
During my time in Mojang, I started writing Python scripts again, I would have scripts to update from git, update submodules, build, run tests, and open IDE, and then I would have scripts like:
updsubmgenbuildsln.pygenbuildrunsln.pybuildtest.py
I wouldn’t type this all out, but rather it would be something like u<Tab>su<Tab>ge<Tab>bu<Tab>… which was rather quick.
However, then I started switching to Clion and realized that now I needed to duplicate all my scripts, which at that time were about 30 unique combinations.
scripter is created
That same weekend I created the scripter repository and started writing a tool that would allow me to configure the steps more easily, I also wanted something more visual to see what was happening at each exact moment. It was inspired a lot by that tool from Ubisoft when it comes to the part of scheduling scripts, but I decided that I want the tool to be ideologically different: instead of being a central tool for building the project, it is going to be just a helper tool that you run only/if you need it, and you could integrate it into the project at any stage, without rewriting your existing scripts.
I’ve got the first version working pretty quickly and then it was a few months of iterating on it before I was fully satisfied with the functionality.
Sharing scripts
One thing I really wanted to add is the ability to share scripts with others without sending them over Slack but also having the ability to change the scripts locally. In Mojang, I stored scripter and the scripts for it in a separate git repo. The scripter has two configurations: local and shared. Local is ignored in git and represents the changes made locally (could add, hide, rearrange scripts), the shared config represents the scripts shared with everyone. One can move the scripts from being local to being shared.
Presets
It is something that others were mentioning when looking at the scripter, that the tool needed some way to save some combinations of scripts to run them faster. I didn’t feel that it was such a useful thing back then, but after implementing it, I realized that it indeed saves a lot of clicks.
Passing environment variables to scripts
This one is a weird one, I wanted to integrate scripter into another tool that we had (which coincidentally started on the same weekend as scripter), that could allow you to set build configurations and flags for building the project. I would start scripter from that tool, but then I would need to somehow pass those flags and arguments to the build scripts.
I did not want scripter to force writing scripter-specific scripts, those should still be scripts that you can run in the console or via other tools if you want so. If I dropped that requirement, I could make it work only with Python scripts of a specific format that could accept arguments and more. I couldn’t pass arguments to scripts since those were meant to configure how the scripts run by the user (or, again, via other tools or when running from the console). What I decided to do instead is to allow to set environment variables to the scripts.
The idea was that when starting scripter I would provide an -env argument with the name and value of the environment variable, and that environment variable would be set to every script. Then the scripts that care about specific parameters would read those environment variables. This required me to write some amount of boilerplate code in my scripts, but in the end, I felt that it was worth it.
I also added the ability to set a custom title, so the tool that started scripter could provide some human-readable information about the context the scripter was running in.
Conclusion
Writing scripts and automating processes is fun, and it should be fun. A lot of people write their own scripts and tools like scripter (we had 3 different tools like scripter at some point in Mojang, at least the ones that I heard of). This makes their life easier, and hopefully, in the end, makes everyone else life easier once those tools are shared.
I started using scripter in my smaller hobby projects as well, and now working at Embark I find myself actively using it again for different tasks that do daily, always finding ways to improve it. I feel that I may be useful for other devs if not directly, but maybe as an inspiration for their own tools.