Traditional build systems like make and the Visual Studio build system suffer from a reliability problem which can sometimes trip you up and cause continuous integration builds to fail even though it worked locally when you tested building your changes incrementally.
The Symptom
The issue is that the MSVC++ build system doesn’t detect various types of changes to the codebase/system which would cause a rebuild in a ‘perfect’ system. The types of configuration changes which can cause this are:
- Moving or deleting a header file, or any other file which is brought into the compilation process via the #include directive.
- Adding files to directories listed in the #include path
The most common scenario is the first. I don’t know how many times I’ve removed some files (I seem to spend a lot of time removing code), tested the change by compiling, and then happily checked it in. Five minutes later the grin is wiped off my face by a stern monkey-mail (we call our CI machines ‘build monkeys’), proclaiming to all that I, Stefan Boberg, broke the build!
Oh the shame.
The Cause
Moved/deleted header files
So why is it then, that the build triggers if I change one of the headers included by a source file, but not if I remove it? Well, the answer lies in the C++ build model, and how dependency scanning and evaluation is typically done. Most build systems don’t actually do a precise analysis of source files to find dependencies. Since the C++ build model is so flexible you would actually have to run a complete preprocessing pass over the source to cover all possibilities. This would be expensive so instead they typically perform a very conservative and simple scan of the file text to identify all #include statements, yielding a list of all files which could potentially be included by the file. This means that the list of included files could contain files for platforms other than the one we’re building for, files for a different compiler, or whatever. Hence, the list of dependencies might very well include files which simply don’t exist on the local machine. The net result of this is that since the dependency list is so conservative and may include many non-existent files, a missing header file is not considered a rebuild condition.
New header files
This issue doesn’t pop up very often, but it’s interesting nonetheless, since we’re talking about dependencies and build conditions. Consider the following scenario:
PS G:\foo> tree \foo /f
Folder PATH listing for volume Bulky
Volume serial number is 96F4-F152
G:\FOO
│ foo.cpp
├───bar
│ config.h
└───war
PS G:\foo> cat foo.cpp
#include "config.h"
int main(int argc, char* argv[])
{
return 0;
}
PS G:\foo> cl /c /Iwar /Ibar foo.cpp
If you compile this fragment using the above commandline, foo.cpp is going to see the “bar/config.h” header file, as it is the only one in the path. But what happens if someone (or you) subsequently add a config.h file in the ‘war’ directory? Then the situation will look like this:
PS G:\foo> tree \foo /f
Folder PATH listing for volume Bulky
Volume serial number is 96F4-F152
G:\FOO
│ foo.cpp
│ foo.obj
├───bar
│ config.h
└───war
config.h
If you were using the Visual Studio build system, this would not cause a rebuild, even though a compilation of the above tree would yield a different result! (Unless of course, the ‘war/config.h’ file isn’t different from ‘bar/config.h’).
The Remedy
One very simple way to avoid both issues is obviously to always make clean builds when removing files from projects. Then all files will be recompiled and you will get errors if any required file is missing. Unfortunately, if you have the memory of a goldfish or keep getting distracted by other more interesting problems, it’s very easy to forget to do this. And unfortunately, without access to the source code of your build system, this is the only solution which works. Of course, if you have a good idea of the dependencies between libraries in your codebase you can get away with cleaning just a part of the tree and save yourself some build time.
If you have more control over the build process, it’s possible to do a little bit better. For example, instead of analyzing dependencies before compiling, some slightly more sophisticated build systems capture the actual list of included files and save it for later use in dependency evaluation. GCC exposes the -MM option which can be used for this purpose. However, even this will not fix the issue where files are added in the include path. To cover that case, there are two different approaches I can see. One is to actually do the pre-scan of dependencies like Visual C++ does, but also note the results of this operation in some state store. Then, the next time you evaluate dependencies you perform the same scan and trigger a build if the resulting set of #include files differs.
Another less direct approach is the one taken by Cascade by Conifer Systems – it uses a custom IFS (installable file system) implementation which tracks all files the command attempts to open – including the failed opens. Come build time, it checks whether opening those files again would yield a different result, and triggers a build if it would.
Further Reading
[one year later, another blog post -- sorry I guess I tend to prioritizing writing code over blogging... most likely, this won't change in the near future!]
