How to save compilation progress using Bazel

Question

I am building tensorflow from source using bazel, within a docker container, and it seems that the progress of the compilation is stored in cache temporarily. This means that if I make a change and recompile, only the modules that are dependent to the change will be rebuilt.

Is it possible to store the compilation progress in a more permanent location, so that the progress gets saved?

I tried using the --disk_cache flag in order to supply a local directory as a cache location, but when I recompiled, everything was compiled from scratch.

Answer 1

--disk_cache is the right way to set a cache. It sounds like the code and environment you're using doesn't get cache hits where you're expecting. There's some documentation on debugging cache hits with local execution (--disk_cache uses the same infrastructure as a truly remote cache, so most of that still applies).

You should start with Ensure re-running the same build/test command produces cache hits. The basic procedure is to run two builds with --execution_log_binary_file=/tmp/exec1.log (use different files each time) and then run a tool from the bazel repository:

bazel build src/tools/execlog:parser
bazel-bin/src/tools/execlog/parser \
  --log_path=/tmp/exec1.log \
  --log_path=/tmp/exec2.log \
  --output_path=/tmp/exec1.log.txt \
  --output_path=/tmp/exec2.log.txt

Then you can diff the resulting text files to see how the cache keys differ, which is why Bazel isn't resusing the cached results.

I recommend starting with a build of a small subset of the code you're building. Most reasons for cache misses will affect everything, and it makes iteration much faster.