How to keep code and output organized?

Question

I am having trouble keeping organized the files for the computational component of my research project. In brief, I have written computer code which I use to run computational experiments. For each experiment, I get a set of output files, from which I generate output such as plots and tables.

The trouble stems from the fact that I have multiple experiments (say exp1, exp2 and exp3) and each experiment has multiple output files (say a.txt, b.txt, c.txt). This is further complicated by the fact that I have multiple versions of each experiment (say exp1 2014-01-02, exp1 2014-05-06, etc).

How should I organize the code and the output in a systematic way? The system should satisfy the following key requirements:

• It has to be easy to rerun version X of experiment Y. I occasionally have to rerun earlier versions of the experiments in order to check the results or to make slight modifications.
• It has to record the output for each version of each experiment. I often have to run many slight variants of a single experiment to tweak some small aspect of the results, so it is essential to record the output for each run.

Answer 1

Funny you should ask, because I'm currently answering this question as I take breather from a project involving just such a family of experiments and variations.

In my own self-organization, I typically distinguish two types of such variants. If some variants are "dead" and would only be referred to on rare occasions for historical purposes, then they get checked into version control and deleted from my working set---they can be exhumed by the powers of SVN/git/Mercurial when needed.

For the "live" versions, some experiments get grouped thematically and some chronologically.

• Thematic is for when I'm it's a purely computer-based experiment (which can be re-run arbitrarily), e.g., "overlay-network", "random-network", "unit-disc-network"
• Chronologically is for when my computer-based runs are based on real data from a physical system that can't be regenerated, but only replicated, e.g., "2013-05-09 Alphavirus", "2013-06-09 Alphavirus run #3", "2013-07-02 Repeats of Failed Samples"

I maintain a strong distinction between several types of files, which must never mix:

• Core code: there is precisely one version of any core code system, maintained by version control. If I need to maintain variants, they have to be set by option flags, not by forking the codebase (that way lies madness).
• Each thematic/dated directory gets a README file, whatever notes are neecessary and (typically) two subdirectories: experiments and analysis
• Experiment scripts for a thematic cluster live in the experiments directory
• Each batch of experimental data lives in its own subdirectory of the experiments directory where its script lives.
• Analysis contains scripts to process the experimental data. Often there are two layers: one to process raw data into results, and another to plot the results. This is because extracting results is often time intensive and figures are frequently tweaked. If there are a lot of results files, they get their own subdirectory too.
• Plots, living in a subdirectory of the analytical scripts directory

I also typically maintain a few master scripts which allow me to re-run large swaths of experiment / analysis when the core code is improved or a bug is found, which happens more frequently than one would like.

Answer 2

Every time I run a script, I pass it two arguments:

1. A file label that gives a name to my experiment
2. A short blurb of text that automatically gets put into a README file. (This precisely documents the conditions of the experiment so I remember what I did.)

At the start of my script, I create a folder inside my "output" directory for the results of my experiment (and the README file) and I name the folder $timestamp_$file_label. All output gets generated inside that folder, and every time I want to change/rerun the script I just change the file label and the new output gets sent to a different folder. (Or I can use the same file label; it doesn't matter since the experiments will have different timestamps).

I also use git for version control so I can easily go back to old versions of my code or take the code in a substantially different direction without erasing what I did earlier.

Answer 3

I have a substantially similar situation, although most of the time it's not about my own code so much as input files for a third-party model. My approach is as follows:

• input files go in a revision control system (I use git). Each time I run the model, that version of the input files is given a tag with a run number (eg "Run_37").

• each time the model is run, output files go in a new folder that is named for the run number.

• I also keep a spreadsheet linking the two with some other information such as the date, a brief description of what I was testing, and so forth.

Where matlab scripts or similar are used to programmatically generate input files, those scripts also get tagged in a similar way so that it is always clear which version of the script was used for each model run.

Answer 4

While there is no right answer: a nested system of folders: the first one is called Experiments, with subfolders called exp1, exp2 and so on. Each expX contains folders for the versions, and the version folders contains (if necessary) folders for code, input, raw data, graphs, and whatever else you can think of. That's how I would do it at least.