4. Map-Reduce with QUACreduce¶

4.3.1. Create sample input¶

$ echo -e 'foo bar baz\nfoo foo' > /tmp/foo1.txt
$ echo -e 'bar' > /tmp/foo2.txt
$ cat /tmp/foo*.txt
foo bar baz
foo foo
bar

4.3.2. Define the map operator¶

This converts standard input into a sequence of key/value pairs, one per line.

We will use tr for this:

$ cat /tmp/foo*.txt | tr '[:blank:]' '\n'
foo
bar
baz
foo
foo
bar

(Note that in the standard map-reduce word count examples, the mapper emits the value 1 for each word. QUACreduce is perfectly happy with null values, and counting the length of a set is the same as summing a set of 1’s of the same size, so we do the former.)

4.3.3. Define the reduce operator¶

This converts a sequence of key/value pairs from the mapper, presented on standard input, into arbitrary output (typically one output item per set of identical keys). All input pairs with the same key are adjacent in the input, but there are otherwise no input ordering guarantees.

We will use uniq to print each input word with the number of times it occurred:

$ echo -e 'b\nb\na\nc\nc\nc' | uniq -c
2 b
1 a
3 c

4.3.4. Test the operators together¶

$ cat /tmp/foo*.txt | tr '[:blank:]' '\n' | sort -sk1 -t '    ' | uniq -c
2 bar
1 baz
3 foo

Congratulations, you’ve just run map-reduce in serial mode, with one mapper and one reducer! The next step is to run lots of mappers and reducers in parallel, which is one thing QUACreduce helps with.

4.3.5. Prepare the job¶

The quacreduce command is used to prepare a makefile as well as a SLURM job file:

$ quacreduce --map 'tr "[:blank:]" "\n"' \
             --reduce 'uniq -c > out/%(RID)' \
             --partitions 2 \
             /tmp/mrjob /tmp/foo*.txt

What’s going on here?

• --map defines the map operator. This can be any shell pipeline; watch quoting carefully! The CWD is the job directory.
• --reduce defines the reduce operator. The variable %(RID) is the reducer ID; this is important for keeping output from different reducers separate. It is substituted by QUACreduce during job construction.
• --partitions defines the number of partitions. There is one reducer per partition, so this limits the available parallelism for the reduce step (as well as downstream map-reduce jobs unless you take other measures). The limiting factor to keep in mind is that if you have \(n\) input files and \(p\) partitions, you will need \(n \times p\) temporary files, which can grow quickly.
• /tmp/mrjob is a directory in which to build the job.
• /tmp/foo*.txt are the input files. There should be lots of these, as only one mapper is run per input file. They can live anywhere but must have unique filenames even if they are in multiple directories.

4.3.6. Run the job with make¶

This approach is simpler but is limited to the parallelism available in a single machine. If you need more, you can use a SLURM cluster (see the next step). For example:

$ cd /tmp/mrjob
$ ls -R
.:
Makefile  slurm_job  out  tmp

./out:

./tmp:

QUACreduce has created two files and two directories:

• Makefile is what you expect; it defines the dependency graph among the temporary and job management files.

  Note: Output files created by your reduce operator are not included in the dependency graph. Therefore, Make has no idea if they are complete or not, so it’s your responsibility to make sure they’re not corrupted on re-runs (which may include new data). It’s best practice to simply overwrite these each time the reducer is run.

• slurm_job is a SLURM batch file to run the Make job on multiple nodes.

• tmp is a directory containing various files used to contain intermediate results and manage job progress. make clean deletes everything in this directory.

• out is a convenience directory for your use. You don’t have to put your output here, but you ought to have a good reason not to. make reallyclean deletes everything here as well as in tmp.

You are now ready to run the job:

$ make -j2
[...FIXME...]
$ ls -R
.:
Makefile  out  slurm_job  tmp

./out:
0  1

./tmp:
0.reduced  foo1.txt.0  foo1.txt.mapped  foo2.txt.1
1.reduced  foo1.txt.1  foo2.txt.0       foo2.txt.mapped

Note that the subdirectories are now populated.

Your output is available with:

$ cat out/*
2 bar
1 baz
3 foo

Note that the output order has changed. In general, you must sort yourself if you care about this order.

4.3.7. Add more input data¶

One of the neat things that QUACreduce can do is add additional data and then only re-run the parts of the job that are affected. For example:

$ echo 'qux' > /tmp/foo3.txt
$ cd /tmp/mrjob
$ quacreduce --update . /tmp/foo*.txt
$ make -j2
[...FIXME...]
$ cat out/*
2 bar
1 baz
3 foo
1 qux

Note that only foo3.txt was mapped, because we already had mapper results for foo1.txt and foo2.txt.

4.3.8. What’s next?¶

For further help, say quacreduce --help.

4.4.1. Example¶

FIXME

$ sbatch -N2 slurm_job -j4

Note that the number of nodes requested from SLURM and -j, which is the total number of tasks make will run simultaneously, must be coordinated for good performance. The above might be appropriate for a cluster with two cores per node. Memory could be a limitation also, along with myriad others.