13.1 Learning Task for Classification and Regression

The first example will command MOA to train the HoeffdingTree classifier and create a model. The moa.DoTask class is the main class for running tasks on the command line. It will accept the name of a task followed by any appropriate parameters. The first task we will use is the LearnModel task. The -l parameter specifies the learner, in this case the HoeffdingTree class. The -s parameter specifies the data stream to learn from, in this case generators.WaveformGenerator , generating a three-class learning problem. The -m option specifies the maximum number of examples to train the learner with, in this case 1,000,000 examples. The -O option specifies a file to output the resulting model to:

This will create a file named model1.moa containing the tree model induced during training.

13.2 Evaluation Tasks for Classification and Regression

The next example will evaluate the model to compute its accuracy on a set of examples generated using a different random seed. The EvaluateModel task is given the parameters needed to load the model produced in the previous step, generate a new waveform stream with random seed 2, and test on another 1,000,000 examples:

Observe the use of nested parameters using parentheses. Quotes have been added around the description of the task, otherwise the operating system might be confused by the parentheses.

After evaluation the following statistics are output:

Note that the two steps above can be rolled into one, avoiding the need to create an external file, as follows:

MOA will create and use the classifier in memory, without storing it in a file.

13.3 Learning and Evaluation Tasks for Classification and Regression

The EvaluatePeriodicHeldOutTest task will train a model while taking snapshots of performance using a holdout test set at periodic intervals. The following command creates a CSV file, trains the HoeffdingTree classifier on the WaveformGenerator data, keeps the first 100,000 examples for testing, trains on a total of 100,000,000 examples, and tests on the holdout set every 1,000,000 examples:

For the purposes of comparison, a bagging learner using 10 decision trees can be trained on the same problem:

Another evaluation method implemented in MOA is the interleaved test-then-train. It produces smoother plots of accuracy over time, as each individual example becomes less and less significant to the overall average. Here is an example of an EvaluateInterleavedTestThenTrain task that creates a CSV file, trains the HoeffdingTree classifier on WaveformGenerator data, trains and tests on a total of 100,000,000 examples, and records accuracy every 1,000,000 examples:

13.4 Comparing Two Classifiers

Suppose we want to compare the learning curves of two classifiers, a decision stump and a Hoeffding Tree. First, we have to execute the EvaluatePeriodicHeldOutTest task to train a model while taking snapshots of performance with a holdout test set at periodic intervals. The following commands create CSV files and train the DecisionStump and the HoeffdingTree classifiers on WaveformGenerator data, using 1,000 examples for holdout testing, training on a total of 100,000 examples, and testing every 10,000 examples:

Assuming that gnuplot is installed on the system, the learning curves can be plotted with the following commands:

This results in the graph shown in figure 13.1 .

For this problem it is obvious that a full tree can achieve higher accuracy than a single stump. The stump has an almost constant accuracy (around 58%) that does not improve with further training, while that of the full tree increases by 10% on the first 100,000 examples.