Mapping batch mode

Mapping batch mode

The execution of trained and fixed mappings is, as far as possible, processed by matrix operations. For some mappings this may result in intermediate datasets that don't fit in memory. This may especially happen if the mapping is applied to a datafile consisting of large images. A solution is to process the mapping in batches on a series of subsets of the data. So instead of B = A*W PRTools may execute B(L,:) = A(L,:)*W in which L runs over all objects of the input and output datasets or datafiles A and B. This is automatically enabled if the appropriate flag of the mapping W has been set. Two routines control this:

• setbatch enables or disables batch processing and sets the batch size (the number of objects in L).
• getbatch retrieves the flag that controls batch processing and the batch size.

Until PRTools version 4.2.3 batch processing was a generic procedure to be switched on or off for all mappings. From version 4.2.4 it applies to a single mapping. It enables default batch sizes of initially 1000 objects. This can be reset by the prglobal command. PRTools commands like plotc that may profit from batch processing have it internally switched on.

Users may set batch processing for fixed and trainable mappings. During training itself it is automatically disabled, but the batch settings are copied from the untrained to the trained classifier. In sequential combining it holds for all mappings involved. Some incorrect and correct examples:

    data_out = data_in*my_fixed_mapping*setbatch
    % wrong! batch processing is set after mapping is applied

    data_out  = data_in*(my_fixed_mapping*setbatch)
    % batch processing used during mapping

    w = trainset*my_untrained_mapping*setbatch
    % batch processing set for the resulting trained mapping
    testset*w
    % batch processing used during execution of testset

    w = a*(pca([],0.90)*my_untrained_mapping*setbatch)
    % batch processing set for the resulting sequential mapping
    % this is just used when w is used for mapping new data

    w = a*(pca([],0.90)*(my_untrained_mapping*setbatch))
    % same as above

    w = a*((pca([],0.90)*setbatch)*my_untrained_mapping)
    % same as above

In case a mapping results in a scalar like for testc and testd the scalar results of batches of objects L, e = A(L,:)*testc are averaged. This may not always be appropriate. The batch flag for routines like these is for that reason neglected. Users writing their own mappings should do something similar during the execution part of their routine when results for objects are to be combined.

Mapping batch mode