Batch processing

Why do I need this step?

The solution is a loop: a list of participant names, with the same steps applied to each name. This is called batch processing.

But a simple loop has a problem: if the seventh participant fails for some reason (a missing file, an odd channel, memory), everything stops, and in the morning you find that only six finished. That is why we wrap each participant in try and catch: if it fails, the error is written to a file, and the loop moves on to the next participant.

In this lesson we process only three participants from D4 (about 700 MB to download). The same script works on all eighteen if you download them.

How do I do it in EEGLAB?

First: download three participants from D4

1. Create a folder named faces_project with a rawdata folder inside it, and make faces_project the MATLAB working folder (cd). Run every script in station 9 from inside this folder. The file organization lesson explains the other folders.
2. From the ds002718 page on OpenNeuro, download into rawdata the top-level files (dataset_description.json, README and task-FaceRecognition_events.json) and the full sub-002, sub-003 and sub-004 folders. Or, from the system command line, if you have the AWS CLI:

aws s3 sync --no-sign-request s3://openneuro.org/ds002718 rawdata --exclude "sub-*" --include "sub-002/*" --include "sub-003/*" --include "sub-004/*"

Second: try the pipeline on one participant in the GUI

3. Open File › then Import data › then Using EEGLAB functions and plugins › then From BIDS folder structure and choose the rawdata folder. Enable importing events and electrode locations from the BIDS files, and take the event type from the trial_type column. Click Ok. A STUDY is created with the three participants.
4. Pick one participant from the Datasets menu and apply the steps in the GUI as in stations 3 to 5, changing only three settings: mains power in the UK is 50 Hz, not 60; the eye channels are EEG061 and EEG062; and the heart channels EEG063 and EEG064 are removed.
5. Type eegh. These lines are what you will put inside the loop.

Third: turn the pipeline into a loop

6. Copy the single-subject script into a new file batch_d4.m. Move the processing steps into a function at the end of the file named process_subject, which takes the participant file and the settings, saves the results, and returns a summary.
7. At the top of the file: the list of participants, the settings, and a for loop that calls the function inside try.
8. In catch: write the participant name and the error message to logs/batch_log.txt, and record the status failed in the summary table.
9. Run the file. ICA on 72 channels (70 EEG and 2 EOG) and fifty minutes of recording takes time for each participant, so let it run. If you have many participants, see the HPC lesson.
10. When it finishes, open batch_summary.csv. Each row is one participant: status, bad channels, removed components, and remaining epochs.

How do I write it as code?

%% D4 batch pipeline: same steps for every subject, errors logged
eeglab nogui

%% Settings
projectRoot = pwd;
bidsRoot    = fullfile(projectRoot, 'rawdata');
derivRoot   = fullfile(projectRoot, 'derivatives', 'eeglab');
logRoot     = fullfile(projectRoot, 'logs');
subjects    = {'sub-002', 'sub-003', 'sub-004'};

cfg.lineFreq    = 50;                          % UK recording: 50 Hz mains
cfg.eogLabels   = {'EEG061', 'EEG062'};        % HEOG, VEOG
cfg.dropLabels  = {'EEG063', 'EEG064'};        % ECG channels
cfg.epochEvents = {'famous_new', 'unfamiliar_new', 'scrambled_new'};
cfg.epochWindow = [-1 2];
cfg.thresholdUv = 100;

if ~exist(logRoot, 'dir'), mkdir(logRoot); end
logFile = fullfile(logRoot, 'batch_log.txt');

%% Import BIDS data once: events from trial_type, locations from electrodes.tsv
[STUDY, ALLEEG] = pop_importbids(bidsRoot, 'eventtype', 'trial_type', ...
    'bidsevent', 'on', 'bidschanloc', 'on', 'subjects', subjects, ...
    'outputdir', derivRoot, 'studyName', 'faces');
clear STUDY ALLEEG   % each subject is reloaded from disk inside the loop

%% Loop over subjects
nSub    = numel(subjects);
status  = repmat({'not run'}, nSub, 1);
nBad    = nan(nSub, 1);
nIcRem  = nan(nSub, 1);
nEpochs = nan(nSub, 1);

for s = 1:nSub
    subj = subjects{s};
    fprintf('\n===== %s (%d of %d) =====\n', subj, s, nSub);
    try
        inFile = fullfile(derivRoot, subj, 'eeg', sprintf('%s_task-FaceRecognition_eeg.set', subj));
        res = process_subject(inFile, cfg);
        status{s}  = 'ok';
        nBad(s)    = res.nBadChans;
        nIcRem(s)  = res.nIcRemoved;
        nEpochs(s) = res.nEpochs;
    catch err
        status{s} = 'failed';
        fid = fopen(logFile, 'a');
        fprintf(fid, '%s  %s  %s\n', datestr(now, 'yyyy-mm-dd HH:MM'), subj, err.message);
        fclose(fid);
        fprintf(2, 'FAILED %s: %s\n', subj, err.message);
    end
end

%% Summary table, one row per subject
summary = table(subjects(:), status, nBad, nIcRem, nEpochs, ...
    'VariableNames', {'subject', 'status', 'bad_channels', 'ics_removed', 'epochs_kept'});
disp(summary);
writetable(summary, fullfile(derivRoot, 'batch_summary.csv'));

%% ------------------------------------------------------------------
function res = process_subject(inFile, cfg)
% Single-subject pipeline from stations 3 to 5, with project settings in cfg

[inDir, baseName] = fileparts(inFile);
baseName = strrep(baseName, '_eeg', '');
EEG = pop_loadset('filename', [baseName '_eeg.set'], 'filepath', inDir);

% Remove ECG channels, keep EOG for ICA
EEG = pop_select(EEG, 'rmchannel', cfg.dropLabels);
fullLocs = EEG.chanlocs;

% High-pass: passband edge at 0.1 Hz (removes slow drift)
EEG = pop_eegfiltnew(EEG, 'locutoff', 0.1);
% Notch: remove a narrow band around the line frequency (revfilt = 1)
EEG = pop_eegfiltnew(EEG, 'locutoff', cfg.lineFreq - 2, 'hicutoff', cfg.lineFreq + 2, 'revfilt', 1);

% Detection with clean_rawdata channel criteria only:
% no drift removal, no ASR, no window rejection, EOG channels ignored
EEG_test = pop_clean_rawdata(EEG, 'FlatlineCriterion', 5, 'ChannelCriterion', 0.8, ...
    'LineNoiseCriterion', 4, 'Highpass', 'off', 'BurstCriterion', 'off', ...
    'WindowCriterion', 'off', 'BurstRejection', 'off', 'Distance', 'Euclidian', ...
    'channels_ignore', cfg.eogLabels);
badChans = setdiff({EEG.chanlocs.labels}, {EEG_test.chanlocs.labels}, 'stable');
fprintf('Bad channels: %s\n', strjoin(badChans, ', '));
if ~isempty(badChans)
    EEG = pop_select(EEG, 'rmchannel', badChans);
end
if ~isempty(badChans)
    EEG = pop_interp(EEG, fullLocs, 'spherical');
end
clear EEG_test

% Average reference, EOG channels excluded
eogIdx = find(ismember({EEG.chanlocs.labels}, cfg.eogLabels));
EEG = pop_reref(EEG, [], 'exclude', eogIdx);

% ICA copy: same data, high-pass at 1 Hz
EEG_main = EEG;
EEG_ica = pop_eegfiltnew(EEG_main, 'locutoff', 1);
EEG_ica = pop_runica(EEG_ica, 'icatype', 'runica', 'extended', 1);

% Copy the decomposition back to the main dataset
EEG_main.icaweights  = EEG_ica.icaweights;
EEG_main.icasphere   = EEG_ica.icasphere;
EEG_main.icachansind = EEG_ica.icachansind;
EEG_main.icawinv     = [];   % recomputed by eeg_checkset
EEG_main.icaact      = [];
EEG = eeg_checkset(EEG_main);
clear EEG_main EEG_ica

% Flag components above 90% Muscle or Eye, then remove them
EEG = pop_iclabel(EEG, 'default');
thresh = [NaN NaN; 0.9 1; 0.9 1; NaN NaN; NaN NaN; NaN NaN; NaN NaN];
EEG = pop_icflag(EEG, thresh);
artifactComps = find(EEG.reject.gcompreject);
if ~isempty(artifactComps)
    EEG = pop_subcomp(EEG, artifactComps, 0);
end

% Low-pass at 30 Hz, postponed until after ICA cleaning
EEG = pop_eegfiltnew(EEG, 'hicutoff', 30);
EEG.setname = [baseName ' clean'];
EEG = pop_saveset(EEG, 'filename', [baseName '_desc-clean_eeg.set'], 'filepath', inDir);

% Epochs, baseline, rejection of epochs beyond the threshold
EEG = pop_epoch(EEG, cfg.epochEvents, cfg.epochWindow, 'epochinfo', 'yes');
EEG = pop_rmbase(EEG, [-200 0]);
EEG = pop_eegthresh(EEG, 1, 1:EEG.nbchan, -cfg.thresholdUv, cfg.thresholdUv, EEG.xmin, EEG.xmax, 0, 0);
if any(EEG.reject.rejthresh)
    EEG = pop_rejepoch(EEG, EEG.reject.rejthresh, 0);
end
EEG.setname = [baseName ' epochs'];
EEG = pop_saveset(EEG, 'filename', [baseName '_desc-epochs_eeg.set'], 'filepath', inDir);

res.nBadChans  = numel(badChans);
res.nIcRemoved = numel(artifactComps);
res.nEpochs    = EEG.trials;
end

Which value should I choose?

How do I check it worked?

• The table printed at the end has three rows, with status ok in each row.
• logs/batch_log.txt does not exist or is empty. If there is a line, it holds the participant name and the reason.
• In each participant’s folder inside derivatives/eeglab there are two new files: desc-clean and desc-epochs.
• The number of remaining epochs for each participant is close to 450 (150 of each type), minus the rejected ones and any epoch that crosses a boundary event (5 in sub-002), because pop_epoch drops those. If it is much lower for one participant, review it.

What mistakes do people make?

• A loop without try and catch. One error in one participant stops all processing, and the night’s hours are lost.
• A silent catch. Catching an error without logging it is worse than stopping: you think every participant finished. Always log the name and the message, and look at the summary table.
• Settings written inside the function. If you write 60 inside process_subject and forget it, you process UK data with a Saudi filter. Every setting goes in cfg.
• Forgetting the differences between datasets. The eye channels in D1 are named EOG1 and EOG2, and in D4 EEG061 and EEG062. Copying the script without checking these names makes ICA and the reference treat eye channels as scalp channels.
• Trusting the loop without checking. A summary saying ok does not mean the cleaning is good. Open one or two files in the GUI, and look at the number of removed components and remaining epochs for each participant. The odd value in the table is the participant you review first.

How do I write it in my paper?

A template: edit the data, settings and versions to match what you ran.

Data from [n] participants (ds002718; EEG-BIDS plugin 10.5, EEGLAB 2026.1) were high-pass filtered (passband edge 0.1 Hz; firfilt plugin 2.8), and line noise was attenuated with a band-stop filter (passband edges 48 and 52 Hz). Bad channels were identified with clean_rawdata (clean_rawdata plugin 2.11; flatline 5 s, channel correlation 0.8) and interpolated, then data were average-referenced, EOG excluded. Independent component analysis (extended Infomax; Lee et al., 1999) was computed on a 1 Hz high-passed copy, and components labelled Eye or Muscle with probability ≥ 0.9 (ICLabel 1.7; Pion-Tonachini et al., 2019) were removed. After low-pass filtering (passband edge 30 Hz), epochs from −1 to 2 s were baseline-corrected (−200 to 0 ms), and epochs beyond ±100 µV were rejected.

Going further

• EEGLAB Tutorial: https://eeglab.org/tutorials/10_Group_analysis/multiple_subject_proccessing_overview.html (10. Group analysis > a. Batch processing)
• EEGLAB Tutorial: https://eeglab.org/tutorials/11_Scripting/Analyzing_EEG_BIDS_data_in_EEGLAB.html (11. Scripting > f. BIDS pipeline)
• Makoto's preprocessing pipeline: https://eeglab.ucsd.edu/wiki/Makoto%27s_preprocessing_pipeline (Batch processing (§17.4))