Running on HPC

Why do I need this step?

A computing cluster (HPC, high-performance computing) is a large group of computers that runs your jobs at the same time. Twenty participants become twenty parallel jobs, finishing in about the time of one participant.

But a cluster shows no windows, and you cannot click anything on it. The script must run from start to end on its own. This lesson prepares your script for that.

When is it worth it? Compute: time for one participant × number of participants. If the result is a few hours, run it on your computer overnight. If it is days, or you rerun often, the cluster is worth it.

How do I do it in EEGLAB?

A cluster runs without a GUI, so the steps here are preparation and submission.

First: prepare the script

1. Start from the batch script and check four conditions:
   • It starts with eeglab nogui, so no window opens.
   • It does not plot, or it saves the plot to a file with print and then closes it.
   • All paths are built from the project folder with fullfile, not paths from your computer such as C:\Users\....
   • It processes one participant, chosen from a number the cluster gives it. This way the cluster sends the same script twenty times with different numbers.
2. Test it on your computer from the system command line (not from MATLAB):

matlab -batch "run_ica_d4"

-batch runs the script without a GUI, then closes MATLAB. If an error occurs, it exits with an error code that the cluster sees.

Second: your university cluster

3. Ask your university computing department: is MATLAB installed? Which job scheduler do they use (SLURM is the most common)? Where should you put EEGLAB?
4. Copy the project folder, the script and EEGLAB to your account on the cluster.
5. Write the job file. This is a SLURM example for three jobs, one per participant. Names and values differ between universities, so adjust them to their instructions:

#!/bin/bash
#SBATCH --job-name=d4-ica
#SBATCH --array=1-3
#SBATCH --cpus-per-task=4
#SBATCH --mem=16G
#SBATCH --time=04:00:00
#SBATCH --output=logs/ica_%a.out

module load matlab
matlab -batch "run_ica_d4"

6. Submit it with sbatch job_ica.sh, and follow it with squeue -u and your user name.

Third: the NSG portal (a free alternative)

7. If your university has no cluster, the Neuroscience Gateway (NSG) portal is free for academic researchers, with MATLAB and EEGLAB ready. Register an account on its website.
8. Install the nsgportal plugin from File › then Manage EEGLAB extensions (the latest release when this lesson was written is 2.1.1).
9. Open Tools › then NSG Tools › then NSG account info and enter your user name and password.
10. Put the script and the data it needs in one folder, then open Tools › then NSG Tools › then Manage NSG jobs, choose the folder, type the script name, and submit the job. From the same window you follow its status and download the results.

How do I write it as code?

The script reads the files that pop_importbids wrote to derivatives/eeglab in the batch processing lesson, so run that import once first. Run it from inside faces_project.

% run_ica_d4.m - one subject per job, no display
% Local test:  matlab -batch "run_ica_d4"
% On SLURM:    the array index selects the subject
eeglab nogui

projectRoot = pwd;
outRoot     = fullfile(projectRoot, 'derivatives', 'eeglab');
logRoot     = fullfile(projectRoot, 'logs');
subjects    = {'sub-002', 'sub-003', 'sub-004'};

% Subject index from the cluster, or 1 when run locally
taskId = str2double(getenv('SLURM_ARRAY_TASK_ID'));
if isnan(taskId), taskId = 1; end
subj = subjects{taskId};
fprintf('Job %d: %s\n', taskId, subj);

cfg.lineFreq  = 50;                     % same settings as the batch lesson
cfg.eogLabels = {'EEG061', 'EEG062'};
cfg.dropLabels = {'EEG063', 'EEG064'};

try
    % File written by pop_importbids in the batch processing lesson
    outDir = fullfile(outRoot, subj, 'eeg');
    EEG = pop_loadset('filename', sprintf('%s_task-FaceRecognition_eeg.set', subj), ...
        'filepath', outDir);

    % Same steps as process_subject up to ICA, so the weights fit its main dataset
    EEG = pop_select(EEG, 'rmchannel', cfg.dropLabels);
    fullLocs = EEG.chanlocs;
    EEG = pop_eegfiltnew(EEG, 'locutoff', 0.1);
    EEG = pop_eegfiltnew(EEG, 'locutoff', cfg.lineFreq - 2, 'hicutoff', cfg.lineFreq + 2, 'revfilt', 1);
    EEG_test = pop_clean_rawdata(EEG, 'FlatlineCriterion', 5, 'ChannelCriterion', 0.8, ...
        'LineNoiseCriterion', 4, 'Highpass', 'off', 'BurstCriterion', 'off', ...
        'WindowCriterion', 'off', 'BurstRejection', 'off', 'Distance', 'Euclidian', ...
        'channels_ignore', cfg.eogLabels);
    badChans = setdiff({EEG.chanlocs.labels}, {EEG_test.chanlocs.labels}, 'stable');
    if ~isempty(badChans)
        EEG = pop_select(EEG, 'rmchannel', badChans);
    end
    if ~isempty(badChans)
        EEG = pop_interp(EEG, fullLocs, 'spherical');
    end
    clear EEG_test
    eogIdx = find(ismember({EEG.chanlocs.labels}, cfg.eogLabels));
    EEG = pop_reref(EEG, [], 'exclude', eogIdx);

    % ICA on a 1-Hz high-passed copy (the slow step worth a cluster)
    EEG_ica = pop_eegfiltnew(EEG, 'locutoff', 1);
    tic;
    EEG_ica = pop_runica(EEG_ica, 'icatype', 'runica', 'extended', 1);
    fprintf('ICA finished in %.1f min\n', toc / 60);

    % Save only the small result: the ICA weights and the bad channel list
    icaInfo = struct('icaweights', EEG_ica.icaweights, 'icasphere', EEG_ica.icasphere, ...
                     'icachansind', EEG_ica.icachansind, 'badChans', {badChans});
    save(fullfile(outDir, sprintf('%s_task-FaceRecognition_desc-ica.mat', subj)), ...
         '-struct', 'icaInfo');
catch err
    % Write the error to a log file, then exit with an error code
    if ~exist(logRoot, 'dir'), mkdir(logRoot); end
    fid = fopen(fullfile(logRoot, sprintf('%s_error.txt', subj)), 'w');
    fprintf(fid, '%s\n', getReport(err, 'extended', 'hyperlinks', 'off'));
    fclose(fid);
    rethrow(err);
end

Which value should I choose?

How do I check it worked?

• matlab -batch "run_ica_d4" on your computer prints Job 1: sub-002, then the ICA duration, and closes MATLAB without error.
• derivatives/eeglab/sub-002/eeg holds the file sub-002_task-FaceRecognition_desc-ica.mat.
• On the cluster: three jobs, three weights files, and no error file in logs.
• size(icaweights) in each file is the data rank × 72, that is 71 × 72 if no channel was interpolated (one rank is lost to the average reference).

What mistakes do people make?

• A command that opens a window. Any pop_ without enough arguments, or eeglab instead of eeglab nogui, tries to open a window, so the script stops or fails. Test with matlab -batch on your computer before submitting.
• Paths from your computer. C:\Users\... does not exist on the cluster. Build every path from the project folder.
• A loop over all participants in a single job. You lose the parallelism, and an error in one participant stops everything if you do not catch it.
• Asking for less time or memory than needed. The cluster stops the job with no result. Read the logs file and the job report before resubmitting.
• Forgetting versions. The EEGLAB or MATLAB version on the cluster may differ from your computer. Print eeg_getversion and version at the start of the script, and write them in your Methods.

How do I write it in my paper?

A template: edit the data, settings and versions to match what you ran.

Independent component analysis (extended Infomax; Lee et al., 1999) was computed with runica (EEGLAB 2026.1) for each participant on a copy of the preprocessed data high-pass filtered at 1 Hz, and the resulting weights were applied to the 0.1 Hz high-pass filtered data; jobs ran one per participant on [computing cluster] with MATLAB [version].

Going further

• EEGLAB and HPC: https://eeglab.org/others/EEGLAB_and_high_performance_computing.html (EEGLAB and the Neuroscience Gateway)
• nsgportal (GitHub wiki): https://github.com/sccn/nsgportal/wiki (Setting up the plug-in; Creating and managing an NSG job using pop_nsg from the command line)
• EEGLAB Tutorial: https://eeglab.org/tutorials/11_Scripting/ (11. Scripting)
• EEGLAB on high performance computing ressources: https://www.youtube.com/watch?v=1_h8NFS_ipI (Full video)