Organizing files & storage

Why do I need this step?

And storage grows faster than you expect. Every continuous copy you save of a participant takes about the same size as the raw file. In D4 the raw file of each participant is about 234 MB, and there are 18 participants, so about 4.2 GB for the raw data alone. If you save five stages per participant, you reach about 21 GB before any epoch or figure.

This lesson gives you three rules before you start any project: a fixed folder structure, fixed file names, and a storage estimate with a decision in advance about what you save.

How do I do it in EEGLAB?

First: the structure

1. Create a project folder with only five folders in it:
   • rawdata: the raw data as you downloaded them. Never write anything here.
   • derivatives/eeglab: everything processing produces, with one folder per participant.
   • code: the scripts. They are what reproduces everything else.
   • figures: the final figures.
   • logs: the error files written by the batch script from the previous lesson.

This is the same logic as BIDS: raw data in one place, derivatives in another, with the same participant names.

Second: the names

2. Name every file from the same parts in the same order: participant, then task, then stage description, then type. Example: sub-002_task-FaceRecognition_desc-clean_eeg.set. The desc- part says which stage this is, and the rest is copied from the raw file.
3. Make the dataset name inside EEGLAB say the same thing. When saving with File › then Save current dataset as, type the full name, and change the Dataset name from Edit › then Dataset info when needed.

Third: storage

4. Open the rawdata folder in your file explorer and look at the size of one participant’s .set file. Multiply it by the number of participants, then by the number of continuous stages you plan to save.
5. Decide what to save before you run. A practical rule:
   • Save what is slow or depends on your judgment: ICA weights, the list of bad channels, and the numbers of the components you removed.
   • Recompute what is fast and automatic: filters, reference, epoching, baseline.
   • Always save the final version that goes into statistics.

Fourth: convert your data to BIDS

6. If your raw data are not in BIDS format, the EEG-BIDS plugin bundled with EEGLAB converts them. Open File › then BIDS tools › then BIDS export wizard (from raw EEG to BIDS) and follow the steps: files, then task, then participants, then events.
7. If you already have a STUDY, use File › then BIDS tools › then Export STUDY to BIDS folder.
8. Check the result with File › then BIDS tools › then Validate BIDS dataset. It opens the official validator page in your browser.

D4 itself is an example of a ready BIDS dataset: open its folder and look at dataset_description.json, README, task-FaceRecognition_events.json, and the sub-002 folders and after.

To share one dataset with someone who uses another program, File › then Export › then Data to EDF/BDF/GDF file writes it in a standard format that most EEG programs read (it needs the Biosig plugin from the install lesson). Use it for sharing only; inside your project, keep the .set files and the BIDS folder.

How do I write it as code?

[ALLEEG, EEG, CURRENTSET, ALLCOM] = eeglab;
eeglabRoot = fileparts(which('eeglab'));

% Project layout: raw data stay untouched, results go to derivatives
projectRoot = pwd;   % run from inside faces_project
bidsRoot    = fullfile(projectRoot, 'rawdata');            % ds002718 lives here
derivRoot   = fullfile(projectRoot, 'derivatives', 'eeglab');
folders = {bidsRoot, derivRoot, fullfile(projectRoot, 'code'), ...
           fullfile(projectRoot, 'figures'), fullfile(projectRoot, 'logs')};
for k = 1:numel(folders)
    if ~exist(folders{k}, 'dir'), mkdir(folders{k}); end
end

% Storage estimate from the raw files on disk
rawFiles  = dir(fullfile(bidsRoot, 'sub-*', 'eeg', '*_eeg.set'));
rawGB     = sum([rawFiles.bytes]) / 1e9;
nSubjects = numel(rawFiles);
fprintf('%d subject(s) on disk, %.2f GB of raw data\n', nSubjects, rawGB);

% Each continuous stage you save costs about one raw copy per subject
nSavedStages = 2;   % clean continuous + epochs
fprintf('Derivatives for these subjects: up to about %.1f GB\n', rawGB * nSavedStages);
fprintf('Full dataset (18 subjects): about %.1f GB raw\n', 18 * rawGB / max(nSubjects, 1));

% BIDS-style derivative name for one subject and one stage
subj   = 'sub-002';
stage  = 'clean';
outDir = fullfile(derivRoot, subj, 'eeg');
if ~exist(outDir, 'dir'), mkdir(outDir); end
outName = sprintf('%s_task-FaceRecognition_desc-%s_eeg.set', subj, stage);
fprintf('Save stage "%s" as %s\n', stage, fullfile(outDir, outName));

% Small files are cheap: keep ICA weights apart from the data
% icaInfo = struct('icaweights', EEG.icaweights, 'icasphere', EEG.icasphere, ...
%                  'icachansind', EEG.icachansind);
% save(fullfile(outDir, sprintf('%s_task-FaceRecognition_desc-ica.mat', subj)), '-struct', 'icaInfo');

% Convert a non-BIDS dataset (here D1) to BIDS with EEG-BIDS
data = struct('file', {fullfile(eeglabRoot, 'sample_data', 'eeglab_data.set')});
% Practice export only: delete d1_bids when you are done
bids_export(data, 'targetdir', fullfile(projectRoot, 'd1_bids'), ...
    'tInfo', struct('TaskName', 'attention', 'PowerLineFrequency', 60));

Which value should I choose?

How do I check it worked?

• The project folder holds the five folders, and no file in rawdata has changed since you downloaded it.
• The script prints the number of participants, the size of their raw data, and the total estimate.
• Every file in derivatives has a name that starts with sub- and contains desc- with its stage name.
• The d1_bids folder holds dataset_description.json and a sub-001 folder.

What mistakes do people make?

• Saving over the raw file. File › then Resave current dataset(s) writes over the open file. Always use File › then Save current dataset as with the stage name.
• Saving every stage “just in case”. Five stages for 18 participants means more than 20 GB from D4 alone, and most of it is recomputed in minutes.
• Names like final2_new_REAL.set. After a month nobody knows the difference. The stage name in desc- is enough, and the settings belong in the script in code.
• Separating the script from the results. A result without the script that produced it cannot be reproduced. Save the script with every run, or use a version control system such as git for the code folder.

Going further

• EEGLAB Tutorial: https://eeglab.org/tutorials/04_Import/BIDS.html (4. Import data > d. BIDS)
• EEG-BIDS (GitHub): https://github.com/sccn/EEG-BIDS (bids_export.m, bids_export_example.m)
• BIDS specification: https://bids-specification.readthedocs.io/ (Derivatives)
• From raw EEG to BIDS in 2 minutes: https://www.youtube.com/watch?v=LIiRj8-BVk4 (Full video)