This is an old revision of the document!
Table of Contents
Scripting of the processing stream (example)
Motivation
While for educational purposes it is quite useful to perform steps more manually, on a daily basis it is much more convenient (and less error prone) to have one or several scripts do the “work” and only take care of their configuration.
Requirements
For this example, the following programs must be installed and information available:
- bash, find, mkdir, mv, rsync, ssh, tar (come with most OSes that have a terminal or shell)
- Matlab v7.0 or higher, SPM5/SPM8, and NeuroElf v0.9a or higher
- location of subject data at the scanning facility (server)
- local folder and knowledge about desired structure
- onsets in a tabular structure where one column codes the subject and negative differences between onset times code the run separation (for an example, see below)
Sub components
As part of this scripted approach requires the shell, there are multiple scripts. Part of the data handling is performed by shell commands (data retrieval, archiving) whereas the processing is implemented via SPM's job manager. The following steps are covered:
- data import (from a remote SSH-accessible server)
- correct renaming and creation of folders (into existing study structure)
- archiving (possibly configured as a regularly executed job via crontab)
- fMRI quality assessment (per run)
- preprocessing (SPM5 or SPM8)
- conversion into BrainVoyager QX's VTC format
- automatic logfile parsing (PRT generation)
- computing a RFX-GLM of all available subjects (upon request)
Data import
- SCANUnit_import_subject.bash
#!/bin/bash # settings ; PLEASE ADAPT LOCAL_FOLDER=/Users/XXX/Desktop/CPU REMOTE_FOLDER=/export/data/scanner/a_to_c/CPU REMOTE_HOST=host.domain.columbia.edu REMOTE_USER=xxx # ensure there are arguments if [[ "a$@" == "a" ]] ; then echo "No subjects requested." echo "USAGE (import one subject): $0 subject1" echo "USAGE (import several subjects): $0 subject1 subject2 subject3 ..." echo "USAGE (import several subjects, pattern based): $0 subj\*patt" echo "---" echo "This script uses ssh. You will be asked to enter" echo "the password of user $REMOTE_USER on host $REMOTE_HOST." echo "---" echo "Data will be stored in $LOCAL_FOLDER." exit 1; fi # change into local folder cd $LOCAL_FOLDER # retrieve requested files ssh -l $REMOTE_USER REMOTE_HOST "cd $REMOTE_FOLDER ; tar -cf - $@" | tar -xf -
Folder naming and creation
- SCANUnit_subject_foldernames.bash
#!/bin/bash # settings ; PLEASE ADAPT LOCAL_FOLDER=/Users/XXX/Desktop/CPU SUBJ_PATTERN=CPU* SUBFOLDERS="functional onsets structural" RAWFOLDER=raw # Note: the SUBFOLDERS variable can be easily adapted to create # several levels of folders: # SUBFOLDERS="functional functional/raw functional/prepro ..." # for every folder found (matching the pattern) find $LOCAL_FOLDER -type d -mindepth 1 -maxdepth 1 -name "$SUBJ_PATTERN" | while read subject_folder ; do # make sure all subfolders we need exist for sub in $SUBFOLDERS ; do mkdir $subject_folder/$sub 2>/dev/null done # rename original folder, if existent, to raw find $subject_folder -type d -mindepth 1 -maxdepth 1 -name "$SUBJ_PATTERN" -exec mv {} $RAWFOLDER \; done
Archiving
For this, there is really no need for a script, simply let rsync do the work:
- rsync_example.sh
rsync -aut /Users/XXX/Desktop/CPU /Volumes/CPU/ImagingData
This can be easily added as a line to crontab (callable via crontab -e
):
- rsync_crontab_snip.txt
30 2 * * * /usr/bin/rsync -aut /Users/XXX/Desktop/CPU /Volumes/CPU/ImagingData
Note: On MacOSX to ensure that the backuping volume is always available, add it to the user's startup items list in System Preferences, and store the password to that network volume in your keychain!
DICOM import
This part of the scripted process is already covered at its own, proper page. See image file conversion for more information.
Preprocessing
Using the spm5_preprojobs
function, we can easily setup a small script that preprocesses all (remaining) subjects:
- SCANUnit_CPU_prepro.m
% folder location subjfolder = '/Users/Desktop/XXX/CPU'; subjpattern = 'CPU*'; funcfolder = 'functional'; anatfolder = 'struct'; % settings (for spm5_preprojobs) funcpattern = 'r*340'; anatpattern = ''; usestruct = true; runjobs = true; disdaqs = 5; smoothmm = 6; sliceorder = 'aio'; tr = 2; % complete settings if ~isempty(funcpattern) ffp = [funcfolder '/' funcpattern]; else ffp = funcfolder; end if ~isempty(anatpattern) anatpattern = [anatfolder '/' anatpattern]; else anatpattern = anatfolder; end ppopts = struct( ... 'fun2str', usestruct, ... 'jobrun', runjobs, ... 'skip', disdaqs, ... 'smk', smoothmm, ... 'sto', sliceorder, ... 'sttr', tr); % find subjects subjects = findfiles(subjfolder, subjpattern, 'depth=1', 'dirs=1'); % for each subject for sc = 1:numel(subjects) % determine whether the subject needs processing rps = findfiles([subjects{sc} '/' funcfolder], 'rp*.txt'); if ~isempty(rps) continue; end % get subject ID [basefld, subjid] = fileparts(subjects{sc}); subjid(subjid == '_') = []; % run fMRI quality on all runs runs = findfiles([subjects{sc} '/' funcfolder], funcpattern, 'dirs=1', 'depth=1'); for rc = 1:numel(runs) q = fmriquality(findfiles(runs{rc}, {'*.img', '*.nii'})); save(sprintf('%s/fmriquality.mat', runs{rc}), 'q'); clear q; end % run preprocessing [j, jh, ppfiles] = spm5_preprojobs(subjects{sc}, ffp, anatpattern, ppopts); % for each run, create one VTC for rc = 1:numel(ppfiles) [basefld, runid] = fileparts(fileparts(ppfiles{rc}{1})); vtc = importvtcfromanalyze(ppfiles{rc}); vtc.TR = round(1000 * tr); vtc.SaveAs(sprintf('%s/%s/%s_%s.vtc', subjects{sc}, funcfolder, subjid, runid); vtc.ClearObject; end end