SpikeGLX Output Directories and Files
Filenames
Filenames are decorated with several indices.
Gate and trigger indices
All files are created under the control of SpikeGLX gate and trigger modes, which determine when to start writing files. When file writing starts, all files for all streams will get tagged by the current g- and t-indices so you know they go together as a set. They will also be grouped together inside a folder that is named with the run name and the current g-index.
An important characteristic of all of the files in the same Run_g*
folder
is that they start writing at the same time. Said another way, they are
time-aligned at their starts if you had sync enabled.
All of the files from the same run have a special relationship to each other.
During a run the settings are held constant and the data continue to stream.
The files that are written are just snapshots from the continuous stream.
The files contain a metadata item firstSample
that identifies where in
that continuum this file was snapped. This allow the files saved from the
same run to be concatenated in a way that reconstructs the true timeline.
The CatGT tool can perform
time-preserving join operations.
Probe index
- 3A probe filename has no index number after
.imec.
, e.g.myrun_g0_t0.imec.ap.meta
. - All later phases have a probe index, e.g.
myrun_g0_t0.imec0.ap.meta
.
CatGT Tip: Use option -prb_3A
for 3A data, and DON'T use -prb=list
!
Directories
Data directory
At the highest level is the data directory
which you set in SpikeGLX
using menu item Options::Choose Data Directory
. All output data will
be stored into this directory.
Note: You can split data into several top-level directories using the
Multidrive Feature
.
Run folder
-
Version
20190214
and later first create aRun Folder
within the data directory. The run folder holds all output from that run, and is named with a g-index, e.g.myrun_g0
. -
Prior to SpikeGLX version
20190214
there were no run folders. Rather, all datafiles output from a run were placed directly into the currentData Directory
.
CatGT Tip: Use option -no_run_fld
for runs before version 20190214.
Probe folders
Xilinx-based 3A, 3B1
These phases are single-probe. No option for probe folders is provided.
Everything else (3B2, 1.0, 2.0, ...)
Version 20190214
and later provide a Folder per probe
option on the
Save
tab:
-
If not selected, no probe folders are created. Rather, all output files for a run are saved into the
Run Folder
. -
If selected (checked):
- All nidq and obx datafiles are saved directly into the
Run Folder
. - A separate
Probe Folder
for each imec probe is first created in the run folder. The probe folder holds all output from that probe, and is named with both g- and probe-indices, e.g.myrun_g0_imec6
.
- All nidq and obx datafiles are saved directly into the
CatGT Tip: Use option -prb_fld
if option selected.
Multidrive output
Before the multidrive
option was introduced SpikeGLX let you specify
at most one data directory
to hold all your run output folders and files.
As of version 20200309
, there is still a main data directory, but now
you can check a box to enable the splitting of runs across additional
drives/directories to access greater storage capacity when doing very
long runs or when using many probes.
Modulo rule
The rule for distributing run output into several data directories is this:
- As always, the (N) probes have logical indices
[0..N-1]
. - The main data directory is
dir-0
. - The set of all (M) directories, including the main one are enumerated
[dir-0..dir-M-1]
. - NI and obx output is always sent to dir-0.
- Probe-j output is sent to
dir-(j modulo M)
. - Within each dir-k SpikeGLX will create a
run folder
and applicableprobe folders
according to the same rules that apply for a single main data directory.
A Modulo B
means "take the remainder of dividing integersA/B
". Modulo is often writtenmod
and is written%
in C/C++ and Python code.
For example, suppose these parameters:
- dir-0 =
D:\Data
- dir-1 =
E:\DataB
- dir-2 =
G:\DataC
- runname =
myRun
- folder per run is enabled
Example output would go here:
- NI:
D:\Data\myrun\myrun_g0_t0.ni.bin(.meta)
- obx0:
D:\Data\myrun\myrun_g0_t0.obx0.obx.bin(.meta)
- obx1:
D:\Data\myrun\myrun_g0_t0.obx1.obx.bin(.meta)
- Imec0:
D:\Data\myrun\myrun_g0_imec0\myrun_g0_t0.imec0.ap.bin(.meta)
- Imec3:
D:\Data\myrun\myrun_g0_imec3\myrun_g0_t0.imec3.ap.bin(.meta)
- Imec4:
E:\DataB\myrun\myrun_g0_imec4\myrun_g0_t0.imec4.ap.bin(.meta)
- Imec8:
G:\DataC\myrun\myrun_g0_imec8\myrun_g0_t0.imec8.ap.bin(.meta)
Note: You can run several OneBox ADC streams, which would be named 'obx0,' 'obx1,' etc. All obx* data are tiny so we put all obx files in the main dir-0 directory.
New offline behaviors
When multidrive output is enabled, a run's files are distributed over several directories. The SpikeGLX features and tools that work on sets of run files are affected as follows:
-
FileViewer/Link: To link files in the same run, that is, to show them and scroll them together, you first open any binary file from a run, then choose
File/Link
in the Viewer window. In the Link dialog you list which other streams in this run to open and link together. This feature only knows how to search for and link other streams that reside in the same parentdata-dir/run-folder
as the open file. If the multidrive option had split the run, you will only be able to link the streams that live together in one of the split run folders. You'll have to separately view and link data that live in separate data directories. -
Tools/Sample Rates From Run: This dialog still looks the same. You select any binary file from a run and you have an option to calibrate just that data stream, or all of the streams it can find in the same run folder. If the multidrive option had split the run, this tool will only be able to locate and calibrate probes it can find in the same split run folder as the selected binary. That is, you'll have to repeat this process once for each separate data directory.
-
CatGT: Tell CatGT which probes to process by pointing to a data directory
-dir=myDir
and listing the probes you want to process, say-prb=0:3
. In CatGT version 1.2.7 and earlier, if any probe is missing from that directory, the run stops and logs a missing file error. As of CatGT version1.2.8
you can add option-prb_miss_ok
telling CatGT to skip missing probes and continue with the next one it finds in that directory. Note that you will have to make a separate command line for each data directory, but each command line can conveniently list all of the probes in the run. You don't have to figure out the modulo rule for each directory. -
TPrime: Unaffected. All of the input and outfile files {tostream, fromstream, events} already have independently specified paths.
Parsing data by probe
The safest and most reliable way to process your data is to use our tools:
If you are determined to roll your own parsing code, you will need to identify the type of probe used. Once identified, you can look up probe characteristics and parameters in the published probe table.
The metadata key that identifies a probe is its part number imDatPrb_pn
.
Virtually all meta files later than phase 3A will contain this value.
fin