TRX Specification#

This page defines the TRX file format specification. The authoritative specification is maintained in the trx-spec repository.

General#

  • Uncompressed ZIP file or simple folder architecture

  • File architecture describes the data

  • Each file basename is the metadata’s name

  • Each file extension is the metadata’s dtype

  • Each file dimension is in the value between basename and metadata

  • 1-dimensional arrays do not have to follow this convention for readability

  • All arrays have a C-style memory layout (row-major)

  • All arrays have a little-endian byte order

  • Compression is optional

    • Use ZIP_STORE if no compression is desired

    • Use ZIP_DEFLATE if compression is desired

    • Compressed TRX files will have to be decompressed before being loaded

Arrays#

positions#

  • Written in world space (RASMM), like TCK files

  • Always float16, float32, or float64 (default: float16)

  • Stored as contiguous 3D array (NB_VERTICES, 3)

offsets#

  • Always uint32 or uint64

  • Contains the index where each streamline starts in the positions array, beginning at 0

  • Two ways of knowing how many vertices there are:

    • Check the header

    • Positions array size / dtype / 3

  • To get streamline lengths: append the total number of vertices to the end of offsets and take the differences between consecutive elements (numpy.ediff1d)

dpv (data per vertex)#

  • Always of size (NB_VERTICES, 1) or (NB_VERTICES, N)

dps (data per streamline)#

  • Always of size (NB_STREAMLINES, 1) or (NB_STREAMLINES, N)

groups#

Groups are tables of indices that allow sparse and overlapping representation (clusters, connectomics, bundles).

  • All indices must satisfy 0 <= id < NB_STREAMLINES

  • Datatype should be uint32

  • Allow efficient extraction of a predefined streamline subset from memmaps

  • Variable sizes across groups

dpg (data per group)#

  • Each subfolder is named after its corresponding group

  • Not all metadata need be present in all groups

  • Always of size (1,) or (N,)

Accepted datatypes#

  • int8, int16, int32, int64

  • uint8, uint16, uint32, uint64

  • float16, float32, float64

  • bit (for binary/boolean arrays)

Example structure#

OHBM_demo.trx
├── dpg
│   ├── AF_L
│   │   ├── mean_fa.float16
│   │   ├── shuffle_colors.3.uint8
│   │   └── volume.uint32
│   ├── AF_R
│   │   ├── mean_fa.float16
│   │   ├── shuffle_colors.3.uint8
│   │   └── volume.uint32
│   ├── CC
│   │   ├── mean_fa.float16
│   │   ├── shuffle_colors.3.uint8
│   │   └── volume.uint32
│   ├── CST_L
│   │   └── shuffle_colors.3.uint8
│   ├── CST_R
│   │   └── shuffle_colors.3.uint8
│   ├── SLF_L
│   │   ├── mean_fa.float16
│   │   ├── shuffle_colors.3.uint8
│   │   └── volume.uint32
│   └── SLF_R
│       ├── mean_fa.float16
│       ├── shuffle_colors.3.uint8
│       └── volume.uint32
├── dpv
│   ├── color_x.uint8
│   ├── color_y.uint8
│   ├── color_z.uint8
│   └── fa.float16
├── dps
│   ├── algo.uint8
│   ├── algo.json
│   ├── clusters_QB.uint16
│   ├── commit_colors.3.uint8
│   └── commit_weights.float32
├── groups
│   ├── AF_L.uint32
│   ├── AF_R.uint32
│   ├── CC.uint32
│   ├── CST_L.uint32
│   ├── CST_R.uint32
│   ├── SLF_L.uint32
│   └── SLF_R.uint32
├── header.json
├── offsets.uint64
└── positions.3.float16

Example code#

from trx_file_memmap import TrxFile, load, save
import numpy as np

trx = load('complete_big_v5.trx')

# Access the header (dict) / streamlines (ArraySequences)
trx.header
trx.streamlines

# Access the dpv (dict) / dps (dict)
trx.data_per_vertex
trx.data_per_streamline

# Access the groups (dict) / dpg (dict)
trx.groups
trx.data_per_group

# Get a random subset of 10000 streamlines
indices = np.arange(len(trx.streamlines._lengths))
np.random.shuffle(indices)
sub_trx = trx.select(indices[0:10000])
save(sub_trx, 'random_1000.trx')

# Get sub-groups only, from the random subset
for key in sub_trx.groups.keys():
    group_trx = sub_trx.get_group(key)
    save(group_trx, '{}.trx'.format(key))

# Pre-allocate memmaps and append 100x the random subset
alloc_trx = TrxFile(nb_streamlines=1500000, nb_vertices=500000000, init_as=trx)
for i in range(100):
    alloc_trx.append(sub_trx)

# Resize to remove the unused portion of the memmap
alloc_trx.resize()