SciberQuest BOV Reader

The SciberQuest BOV Reader designed for fast parallel I/O of binary Brick-Of-Values files where the brick files contain arrays in single precision IEEE floating point (in FORTRAN speak real*4) format, one array per file. Any given dataset will contain a number of files per time step and a number of timesteps. Arrays may be classified as scalars(1 component), vectors(3 coponents), symetric tensors(6 components) and tensors(9 components), each with one component per-file. Multi-dimensional arrays are expected to be ordered with x-dimension changing fastest, followed by y, followed by z, which is the same ordering as most FORTRAN simulations use and used by VTK. The SQ BOV Reader identifies a dataset which it can read by presence of a single file that has the .bov file extension.

Using the reader in ParaView

As of version 3.98.0 the reader comes in ParaView's SciberQuestToolKit plugin which must be loaded on both the client and server before attempting to open a file. The reader depends on MPI-I/O library and thus ParaView must be compiled with MPI enabled (set PARAVIEW_USE_MPI=ON in the CMakeCache). Additionally ParaView must be launched in client-server mode and the server must be launched by mpirun (or the system's equivelent). When ParaView is compiled with MPI enabled and run in client server mode with the server launched by mpirun, the reader will recognize ".bov"files.

BOV Dataset Format

The SQ BOV meta-data format is illustrated by the following example:

# comments in the metadata file are ignored

# grid dimensions are specified using the following convention:
nx=1024, ny=1024, nz=512

# Data arrays are expected in single precision IEEE floating 
# point format, one array per file, in the same folder, and
# named following the pattern:
#
# NAME_XXXX.gda
#
# where "NAME" is an aribtrary identifier and optionally a 
# component id (eg, den, Vx, or pi-xx). "XXXX" identifies
# the time step id and must be present even when there is
# only a single timestep.
#
# Arrays must be enumerated in the metadata file using the
# following conventions typed and named using the following
# conventions:

# for scalars name a single array
scalar:den

# for vectors the the component arrays are grouped
# into VTK vector for visualization. All vector 
# components must be present and named following the
# convention NAMEx, NAMEy, NAMEz
vector:B

# for tensors all components must be present
# and follow the convention NAME-xx, NAME-xy, ... NAME-zz
tensor:V

# for symetric tensors 6 components must be present
# following the same convention
stensor:pi

# vector, stensor, and tensor components could be treated as 
# scalars by enumerating them as follows:
scalar:Bx
scalar:By
scalar:Bz
scalar:pi-xx
scalar:pi-xy
scalar:pi-xz
scalar:pi-yy
scalar:pi-yz
scalar:pi-zz

Limitations of MPI-I/O

There are a couple of limitations of MPI-I/O to keep in mind. The first being that the reader depends on MPI and must be run with mpirun. The second is that because MPI using int's in its api a single MPI process cannot handle more than 2GB array. This isn't too much of an issue because you can always run more MPI processes if needed.

Shared File I/O on Lustre

when working with large shared files on Lustre there are a few things to be aware of.

file striping parameters affect shared file IO performance
file striping parameters are inherited from the directory in which the file is created
moving a file preserves its striping parameters. file striping parameters can not be modified in place, the file must be copied.
the default striping parameters on most HPC systems do not favor large files and shared I/O so you must set the correct striping parameters if you expect to get reasonable performance.

Our reader (and shared file io in general) runs best when the stripe count on a file is set to its maximal value (use -1) and a alarge stripe size (for eg 512M). You can see the current settings on a file by the command:

lfs getstripe -v /path/to/a/file | grep -E count\|size

The easiest way to set the stripe parameters on file is to create a directory with the desired stripe parameters and either create the file there, or copy the file in. For example:

lfs setstripe -s 512M -c -1 /path/to/directory