Dear SIENA users and Steve,
I thought I'd share my recent experiences on "how *not* to run SIENA
in parallel"... and maybe make a few feature-requests in the process ;-)
First, SIENA uses temporary files, created in the current working
directory (and that the original images must be in the CWD), some of
which have names based on just one of the images. I think this can
make things go awry if you have multiple SIENA runs sharing images,
for example, with more than two time-points, etc. So e.g.
submit_job siena patient_time1 patient_time2
submit_job siena patient_time2 patient_time3
might lead to one job messing up the temporary files of the other.
My solution to this at the moment is to script things up to run each
SIENA batch in its own directory. There is a minor annoyance with
this, in that SIENA doesn't seem happy reading symbolic links, so if
you have a lot of SIENA jobs sharing images then you'll end up wasting
a lot of space duplicating the shared original images in separate
directories. (the duplication of the temporary files is fine). It
might be nice if SIENA could be called with full paths to images,
which might be symbolic links (as I believe is fine for other FSL
tools) and a unique output directory (defaulting to something like
$CWD/im1_to_im2). I could probably make these changes myself, but
would wait until after the new FSL release...
Secondly, if you are running a lot of parallel jobs you might be
tempted to automate the extraction of the results, for example with
something like the following inside a (ba)sh loop:
pbvc=`grep -e 'finalPBVC' $siena_results | cut -d' ' -f 2`
This could lead to a bit of a minor disaster, due to the following
point...
SIENA doesn't do any sanity checking of results, or verification of
return-codes from its sub-programs, which means if any of them fail,
it continues regardless. You obviously do get error messages to
stdout/stderr or possibly to the results.siena file, but unfortunately
you also get a "finalPBVC = ..." --- worse still, if the line:
pbvc_average=`echo "10 k $pbvc_forward $pbvc_backward - 2.0 / p" |
dc -`
gets empty variables for its pbvc values, it gives the result:
# echo "10 k - 2.0 / p" | dc -
dc: stack empty
dc: stack empty
2.0
So any automatic grepping of your results will silently return a brain
volume change of 2% for any SIENA runs which went horribly wrong...
Since successful runs should return a value with ten decimal places,
e.g. -1.0123456789 or perhaps 2.0000000000, you can still spot these
exact 2.0 answers as being suspect -- once you know what you're
looking for! Again, I could hack SIENA to check for empty PBVCs, but I
won't do this before the next release (to avoid doing it twice), and I
expect Steve could do a better job of it anyway ;-)
Best,
Ged.
P.S. Steve, upload ID 171020 contains a tarball with two different
examples of SIENA going "horribly wrong" for me. They are using
unrealistic images (e.g. very strong Rician noise added to just one
time-point) but they might still be of interest as examples of the
output that can occur. Note that siena_failure_1/comic_debug/*siena
fails in a slightly different way, getting "nan" for each PBVC (and
then still 2.0% for final PBVC), and seems different on different
machines (debug is on a 32-bit Linux box, comic_debug is on a 64-bit
Linux cluster). I hope some of that is of interest!
|