Hi Daniel,
This is a known issue which will be fixed in the next release of FSL.
Flirt has output the transformation matrix in hexadecimal format, but most of the FSL tools require the transformation matrices to be in decimal format.
In the meantime, you can use the following bash script to convert your matrix file to decimal format.
Cheers,
Paul
----
#!/bin/bash
# Read from specified file, or from standard input
infile="${1:-/dev/stdin}"
while read line; do
for number in $line; do
printf "%f " "$number"
done
echo
done < $infile