Dear Sasha,
I would have to look at it in more details, but I would guess that more
changes are required (it was the case for @file_array) and some of them
can be well hidden.
Again, SPM can handle large files (as is often the case with M/EEG) but
not if the 'spm_vol' C library is used.
If you are at the modelling stage and spm_global is the only problem,
you can delete the MEX files spm_global.mex*, edit the function
spm_global.m and uncomment the MATLAB code. You will also probably have
to change one line so that it reads
D = spm_data_read(V(i),:,:,:); % assuming 3D data
This should be enough to run all the stats as the rest of the code uses
the @file_array library. Let me know if you come across something else
that I overlooked.
Out of curiosity, what is the size of your data (file size and image
dimensions)?
Best regards,
Guillaume.
On 08/10/13 15:32, Alejandro Vicente Grabovetsky wrote:
> Dear Guillaume,
>
> SPM12b still breaks with input files larger than 2gb when running
> spm_globals, with the very same "file too small error".
>
> Is the fix suggested by Thierry valid, and would it be possible to
> implement it into SPM8 and/or 12, as it would be very helpful for
> individuals working with very large datasets?
>
> Best regards,
> Sasha
>
>
> On 24 April 2013 12:26, Guillaume Flandin <[log in to unmask]
> <mailto:[log in to unmask]>> wrote:
>
> Dear Thierry,
>
> Thanks for the feedback.
> The code in spm_mapping was not written with Large File Support (LFS) in
> mind at the time so there is indeed a file size limitation of 2GB. This
> is not the case if you access files with a second library that SPM
> provides, file_array. This was mentioned in an earlier post on this
> list:
> https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=spm;5f45230c.1001
> We hope to have them unified at some point and, for example, model
> fitting (spm_spm.m) in SPM12b now uses the second library.
>
> Best regards,
> Guillaume.
>
>
> On 24/04/13 09:35, Thierry Chaminade wrote:
> > Dear SPMers,
> >
> > Recently I sent a message to the list but didn't get response from the
> > FIL so we kept on digging. I think similar issues have been
> reported in
> > the past
> > (eg https://www.jiscmail.ac.uk/cgi-bin/webadmin?A2=spm;a029bc92.1302).
> >
> > I had my institute IT expert Florent Jaillet help me with the code to
> > track down the origin of this limit. He found a fix to the code that
> > worked perfectly for me, my large GLM-Flex design was computed without
> > any more error. So I think at least, we're on the good track. It's a
> > quick fix because we don't master all details of John's code. So I
> would
> > greatly encourage SPM programmers (particularly John as he wrote these
> > files) to read the bug report Florent wrote copied hereunder.
> >
> > In my layman language, the quick fix is using "long long" type for
> > variable "off" instead of "int" in spm_mapping.c to allow the offset
> > pointer reading the disk to go further than the 2^31 limit of int. But
> > please read the full report hereunder:
> >
> >
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-
> >
> > The problem we reported seems to be due to a bug in spm_mapping.c
> making
> > the functions that use spm_mapping unable to handle correctly files
> > bigger than around 2GB. We noticed the problem with spm_slice_vol and
> > spm_sample_vol, but I guess that some other SPM mex functions
> could have
> > the same problem.
> >
> > Here are the details about how to reproduce the bug and a quick and
> > dirty modification that seems to solve the problem in our case:
> >
> > Our configuration:
> > Operating system: Debian GNU/Linux 6 (Squeeze) 64 bits
> > MATLAB version: R2012b (8.0.0.783)
> > SPM 8 update revision number: 5236 (04-Feb-2013)
> >
> > To reproduce the bug, run the following code:
> > %%%
> > spm8_path = '/opt/spm8'; % put the path to your SPM8 installation here
> > existing_file_name = [spm8_path, '/canonical/avg152PD.nii'];
> > test_file_name = 'bug_test.nii';
> > V = spm_vol(existing_file_name);
> > [Y, xyz] = spm_read_vols(V);
> > V.fname = test_file_name;
> > ind = 2380; % there is no bug for ind < 2380
> > V.n = [ind, 1];
> > spm_write_vol(V, Y);
> > V1000 = spm_vol([test_file_name, ',', num2str(ind)]);
> > [Y, xyz] = spm_read_vols(V1000);
> > %%%
> >
> > You should get the following error:
> >
> > %%%
> > Error using spm_slice_vol
> > File too small.
> >
> > Error in spm_read_vols (line 34)
> > Y(:,:,p,i) = spm_slice_vol(V(i),spm_matrix(__[0 0
> p]),V(i).dim(1:2),0);
> >
> > Error in minimal_bug (line 11)
> > [Y, xyz] = spm_read_vols(V1000);
> > %%%
> >
> > You can solve this problem by modifying the type for the variable
> "off"
> > in spm_mapping.c from "int" to "long long" and recompile the SPM
> mex files.
> > That means changing line 206 in spm_mapping.c which currently is:
> > int dsize = 0, off;
> > to:
> > int dsize = 0;
> > long long off;
> >
> > and line 361 which is:
> > off = (int)fabs(pr[2]);
> > to:
> > off = (long long)fabs(pr[2]);
> >
> > I didn't get into the details of the code, so I'm not sure about
> what is
> > happening, but it looks like "off" stands for "offset" and that the
> > value of this offset being stored in an int, so a 32 bits signed
> integer
> > on our platform, is limited to a maximal value of around 2^31. That
> > offset is likely to be counting a number of bytes, and 2^31 B is 2GB,
> > the approximate limit of the file size leading to the bug. From what I
> > saw when debugging, if we try to reach data that are beyond the
> limit of
> > 2GB, the conversion from the double value of "pr[2]" (which seems
> to be
> > right) to the int value "off" goes wrong due to the int overflow,
> > leading to a wrong value of the variable "off" (it becomes negative).
> >
> > I'm not sure that changing the type of the variable "off" is the right
> > solution to the problem as I don't know what the code is doing.
> > But I think that some SPM developers should have a thorough look
> at this
> > problem, find the best solution and correct the bug everywhere it
> can be
> > (I don't know if spm_mapping.c is the only affected file).
> > For example it is very likely that line 380 of spm_mapping.c
> should also
> > be modified:
> > off = (int)fabs(pr[2+j*3]);
>
> --
> Guillaume Flandin, PhD
> Wellcome Trust Centre for Neuroimaging
> University College London
> 12 Queen Square
> London WC1N 3BG
>
>
>
>
> --
> Vicente Grabovetsky, Alejandro (Sasha)
> Postdoctoral researcher
> Donders Centre for Cognitive Neuroimaging
> http://www.doellerlab.com
--
Guillaume Flandin, PhD
Wellcome Trust Centre for Neuroimaging
University College London
12 Queen Square
London WC1N 3BG
|
