Dear users,
As a last and final answer as to why the fopen failed for large files, I
got some answers and hints from both Philip Kostov and Jurgen Doornik.
The first wrote
> sorry for replying in this way. For some reason the system does not recognise my e-mail address to reply directly to the list.
>
> I recollect (you may search the archives) that Jurgen stated something like 3Gb as an absolute internal memory limit to Ox. I.e. even if you allocate more physical memory to you computer, either physically or in Windows through allocated RAM drive (I realise you are probably using Unix-like system to do piping) the objects held in the memory cannot exceed this limit. It had to do something with the OS, since the limit was greater for 64 bit OSs, (s.t. availability of a true 64 bit Ox version). In reality the physical memory restrictions of your computer are likely in may cases to be more resrictive.
>
> As far as I know the latest version of Splus has some capabilities of reading large data files but have no clue whether the approach they use is documented and whether in can be applied in a different environment. In principle Splus writes all objects to temporary files (unlike R which keeps them in the memory), so even if their approach can be easily translated, it would probably be much slower than the one you are using.
>
> Best,
> Philip
>
>
whereas the other
> Hi Charles,
>
> Does thgis have something to do with large file support (after a quick google I only read http://www.suse.de/~aj/linux_lfs.html and http://www.lrz-muenchen.de/services/compute/linux-cluster/develop/LF.html).
>
> Ox uses the standard C library to open and read files. Would this file open with a tiny C program that uses fopen?
>
> Bye, Jurgen.
After some tries, I found out that indeed in a simple C program the
fopen command only opens files less than 2^31=2GB in size, when compiled
as 32 bit program. For a 64-bit version on a 64-bit operating system,
larger files can be read.
So it was not especially the large file support, but a limit in a 32-bit
operating system. Indeed
> (s.t. availability of a true 64 bit Ox version)
the problem should disappear.
Greetings,
Charles Bos
|