Rather than writing your own code to store the matrix on the hard disk, you
can very easily store them on the harddisk by allocating a chunk of your hard
disk as "virtual memory" and pretending that it is RAM - that's the virtual
memory Jurgen was referring to.
http://en.wikipedia.org/wiki/Virtual_memory
The problem with this is that accessing data on hard disks is very slow when
compared to RAM, as Jurgen pointed out.
On Friday 29 April 2005 22:39, Dr Richard Hudson wrote:
> Max, in refence to your note on matrix products, could not
> a program be written to sequentially write to disc the result of each
> product
> done? After all, matrices are really stored on disc as one column vector
> and its
> the printout that makes them appear as m x m (or m x n).
> In such case the original column vector of nums would be operated upon and
> the product result be written to another HDD in the machine.
> (Only trying to help the chap, thats all. Seems interesting)
> Richard Hudson
>
>
>
>
>
>
>
>
>
> ----- Original Message -----
> From: "Max Bruche" <[log in to unmask]>
> To: <[log in to unmask]>
> Sent: Friday, April 29, 2005 11:46 PM
> Subject: Re: Large matrix (run out of memory)
>
> >I just thought I would point out how Jurgen came up with this number in
> >case
> > someone wants to work out how much memory a particular matrix requires:
> >
> > Elements of matrices are stored as double precision floating point
> > numbers,
> > which require 64 bits of storage. 8 bit = 1 byte, so this is 8 bytes of
> > storage. If all elements are stored (the matrix is not sparse), this
> > means 55'000 * 55'000 * 8 = 24'200'000'000 bytes.
> >
> > If 1KB = 1000 Bytes, and 1MB = 1000 KB, and 1 GB = 1000 MB (as people
> > usually
> > assume when they talk about hard disk capacity), this matrix will take up
> > 24.2 GB.
> >
> > If 1 KB = 1024 Bytes, 1 MB = 1024 KB and 1 GB = 1024 MB (as people
> > usually assume when they talk about computer memory/ RAM):
> > 24'200'000'000 / (1024^3) = ~ 22.54 GB
> >
> > Max
> >
> > On Friday 29 April 2005 11:25, Jurgen Doornik wrote:
> >> Such a matrix requires >24 GB, which wouldn't fit. I expect that if you
> >> increase your virtual memory (perhaps allowing for a copy as well),
> >> you'll have to wait forever to do the computations.
> >>
> >> The only way round is to make the mathematical expressions more
> >> efficient. For example, it may be possible to maintain the kronecker
> >> structure throughout, so that the actual full matrix need never be
> >> computed. Similarly `hat' matrices can usually be avoided.
> >>
> >> Jurgen.
> >>
> >> oxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxox
> >> oxoxoxox 3rd OxMetrics user conference August 2005
> >> oxoxoxox Cass Business School, London
> >> oxoxoxox Dates still to be announced
> >> oxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxox
> >> Dr Jurgen A Doornik
> >> Nuffield College, Oxford OX1 1NF, UK
> >> tel. UK: +44-1865-278610 fax +44-1865-278621
> >> http://www.doornik.com
> >> http://www.oxmetrics.net
> >> oxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxoxox
> >>
> >> V. Sarafidis wrote:
> >> > Hello,
> >> >
> >> > I would like to ask you the following:
> >> >
> >> > I have a panel N=5500, T=10. I am trying to create the Kronecker
> >> > product
> >> > of two matrices (N*N)*(T*T), so the end matrix should be a square
> >> > matrix
> >> > (N*T)*(N*T)=55,000*55,000.
> >> >
> >> > The problem is that I get the message "out of memory". The pc I am
> >> > using
> >> > has 512 MB actual memory and 2 GB virtual memory. But is seems that
> >> > this
> >> > is not sufficient.
> >> >
> >> > When I cut my sample to N=1075 the system reports committed
> >> > memory (using the task manager) at about 1150 MB (which is less
> >> > than half what is available). But if I increase N to more than 1075,
> >> > committed memory usage doesn't go more than 400MB.
> >> >
> >> > Is it possible then to handle a matrix with this size?
> >> >
> >> > Many thanks in advance
> >> >
> >> > Vasilis
> >
> > --
> > No virus found in this incoming message.
> > Checked by AVG Anti-Virus.
> > Version: 7.0.308 / Virus Database: 266.10.4 - Release Date: 27/04/2005
>
> --
> No virus found in this outgoing message.
> Checked by AVG Anti-Virus.
> Version: 7.0.308 / Virus Database: 266.11.0 - Release Date: 29/04/2005
-------------------------------------------------------
|