 |
 |
It's always nice to see a practical programmer. In fact, I use this method to read a large survey file with multiple record types. There's Household, family level and person level data. I have a program that reads the raw data which has one record per person that is fixed in size and which repeats the relevant household and family data on each person record. The file also contains longitudinal data up to 48 months. The file size is over 1 GB. The programs that use this data is a micro simulation model which can create new variables on the file. I keep track of the variables on the files by creating an ASCII text file with variable names and sizes on the file. I wanted to create read statements that would read this data quickly. So I ended up breaking the data into three different arrays, one each for Household data which occurs 1 time, family records which can occur multiple times for each household and person records which can occur multiple times per household. In the end, I wrote my program so that the file was written out as a binary file, with a few variables at the beginning which defined the dimensions for each of my 3 arrays. The household array is one dimension, the family and person arrays are 3 dimensional with the number of families, by the number of month and finally by the number of variables in the family array. The person array is the same but dimensioned by persons instead of families. So before I read the arrays, I have enough information to allocate arrays of exactly the size I need. So all I need to do is read the allocated arrays in a read statement that looks like: READ(8,END=NNNN) HHLD, FAMILY, PERSON. It's an extremely fast read. Of course I use equivalences so I don't need to create 6 arrays. Now the powers that be don't seem to like equivalences, but it's not hard to keep track of the few real variables. I read the ASCII data dictionary in once at the beginning, this file tells me the location in the variable dimension. This way I don't need to have the input variables in a fixed location since I can lookup the location in my data dictionary once. I hope this gives you some idea of the flexibility of writing out entire arrays and associating a data dictionary with the data. From: Jing Guo <[log in to unmask]> At 05:00 PM 7/10/01 -0400, you wrote: > > > > It's maybe a simple question. > >It may be a simple question, but there is no simple answer. > >If I were your programmer given this assignment, I would first ask you >for the specifications of the file format and the data to be extracted >from the file. I would say that the size of the file is not the same >thing as the size of the data. There may be and may be not a simple >relationship between them. > >Assume the data in the file were written out as a flat data stream. In >this case, one may be able to link the file size to the data size in a >simple expression. Even so, one won't be able to tell if the data is a >simple 1-d array for a row or a column vector or a 2-d array for a >matrix. All that information, if not specified in the file, would have >to be specified somewhere else (Even the software engineers of MATLAB >would have to either store dimensional information in a data file or ask >users to specify them). > >Now it may be geting more controvertial. > >For your specific problem, I would first suggest a long term solution, >by redefine your output file format, such that all dimensional >information are the _first_ piece of data one can get from a file (why >does anyone want to read a file twice?). It is simply because losing >information is not a reversiable adiabatic process. It is not always >possible to reverse engineering data dimensional information from file >sizes. > >If you have no control over the output data format, and the file size >data can be easily mapped to data dimensional information, why don't you >collect the file size data _before_ you use them? For instance, one can >create a list of files with their sizes listed next to their names, such >as: > >13579 "file1.dat" >2468 "file2.dat" >999 "file3.dat" > >I won't suggest some "smart" solutions in Fortran. In my programming >life, I tried to program some "smart" solutions myself. I now believe >all those solutions were wrong. If the data integraty is broken, the >best solution should be to patch the data, not to create some generic >solutions that in the best require some very case-specific information >to work. > > > > Suppose you have several data files, each > > > of them have different length and will be read in by your program. Usually > > you have to specify the corresponding array length in your program > > otherwise you would run in trouble in read. I want to know if there is a > > mean in Fortran to read data file without specify the size, something like > > that in MATLAB: > > > > r1=myfile(:,1) > > > > or > > > > fscanf(fid,'%g %g',[2 inf]); > > > > Thus, all the data will be read in and we can get the size of array by > > simple command size(...). > > > > Many thanks. > > > > Yongcheng > > > > >-- >________________________________ _-__-_-_ _-___--- >Jing Guo, [log in to unmask], (301)614-6172(o), (301)614-6297(fx) >Data Assimilation Office, Code 910.3, NASA/GSFC, Greenbelt, MD 20771 Bob Cohen (703) 534-7618 [log in to unmask]