 |
 |
One thing to keep in mind is that there's usually a trade-off between setup (writing and testing) and execution time. For one-off data processing, I'd focus on implementation speed rather than execution speed (in other words, FORTRAN might not be ideal unless you're already fluent with it). That said, I'd take a look at python, octave or R. Python's relatively easy to learn, and more flexible than octave/R; but it doesn't have the built-in statistic functions that octave and R do. One other tip which you've probably already though of - Depending on your runtimes (I don't think 100s MB of data is usually considered an enormous amount, but it'll depend on what you're doing) it may be worth getting things working on a small subset of the data first. Pete Jacob Keller wrote: > Dear List, > > since this probably comes up a lot in manipulation of pdb/reflection files > and so on, I was curious what people thought would be the best language for > the following: I have some huge (100s MB) tables of tab-delimited data on > which I would like to do some math (averaging, sigmas, simple arithmetic, > etc) as well as some sorting and rejecting. It can be done in Excel, but > this is exceedingly slow even in 64-bit, so I am looking to do it through > some scripting. Just as an example, a "sort" which takes >10 min in Excel > takes ~10 sec max with the unix command sort (seems crazy, no?). Any > suggestions? > > Thanks, and sorry for being off-topic, > > Jacob >