 |
 |
Thank you for allowing me into this list! I'm an HPC/scientific-computing person tasked with tuning future HPC systems for better Relion/EM support in future designs. My particular focus is to stress test and benchmark various large-scale storage offerings including very large parallel filesystems to see which platforms and which configuration options are most suitable for supporting large-scale Relion usage. I know coming from the genomics world that storage design has a large impact on research throughput and there are key metrics like small-file performance that are indicators for how a storage platform will handle a genomics-heavy workload -- I want to learn similar optimizations and key metrics for EM related scientific workflows. I've been reading the documentation, papers, tutorials and published benchmarks and it looks like: - The overwhelming focus of published benchmarks centers on CPU vs GPU performance on single-node and MPI-connected systems with little to no reported data about storage related benchmarks and optimizations - The standard benchmark data set used in various papers and sites online appears pretty small - small enough now to fit in RAM on larger nvlinked GPU or large memory compute systems and small enough to not really put much stress on a very large or very fast parallel filesystem when writing output or reading in particles or maps If this is not too intrusive of a query I'd welcome some advice and guidance on ... 1) Relion-friendly datasets structured similarly to the popular benchmarking data where particles and maps are already present and can be easily fed into command-line invocations of relion so that I can go out and hammer some big filesystems with reproducible benchmarking runs 2) Guidance on which portions of the relion3 workflow are most storage-intensive (reads and writes, ideally). I think I have a good idea of this from the online tutorial and other published materials. Since others have already focused on GPU vs CPU vs Mixed I figured I can focus a bit more on storage and IO optimization And in the interest of reproducibility if someone has already done large/parallel filesystem testing and tuning I'd love to use the same methods & input data so that I can add more data to what has already been collected. Regards Chris ######################################################################## To unsubscribe from the CCPEM list, click the following link: https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=CCPEM&A=1