One point I didn't make about all this. For the CCDPACK commands we have N
input and N outputs, this seems to scale the additional conversions to
NxN, not just N, so if you have 100 inputs/output this gets very serious.
On Wed, 14 Apr 2010, Peter W. Draper wrote:
> Hi David,
>
> just been tracking down a serious slow down in CCDPACK when using FITS files.
> Applications like CALCOR, DEBIAS etc. that output many files are spending a
> lot of time closing down.
>
> It turns out this is because of the automatic provenance update changes
> you've added (admitably this seems to have started several years ago, but
> this is the first time someone has complained to me) which means that all the
> output FITS files are re-opened and converted to NDF so that you can check if
> they need their provenances updating and then converted back to FITS again...
>
> You can also see this effect in simple KAPPA applications like CADD:
>
> > setenv NDF_SHCVT 1
> > cadd
> IN - Input NDF data structure [log in to unmask] > d_out1.fit
> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> to give: NDF object /tmp/temp_fits_d_out1
> command: $<CONVERT_DIR>/convertndf from 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
> '/tmp/temp_fits_d_out1'
> --> Importing: extension CCDPACK
> into: NDF object /tmp/temp_fits_d_out1
> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>
> SCALAR - Value to be added /1/ >
> OUT - Output NDF > temp.fits
>
> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> --> Deleting: NDF object /tmp/temp_fits_d_out1
> --> Exporting: extension CCDPACK
> from: NDF object /tmp/temp_fits_temp
> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
> -->Converting: NDF object /tmp/temp_fits_temp
> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CONVERT_DIR>/convertndf to 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' ' '/tmp/temp_fits_temp'
> --> Deleting: NDF object /tmp/temp_fits_temp
> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> to give: NDF object /tmp/temp_fits_temp
> command: $<CONVERT_DIR>/convertndf from 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' ' '/tmp/temp_fits_temp'
> --> Importing: extension CCDPACK
> into: NDF object /tmp/temp_fits_temp
> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> to give: NDF object /tmp/temp_fits_d_out1
> command: $<CONVERT_DIR>/convertndf from 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
> '/tmp/temp_fits_d_out1'
> --> Importing: extension CCDPACK
> into: NDF object /tmp/temp_fits_d_out1
> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
> --> Deleting: NDF object /tmp/temp_fits_d_out1
> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> --> Exporting: extension CCDPACK
> from: NDF object /tmp/temp_fits_temp
> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
> -->Converting: NDF object /tmp/temp_fits_temp
> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CONVERT_DIR>/convertndf to 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' ' '/tmp/temp_fits_temp'
> --> Deleting: NDF object /tmp/temp_fits_temp
>
>
> That's a lot of conversions.... Whereas:
>
> > setenv AUTOPROV 0
> > cadd
> IN - Input NDF data structure [log in to unmask] > d_out1.fit
> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> to give: NDF object /tmp/temp_fits_d_out1
> command: $<CONVERT_DIR>/convertndf from 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
> '/tmp/temp_fits_d_out1'
> --> Importing: extension CCDPACK
> into: NDF object /tmp/temp_fits_d_out1
> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>
> SCALAR - Value to be added /1/ >
> OUT - Output NDF > temp.fits
>
> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> --> Deleting: NDF object /tmp/temp_fits_d_out1
> --> Exporting: extension CCDPACK
> from: NDF object /tmp/temp_fits_temp
> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
> -->Converting: NDF object /tmp/temp_fits_temp
> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
> command: $<CONVERT_DIR>/convertndf to 'FITS'
> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' ' '/tmp/temp_fits_temp'
> --> Deleting: NDF object /tmp/temp_fits_temp
>
> Whoops.
>
> So I think the question is, should automatic provenance updates be enabled
> for foreign file conversions? Or should that require AUTOPROV to be
> explicitly set, unlike for NDFs?
>
> Cheers,
>
> Peter.
>
--
Peter W. Draper, http://astro.dur.ac.uk/~pdraper
|