Hi David,
just been tracking down a serious slow down in CCDPACK when using FITS
files. Applications like CALCOR, DEBIAS etc. that output many files are
spending a lot of time closing down.
It turns out this is because of the automatic provenance update changes
you've added (admitably this seems to have started several years ago, but
this is the first time someone has complained to me) which means that all
the output FITS files are re-opened and converted to NDF so that you can
check if they need their provenances updating and then converted back to
FITS again...
You can also see this effect in simple KAPPA applications like CADD:
> setenv NDF_SHCVT 1
> cadd
IN - Input NDF data structure [log in to unmask] > d_out1.fit
-->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
to give: NDF object /tmp/temp_fits_d_out1
command: $<CONVERT_DIR>/convertndf from 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
'/tmp/temp_fits_d_out1'
--> Importing: extension CCDPACK
into: NDF object /tmp/temp_fits_d_out1
derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
SCALAR - Value to be added /1/ >
OUT - Output NDF > temp.fits
--> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
--> Deleting: NDF object /tmp/temp_fits_d_out1
--> Exporting: extension CCDPACK
from: NDF object /tmp/temp_fits_temp
destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
-->Converting: NDF object /tmp/temp_fits_temp
to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CONVERT_DIR>/convertndf to 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
'/tmp/temp_fits_temp'
--> Deleting: NDF object /tmp/temp_fits_temp
-->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
to give: NDF object /tmp/temp_fits_temp
command: $<CONVERT_DIR>/convertndf from 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
'/tmp/temp_fits_temp'
--> Importing: extension CCDPACK
into: NDF object /tmp/temp_fits_temp
derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
-->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
to give: NDF object /tmp/temp_fits_d_out1
command: $<CONVERT_DIR>/convertndf from 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
'/tmp/temp_fits_d_out1'
--> Importing: extension CCDPACK
into: NDF object /tmp/temp_fits_d_out1
derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
--> Deleting: NDF object /tmp/temp_fits_d_out1
--> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
--> Exporting: extension CCDPACK
from: NDF object /tmp/temp_fits_temp
destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
-->Converting: NDF object /tmp/temp_fits_temp
to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CONVERT_DIR>/convertndf to 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
'/tmp/temp_fits_temp'
--> Deleting: NDF object /tmp/temp_fits_temp
That's a lot of conversions.... Whereas:
> setenv AUTOPROV 0
> cadd
IN - Input NDF data structure [log in to unmask] > d_out1.fit
-->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
to give: NDF object /tmp/temp_fits_d_out1
command: $<CONVERT_DIR>/convertndf from 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
'/tmp/temp_fits_d_out1'
--> Importing: extension CCDPACK
into: NDF object /tmp/temp_fits_d_out1
derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
SCALAR - Value to be added /1/ >
OUT - Output NDF > temp.fits
--> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
--> Deleting: NDF object /tmp/temp_fits_d_out1
--> Exporting: extension CCDPACK
from: NDF object /tmp/temp_fits_temp
destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
-->Converting: NDF object /tmp/temp_fits_temp
to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
command: $<CONVERT_DIR>/convertndf to 'FITS'
'/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
'/tmp/temp_fits_temp'
--> Deleting: NDF object /tmp/temp_fits_temp
Whoops.
So I think the question is, should automatic provenance updates be enabled
for foreign file conversions? Or should that require AUTOPROV to be
explicitly set, unlike for NDFs?
Cheers,
Peter.
|