Hi Peter,
I'm on it. I'm changing the default for AUTOPROV to
be NO for foreign format files. This requires a few changes so it may
take me an hour or so.
David
On 14 April 2010 12:12, Peter W. Draper <[log in to unmask]> wrote:
> One point I didn't make about all this. For the CCDPACK commands we have N
> input and N outputs, this seems to scale the additional conversions to NxN,
> not just N, so if you have 100 inputs/output this gets very serious.
>
> On Wed, 14 Apr 2010, Peter W. Draper wrote:
>
>> Hi David,
>>
>> just been tracking down a serious slow down in CCDPACK when using FITS
>> files. Applications like CALCOR, DEBIAS etc. that output many files are
>> spending a lot of time closing down.
>>
>> It turns out this is because of the automatic provenance update changes
>> you've added (admitably this seems to have started several years ago, but
>> this is the first time someone has complained to me) which means that all
>> the output FITS files are re-opened and converted to NDF so that you can
>> check if they need their provenances updating and then converted back to
>> FITS again...
>>
>> You can also see this effect in simple KAPPA applications like CADD:
>>
>> > setenv NDF_SHCVT 1
>> > cadd
>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> to give: NDF object /tmp/temp_fits_d_out1
>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>> '/tmp/temp_fits_d_out1'
>> --> Importing: extension CCDPACK
>> into: NDF object /tmp/temp_fits_d_out1
>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>
>> SCALAR - Value to be added /1/ >
>> OUT - Output NDF > temp.fits
>>
>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>> --> Exporting: extension CCDPACK
>> from: NDF object /tmp/temp_fits_temp
>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>> -->Converting: NDF object /tmp/temp_fits_temp
>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>> '/tmp/temp_fits_temp'
>> --> Deleting: NDF object /tmp/temp_fits_temp
>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> to give: NDF object /tmp/temp_fits_temp
>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>> '/tmp/temp_fits_temp'
>> --> Importing: extension CCDPACK
>> into: NDF object /tmp/temp_fits_temp
>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> to give: NDF object /tmp/temp_fits_d_out1
>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>> '/tmp/temp_fits_d_out1'
>> --> Importing: extension CCDPACK
>> into: NDF object /tmp/temp_fits_d_out1
>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> --> Exporting: extension CCDPACK
>> from: NDF object /tmp/temp_fits_temp
>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>> -->Converting: NDF object /tmp/temp_fits_temp
>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>> '/tmp/temp_fits_temp'
>> --> Deleting: NDF object /tmp/temp_fits_temp
>>
>>
>> That's a lot of conversions.... Whereas:
>>
>> > setenv AUTOPROV 0
>> > cadd
>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> to give: NDF object /tmp/temp_fits_d_out1
>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>> '/tmp/temp_fits_d_out1'
>> --> Importing: extension CCDPACK
>> into: NDF object /tmp/temp_fits_d_out1
>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>
>> SCALAR - Value to be added /1/ >
>> OUT - Output NDF > temp.fits
>>
>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>> --> Exporting: extension CCDPACK
>> from: NDF object /tmp/temp_fits_temp
>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>> -->Converting: NDF object /tmp/temp_fits_temp
>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>> '/tmp/temp_fits_temp'
>> --> Deleting: NDF object /tmp/temp_fits_temp
>>
>> Whoops.
>>
>> So I think the question is, should automatic provenance updates be enabled
>> for foreign file conversions? Or should that require AUTOPROV to be
>> explicitly set, unlike for NDFs?
>>
>> Cheers,
>>
>> Peter.
>>
>
> --
> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>
