On Wed, 14 Apr 2010, David Berry wrote:
> I'm on it. I'm changing the default for AUTOPROV to be
> NO for foreign format files. This requires a few changes so it may take
> me an hour or so.
thanks.
> On 14 April 2010 12:12, Peter W. Draper <[log in to unmask]> wrote:
>> One point I didn't make about all this. For the CCDPACK commands we have N
>> input and N outputs, this seems to scale the additional conversions to NxN,
>> not just N, so if you have 100 inputs/output this gets very serious.
>>
>> On Wed, 14 Apr 2010, Peter W. Draper wrote:
>>
>>> Hi David,
>>>
>>> just been tracking down a serious slow down in CCDPACK when using FITS
>>> files. Applications like CALCOR, DEBIAS etc. that output many files are
>>> spending a lot of time closing down.
>>>
>>> It turns out this is because of the automatic provenance update changes
>>> you've added (admitably this seems to have started several years ago, but
>>> this is the first time someone has complained to me) which means that all
>>> the output FITS files are re-opened and converted to NDF so that you can
>>> check if they need their provenances updating and then converted back to
>>> FITS again...
>>>
>>> You can also see this effect in simple KAPPA applications like CADD:
>>>
>>>> setenv NDF_SHCVT 1
>>>> cadd
>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> to give: NDF object /tmp/temp_fits_d_out1
>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>> '/tmp/temp_fits_d_out1'
>>> --> Importing: extension CCDPACK
>>> into: NDF object /tmp/temp_fits_d_out1
>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>
>>> SCALAR - Value to be added /1/ >
>>> OUT - Output NDF > temp.fits
>>>
>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>> --> Exporting: extension CCDPACK
>>> from: NDF object /tmp/temp_fits_temp
>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>> -->Converting: NDF object /tmp/temp_fits_temp
>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>> '/tmp/temp_fits_temp'
>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> to give: NDF object /tmp/temp_fits_temp
>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>> '/tmp/temp_fits_temp'
>>> --> Importing: extension CCDPACK
>>> into: NDF object /tmp/temp_fits_temp
>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> to give: NDF object /tmp/temp_fits_d_out1
>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>> '/tmp/temp_fits_d_out1'
>>> --> Importing: extension CCDPACK
>>> into: NDF object /tmp/temp_fits_d_out1
>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> --> Exporting: extension CCDPACK
>>> from: NDF object /tmp/temp_fits_temp
>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>> -->Converting: NDF object /tmp/temp_fits_temp
>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>> '/tmp/temp_fits_temp'
>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>
>>>
>>> That's a lot of conversions.... Whereas:
>>>
>>>> setenv AUTOPROV 0
>>>> cadd
>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> to give: NDF object /tmp/temp_fits_d_out1
>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>> '/tmp/temp_fits_d_out1'
>>> --> Importing: extension CCDPACK
>>> into: NDF object /tmp/temp_fits_d_out1
>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>
>>> SCALAR - Value to be added /1/ >
>>> OUT - Output NDF > temp.fits
>>>
>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>> --> Exporting: extension CCDPACK
>>> from: NDF object /tmp/temp_fits_temp
>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>> -->Converting: NDF object /tmp/temp_fits_temp
>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>> '/tmp/temp_fits_temp'
>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>
>>> Whoops.
>>>
>>> So I think the question is, should automatic provenance updates be enabled
>>> for foreign file conversions? Or should that require AUTOPROV to be
>>> explicitly set, unlike for NDFs?
>>>
>>> Cheers,
>>>
>>> Peter.
>>>
>>
>> --
>> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>>
>
--
Peter W. Draper, http://astro.dur.ac.uk/~pdraper
|