Hi Peter,
I've pushed a fix for this to JAC. So now, foreign
NDFs are ignored unless AUTOPROV is set explicitly to '1'. The default
behaviour for native NDFs is unchanged.
David
On 14 April 2010 12:47, Peter W. Draper <[log in to unmask]> wrote:
> On Wed, 14 Apr 2010, David Berry wrote:
>
>> I'm on it. I'm changing the default for AUTOPROV to be NO
>> for foreign format files. This requires a few changes so it may take me an
>> hour or so.
>
> thanks.
>
>> On 14 April 2010 12:12, Peter W. Draper <[log in to unmask]> wrote:
>>>
>>> One point I didn't make about all this. For the CCDPACK commands we have
>>> N
>>> input and N outputs, this seems to scale the additional conversions to
>>> NxN,
>>> not just N, so if you have 100 inputs/output this gets very serious.
>>>
>>> On Wed, 14 Apr 2010, Peter W. Draper wrote:
>>>
>>>> Hi David,
>>>>
>>>> just been tracking down a serious slow down in CCDPACK when using FITS
>>>> files. Applications like CALCOR, DEBIAS etc. that output many files are
>>>> spending a lot of time closing down.
>>>>
>>>> It turns out this is because of the automatic provenance update changes
>>>> you've added (admitably this seems to have started several years ago,
>>>> but
>>>> this is the first time someone has complained to me) which means that
>>>> all
>>>> the output FITS files are re-opened and converted to NDF so that you can
>>>> check if they need their provenances updating and then converted back to
>>>> FITS again...
>>>>
>>>> You can also see this effect in simple KAPPA applications like CADD:
>>>>
>>>>> setenv NDF_SHCVT 1
>>>>> cadd
>>>>
>>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>> '/tmp/temp_fits_d_out1'
>>>> --> Importing: extension CCDPACK
>>>> into: NDF object /tmp/temp_fits_d_out1
>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>>
>>>> SCALAR - Value to be added /1/ >
>>>> OUT - Output NDF > temp.fits
>>>>
>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>> --> Exporting: extension CCDPACK
>>>> from: NDF object /tmp/temp_fits_temp
>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>> '/tmp/temp_fits_temp'
>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> to give: NDF object /tmp/temp_fits_temp
>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>> '/tmp/temp_fits_temp'
>>>> --> Importing: extension CCDPACK
>>>> into: NDF object /tmp/temp_fits_temp
>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>> '/tmp/temp_fits_d_out1'
>>>> --> Importing: extension CCDPACK
>>>> into: NDF object /tmp/temp_fits_d_out1
>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> --> Exporting: extension CCDPACK
>>>> from: NDF object /tmp/temp_fits_temp
>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>> '/tmp/temp_fits_temp'
>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>>
>>>>
>>>> That's a lot of conversions.... Whereas:
>>>>
>>>>> setenv AUTOPROV 0
>>>>> cadd
>>>>
>>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>> '/tmp/temp_fits_d_out1'
>>>> --> Importing: extension CCDPACK
>>>> into: NDF object /tmp/temp_fits_d_out1
>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>>
>>>> SCALAR - Value to be added /1/ >
>>>> OUT - Output NDF > temp.fits
>>>>
>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>> --> Exporting: extension CCDPACK
>>>> from: NDF object /tmp/temp_fits_temp
>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>> '/tmp/temp_fits_temp'
>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>>
>>>> Whoops.
>>>>
>>>> So I think the question is, should automatic provenance updates be
>>>> enabled
>>>> for foreign file conversions? Or should that require AUTOPROV to be
>>>> explicitly set, unlike for NDFs?
>>>>
>>>> Cheers,
>>>>
>>>> Peter.
>>>>
>>>
>>> --
>>> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>>>
>>
>
> --
> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>
|