On Wed, 14 Apr 2010, David Berry wrote:
> I've pushed a fix for this to JAC. So now, foreign
> NDFs are ignored unless AUTOPROV is set explicitly to '1'. The default
> behaviour for native NDFs is unchanged.
Thanks for the quick work, seems to be doing the trick and things are much
faster now.
Cheers,
Peter.
> On 14 April 2010 12:47, Peter W. Draper <[log in to unmask]> wrote:
>> On Wed, 14 Apr 2010, David Berry wrote:
>>
>>> I'm on it. I'm changing the default for AUTOPROV to be NO
>>> for foreign format files. This requires a few changes so it may take me an
>>> hour or so.
>>
>> thanks.
>>
>>> On 14 April 2010 12:12, Peter W. Draper <[log in to unmask]> wrote:
>>>>
>>>> One point I didn't make about all this. For the CCDPACK commands we have
>>>> N
>>>> input and N outputs, this seems to scale the additional conversions to
>>>> NxN,
>>>> not just N, so if you have 100 inputs/output this gets very serious.
>>>>
>>>> On Wed, 14 Apr 2010, Peter W. Draper wrote:
>>>>
>>>>> Hi David,
>>>>>
>>>>> just been tracking down a serious slow down in CCDPACK when using FITS
>>>>> files. Applications like CALCOR, DEBIAS etc. that output many files are
>>>>> spending a lot of time closing down.
>>>>>
>>>>> It turns out this is because of the automatic provenance update changes
>>>>> you've added (admitably this seems to have started several years ago,
>>>>> but
>>>>> this is the first time someone has complained to me) which means that
>>>>> all
>>>>> the output FITS files are re-opened and converted to NDF so that you can
>>>>> check if they need their provenances updating and then converted back to
>>>>> FITS again...
>>>>>
>>>>> You can also see this effect in simple KAPPA applications like CADD:
>>>>>
>>>>>> setenv NDF_SHCVT 1
>>>>>> cadd
>>>>>
>>>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>>> '/tmp/temp_fits_d_out1'
>>>>> --> Importing: extension CCDPACK
>>>>> into: NDF object /tmp/temp_fits_d_out1
>>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>>>
>>>>> SCALAR - Value to be added /1/ >
>>>>> OUT - Output NDF > temp.fits
>>>>>
>>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>>> --> Exporting: extension CCDPACK
>>>>> from: NDF object /tmp/temp_fits_temp
>>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>>> '/tmp/temp_fits_temp'
>>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> to give: NDF object /tmp/temp_fits_temp
>>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>>> '/tmp/temp_fits_temp'
>>>>> --> Importing: extension CCDPACK
>>>>> into: NDF object /tmp/temp_fits_temp
>>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_temp
>>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>>> '/tmp/temp_fits_d_out1'
>>>>> --> Importing: extension CCDPACK
>>>>> into: NDF object /tmp/temp_fits_d_out1
>>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> --> Exporting: extension CCDPACK
>>>>> from: NDF object /tmp/temp_fits_temp
>>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>>> '/tmp/temp_fits_temp'
>>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>>>
>>>>>
>>>>> That's a lot of conversions.... Whereas:
>>>>>
>>>>>> setenv AUTOPROV 0
>>>>>> cadd
>>>>>
>>>>> IN - Input NDF data structure [log in to unmask] > d_out1.fit
>>>>> -->Converting: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> to give: NDF object /tmp/temp_fits_d_out1
>>>>> command: $<CONVERT_DIR>/convertndf from 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'd_out1' '.fit' ' '
>>>>> '/tmp/temp_fits_d_out1'
>>>>> --> Importing: extension CCDPACK
>>>>> into: NDF object /tmp/temp_fits_d_out1
>>>>> derived from: FITS file /loc/pwda/pdraper/scratch/ccdpack/d_out1.fit
>>>>> command: $<CCDPACK_DIR>/ccdimp.csh /tmp/temp_fits_d_out1
>>>>>
>>>>> SCALAR - Value to be added /1/ >
>>>>> OUT - Output NDF > temp.fits
>>>>>
>>>>> --> Deleting: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> --> Deleting: NDF object /tmp/temp_fits_d_out1
>>>>> --> Exporting: extension CCDPACK
>>>>> from: NDF object /tmp/temp_fits_temp
>>>>> destined for: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CCDPACK_DIR>/ccdexp.csh /tmp/temp_fits_temp
>>>>> -->Converting: NDF object /tmp/temp_fits_temp
>>>>> to give: FITS file /loc/pwda/pdraper/scratch/ccdpack/temp.fits
>>>>> command: $<CONVERT_DIR>/convertndf to 'FITS'
>>>>> '/loc/pwda/pdraper/scratch/ccdpack/' 'temp' '.fits' ' '
>>>>> '/tmp/temp_fits_temp'
>>>>> --> Deleting: NDF object /tmp/temp_fits_temp
>>>>>
>>>>> Whoops.
>>>>>
>>>>> So I think the question is, should automatic provenance updates be
>>>>> enabled
>>>>> for foreign file conversions? Or should that require AUTOPROV to be
>>>>> explicitly set, unlike for NDFs?
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Peter.
>>>>>
>>>>
>>>> --
>>>> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>>>>
>>>
>>
>> --
>> Peter W. Draper, http://astro.dur.ac.uk/~pdraper
>>
>
--
Peter W. Draper, http://astro.dur.ac.uk/~pdraper
|