Update on the issue MICE reported at last week's meeting:
Since I upgraded a MICE-specific DPM at Brunel from 1.8.8 to EMI3 1.8.10,
I wasn't able to retrieve uploaded files bigger than 2 GB(*). Smaller
files and older files were both OK. The client received an error code 500,
whether it was wget, lcg-cp or direct globus-url-copy.
The logs show errors along the lines of:
May 27 20:50:46 dgc-grid-38 httpd[11364]: [4] dmlite Mysql dumpCStat : name: data_all_miceecserv1_20160214.tar.gz parent: 240605 csumtype: AD csumvalue: ee4e67e3 acl:
May 27 20:50:46 dgc-grid-38 httpd[11364]: dmlite DmException !! setMessage : [#01.000350] Probably malformed JSON data (<unspecified file>(1): ...
hence my babbling last week about thinking it's related to permissions
getting mangled.
Looking closely, I found that simple uploads were fine; the file was
actually becoming inaccessible when the server completed it's
checksumming, either at the end of a transfer or in response to an
lcg-get-checksum. This seems to have been because of a change in the way
the data is stored in the database:
| name | fileclass | filemode | filesize | status | csumtype | csumvalue | acl | xattr |
| 07980.tar | 0 | 33204 | 4239974400 | - | AD | b5846ea3 | | {"type": 0, "checksum.adler32": "b5846ea3", "filesize": 4239974400} |
whereas an older file has
| name | fileclass | filemode | filesize | status | csumtype | csumvalue | acl | xattr |
| 07580.tar | 0 | 33060 | 2641233920 | - | | | | {"type": 0, "checksum.adler32": "65fd05ce"} |
Following Raul's advice that this is down to a dodgy dmlite-libs, I've
upgraded to 1.8.11 (dmlite-0.7.6-3) by pointing YUM at the EPEL repo and
doing a yum update (rebooted, but not run YAIM). This seems to have fixed
the problem so far - the inaccessible files are now visible again, and I
can upload and retrieve large files.
So for me the issue has apparently disappeared with the software update; I
don't recall seeing any updated dmlite-libs in the older repos.
Thanks
Henry
* i.e. half a potato.
On Tue, 19 Jul 2016, Jensen, Jens (STFC,RAL,SC) wrote:
> Hi,
>
> Tomorrow, I will be at ECMWF for a project on climate data - so will be
> unable to run the storage meeting. Also, Sam is away. I have talked with
> Brian; he (and Marcus?) should be able to chair tomorrow's meeting.
>
> Don't forget to discuss operational blog posts... and the September
> workshop, it now looks like Brian will be the lucky person to go and
> present.
>
> Also we have tentatively scheduled next week's meeting (27th) to be
> cancelled, economising effort in the summer heat with a fortnightly
> schedule.
>
> Thanks
> --jens
>
--
Dr. Henry Nebrensky [log in to unmask]
http://people.brunel.ac.uk/~eesrjjn
"The opossum is a very sophisticated animal.
It doesn't even get up until 5 or 6 p.m."
|