Hi Sam,

AFAIK The dmlite puppet module configures /etc/bdii/bdii.conf, which looks relatively simple [1].

Looking through the log file it looks like the dpm-listspaces is crashing [2], however if I execute the command mentioned in the log

dpm-listspaces --gip --protocols --basedir home --site UKI-SOUTHGRID-BRIS-HEP --glue2 --use-dmlite-conf /etc/dmlite.conf

by hand, I do get a nicely looking response which includes

GLUE2StorageServiceCapacityType: online

GLUE2StorageServiceCapacityTotalSize: 636879

GLUE2StorageServiceCapacityFreeSize: 64853

GLUE2StorageServiceCapacityUsedSize: 572026

GLUE2StorageServiceCapacityReservedSize: 0

compared to

dmlite-shell -e 'qryconf'

POOLS hdfs_pool DEFSIZE 0 GC_START_THRESH 0 GC_STOP_THRESH 0 DEF_LIFETIME 0 DEFPINTIME 0 MAX_LIFETIME 0 MAXPINTIME 0 GROUPS [0] FSS_POLICY GC_POLICY MIG_POLICY RS_POLICY RET_POLICY S_TYPE

17/08/30 14:02:39 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

CAPACITY 579.24TB FREE 58.97TB ( 10.2%)

This looks to me if whatever is executing the script is calling an incompatible java version [3].

Since we have both Java 6 and 7 installed and HDFS only works with Java 7 I tried removing Java 6 - however it tries to pull the DMLite HDFS plugin with it :(.

After checking the java installed I realised that only Java 6 has the -devel package installed - /usr/sbin/alternatives is confused by it!

So, once that was fixed the error message is much shorter:

17/08/30 14:18:49 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable

2017-08-30 14:18:50,419: [DEBUG] Running /var/lib/bdii/gip/provider/se-dpm

send2nsd: NS002 - send error : client_establish_context: Could not find or use a credential

ERROR: unable to open DPM root (/dpm)

2017-08-30 14:18:50,540: [DEBUG] Running /var/lib/bdii/gip/provider/service-srm2.2

/usr/bin/glite-info-service-dpm: line 26: /etc/sysconfig/srmv2.2: No such file or directory

However, it seems that the GLUE information is now published (please confirm).

Cheers,

Luke

[1]

/etc/bdii/bdii.con

BDII_LOG_FILE=/var/log/bdii/bdii-update.log

BDII_LOG_LEVEL=DEBUG

BDII_LDIF_DIR=/var/lib/bdii/gip/ldif

BDII_PROVIDER_DIR=/var/lib/bdii/gip/provider

BDII_PLUGIN_DIR=/var/lib/bdii/gip/plugin

BDII_PORT=2170

BDII_BREATHE_TIME=120

BDII_READ_TIMEOUT=300

BDII_ARCHIVE_SIZE=0

BDII_DELETE_DELAY=0

BDII_USER=ldap

BDII_VAR_DIR=/var/lib/bdii

[2]

2017-08-30 14:04:37,113: [DEBUG] Running /var/lib/bdii/gip/provider/se-dpm

loadFileSystems error:

java.lang.UnsupportedClassVersionError: org/apache/hadoop/fs/FileSystem : Unsupported major.minor version 51.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:648)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)

at java.net.URLClassLoader.access$000(URLClassLoader.java:68)

at java.net.URLClassLoader$1.run(URLClassLoader.java:207)

at java.net.URLClassLoader$1.run(URLClassLoader.java:201)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:200)

at java.lang.ClassLoader.loadClass(ClassLoader.java:325)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)

at java.lang.ClassLoader.loadClass(ClassLoader.java:270)

hdfsBuilderConnect(forceNewInstance=1, nn=jt-37-00.dice.priv, port=0, kerbTicketCachePath=(NULL), userName=dpmmgr) error:

java.lang.UnsupportedClassVersionError: org/apache/hadoop/conf/Configuration : Unsupported major.minor version 51.0

at java.lang.ClassLoader.defineClass1(Native Method)

at java.lang.ClassLoader.defineClass(ClassLoader.java:648)

at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)

at java.net.URLClassLoader.defineClass(URLClassLoader.java:272)

at java.net.URLClassLoader.access$000(URLClassLoader.java:68)

at java.net.URLClassLoader$1.run(URLClassLoader.java:207)

at java.net.URLClassLoader$1.run(URLClassLoader.java:201)

at java.security.AccessController.doPrivileged(Native Method)

at java.net.URLClassLoader.findClass(URLClassLoader.java:200)

at java.lang.ClassLoader.loadClass(ClassLoader.java:325)

at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:296)

at java.lang.ClassLoader.loadClass(ClassLoader.java:270)

Traceback (most recent call last):

File "/usr/bin/dpm-listspaces", line 1801, in <module>

capacity, free = dmlite.getPoolInfo(pool.poolname)

File "/usr/bin/dpm-listspaces", line 1503, in getPoolInfo

self.poolHandler = self.poolDriver.createPoolHandler(poolname)

pydmlite.DmException: [#01.000255] Could not create a HdfsPoolDriver: cannot connect to Hdfs

/var/lib/bdii/gip/provider/se-dpm: line 5: 4288 Aborted dpm-listspaces --gip --protocols --basedir home --site UKI-SOUTHGRID-BRIS-HEP --glue2 --use-dmlite-conf /etc/dmlite.conf

2017-08-30 14:04:38,036: [DEBUG] Running /var/lib/bdii/gip/provider/service-srm2.2

/usr/bin/glite-info-service-dpm: line 26: /etc/sysconfig/srmv2.2: No such file or directory

srmv2.2: unrecognized service

[3]

java -version

java version "1.7.0_151"

OpenJDK Runtime Environment (rhel-2.6.11.0.el6_9-x86_64 u151-b00)

OpenJDK 64-Bit Server VM (build 24.151-b00, mixed mode)

/usr/lib/jvm/jre-1.6.0-openjdk.x86_64/bin/java -version

java version "1.6.0_41"

OpenJDK Runtime Environment (IcedTea6 1.13.13) (rhel-1.13.13.1.el6_8-x86_64)

OpenJDK 64-Bit Server VM (build 23.41-b41, mixed mode)

lrwxrwxrwx 1 root root 46 Aug 16 05:49 java -> /usr/lib/jvm/jre-1.7.0-openjdk.x86_64/bin/java

lrwxrwxrwx 1 root root 48 Aug 16 05:49 java.1.gz -> /usr/share/man/man1/java-java-1.7.0-openjdk.1.gz

lrwxrwxrwx 1 root root 48 Jan 14 2017 javac -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/javac

lrwxrwxrwx 1 root root 49 Jan 14 2017 javac.1.gz -> /usr/share/man/man1/javac-java-1.6.0-openjdk.1.gz

lrwxrwxrwx 1 root root 50 Jan 14 2017 javadoc -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/javadoc

lrwxrwxrwx 1 root root 51 Jan 14 2017 javadoc.1.gz -> /usr/share/man/man1/javadoc-java-1.6.0-openjdk.1.gz

lrwxrwxrwx 1 root root 48 Jan 14 2017 javah -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/javah

lrwxrwxrwx 1 root root 49 Jan 14 2017 javah.1.gz -> /usr/share/man/man1/javah-java-1.6.0-openjdk.1.gz

lrwxrwxrwx 1 root root 48 Jan 14 2017 javap -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64/bin/javap

lrwxrwxrwx 1 root root 49 Jan 14 2017 javap.1.gz -> /usr/share/man/man1/javap-java-1.6.0-openjdk.1.gz

lrwxrwxrwx 1 root root 38 Jan 14 2017 java_sdk -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64

lrwxrwxrwx 1 root root 38 Jan 14 2017 java_sdk_1.6.0 -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64

lrwxrwxrwx 1 root root 46 Jan 14 2017 java_sdk_1.6.0_exports -> /usr/lib/jvm-exports/java-1.6.0-openjdk.x86_64

lrwxrwxrwx 1 root root 46 Jan 14 2017 java_sdk_exports -> /usr/lib/jvm-exports/java-1.6.0-openjdk.x86_64

lrwxrwxrwx 1 root root 38 Jan 14 2017 java_sdk_openjdk -> /usr/lib/jvm/java-1.6.0-openjdk.x86_64

lrwxrwxrwx 1 root root 46 Jan 14 2017 java_sdk_openjdk_exports -> /usr/lib/jvm-exports/java-1.6.0-openjdk.x86_64

**************************************************
Dr Lukasz Kreczko
Research Associate

Department of Physics

Particle Physics Group

University of Bristol
HH Wills Physics Lab
University of Bristol
Tyndall Avenue
Bristol
BS8 1TL

+44 (0)117 928 8724

[log in to unmask]

A top 3 UK university with leading employers (2017)

A top 5 UK university for research (2014 REF)

A world top 50 university (QS Ranking 2016)

**************************************************

(Specifically, it would be useful to try executing the two provider scripts in /var/lib/bdii/gip/providers/ and see what they output.)

On Wed, Aug 30, 2017 at 12:52 PM Sam Skipsey <[log in to unmask]> wrote:

Hi Winnie:

So, the BDII stuff is configured by

/etc/bdii/bdii.conf

which points (usually) at

/var/lib/bdii/gip

as the base path for stuff.

On my SE, I see in that path:

glite-info-service-srm2.2.conf

and the usual plugins, providers, ldif directories

plugins is empty

ldif contains a basic default.ldif

providers contains two scripts:

se-dpm service-srm2.2

Obviously, as you don't have an SRM, I would guess you only should have se-dpm?

Our se-dpm does what I expected it to (we discussed this in the Storage Group meeting this morning), and runs dpm-listspaces in --glue2 mode:

export X509_USER_CERT=/var/lib/ldap/hostcert.pem

export X509_USER_KEY=/var/lib/ldap/hostkey.pem

dpm-listspaces --gip --protocols --basedir home --site UKI-SCOTGRID-GLASGOW --glue2

Can you check which of these things exist on your SE?

Sam

On Wed, Aug 23, 2017 at 10:43 AM Winnie Lacesso <[log in to unmask]> wrote:

Good day all,

Not a meeting today but can request SE help anyway.

Bristol SE doesna speak SRM. It is publishing a GlueService object, but main
GlueSE missing. (CAVEAT: Sadly, my bdii/GLUE knowhow = .00001)

I asked Stephen Burke (GLUE Guru) about this, he said:

> In GLUE 1 your SE is publishing a GlueService object, but the main GlueSE
> is missing:
>
>ldapsearch -x -h lcgbdii02.phy.bris.ac.uk -p 2170 -b o=grid objectclass=GlueSE
># extended LDIF
>#
># LDAPv3
># base <o=grid> with scope subtree
># filter: objectclass=GlueSE
># requesting: ALL
>#
>
># search result
>search: 2
>result: 0 Success
>
># numResponses: 1
>
>Similarly in GLUE 2:
>
>ldapsearch -x -h lcgbdii02.phy.bris.ac.uk -p 2170 -b o=glue objectclass=GLUE2StorageService
># extended LDIF
>#
># LDAPv3
># base <o=glue> with scope subtree
># filter: objectclass=GLUE2StorageService
># requesting: ALL
>#
>
># search result
>search: 2
>result: 0 Success
>
># numResponses: 1
>
> The publishing isn't intrinsically connected to whether it's an SRM or
> not, but if no-one has written an info provider for it then obviously
> it won't be visible.

Possibly relevant installed:

apr-util-ldap-1.3.9-3.el6_0.1.x86_64.rpm
bdii-5.2.23-1.el6.noarch.rpm
openldap-2.4.40-6.el6_7.x86_64.rpm
openldap-clients-2.4.40-6.el6_7.x86_64.rpm
openldap-servers-2.4.40-6.el6_7.x86_64.rpm
perl-LDAP-0.40-1.el6.noarch.rpm
python-ldap-2.3.10-1.el6.x86_64.rpm

also possibly relevant installed:

dmlite-plugins-hdfs-0.7.8-1.20161117.1205rc.el6.x86_64.rpm
dmlite-plugins-memcache-0.8.5-1.el6.x86_64.rpm
dmlite-plugins-mysql-0.8.5-1.el6.x86_64.rpm
gstreamer-plugins-base-0.10.29-2.el6.x86_64.rpm
hadoop-0.20-mapreduce-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-client-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-hdfs-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-libhdfs-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-mapreduce-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
hadoop-yarn-2.6.0+cdh5.7.1+1335-1.cdh5.7.1.p0.15.el6.x86_64.rpm
perl-Module-Pluggable-3.90-141.el6.x86_64.rpm
xrootd-server-atlas-n2n-plugin-2.2-0.el6.x86_64.rpm

in /etc/bdii:
-rw-r----- 1 ldap ldap 3720 Jun 8 2016 bdii-slapd.conf
-rw-r--r-- 1 root root 329 Mar 18 2015 bdii.conf
drwxr-xr-x 2 root root 4096 Mar 18 2015 gip/
-rw-r----- 1 ldap ldap 4012 Mar 18 2015 bdii-top-slapd.conf
-rw-r--r-- 1 root root 3970 Oct 13 2014 BDII.schema
-rw-r--r-- 1 root root 276 Oct 13 2014 DB_CONFIG
-rw-r--r-- 1 root root 510 Oct 13 2014 DB_CONFIG_top

gip is empty.

In /var/lib/bdii:
-rw-r--r-- 1 ldap ldap 0 Dec 15 2016 add.err
-rw-r--r-- 1 ldap ldap 52 Aug 23 10:36 add.ldif
drwxr-xr-x 2 ldap ldap 4096 Aug 23 10:36 archive/
drwxr-xr-x 5 ldap ldap 100 Dec 15 2016 db/
-rw-r--r-- 1 ldap ldap 0 Sep 30 2016 delete.err
-rw-r--r-- 1 ldap ldap 0 Aug 23 10:36 delete.ldif
drwxr-xr-x 5 ldap ldap 4096 Mar 18 2015 gip/
-rw-r--r-- 1 ldap ldap 0 Aug 23 10:36 modify.err
-rw-r--r-- 1 ldap ldap 82391 Aug 23 10:36 modify.ldif
-rw-r--r-- 1 ldap ldap 58419 Aug 23 10:36 new.ldif
-rw-r--r-- 1 ldap ldap 0 Aug 23 10:36 old.err
-rw-r--r-- 1 ldap ldap 2785 Aug 23 10:36 old.ldif
root@lcgse01> ls -lR gip

ls -lR gip:
total 16
-rwxr-xr-x 1 ldap ldap 648 Mar 18 2015 glite-info-service-srm2.2.conf*
drwxr-xr-x 2 ldap ldap 4096 Mar 18 2015 ldif/
drwxr-xr-x 2 ldap ldap 4096 Oct 13 2014 plugin/
drwxr-xr-x 2 ldap ldap 4096 Oct 27 2015 provider/

gip/ldif:
total 4
-rw-r--r-- 1 ldap ldap 439 Oct 13 2014 default.ldif

gip/plugin:
total 0

gip/provider:
total 8
-rwxr-xr-x 1 root root 287 Oct 27 2015 se-dpm*
-rwxr-xr-x 1 ldap ldap 146 Mar 18 2015 service-srm2.2*

Can anyone advise how to have this dmlite-hdfs SE publish
objectclass=GlueSE in GLUE 1 & objectclass=GLUE2StorageService in GLUE 2?