Hi Patrick,
This is what we open at Glasgow:
- name: Enable TCP on External port(s)
firewalld:
zone: external
port: "{{ item }}"
permanent: yes
state: enabled
with_items:
- 443/tcp
- 2135/tcp
- 2811/tcp
- 9000-9300/tcp
- 9000-9300/udp
notify: Reload firewalld
Thanks,
Gareth
________________________________
From: Testbed Support for GridPP member institutes <[log in to unmask]> on behalf of Patrick Smith <[log in to unmask]>
Sent: 02 December 2019 10:00
To: [log in to unmask] <[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Gareth,
Thanks for getting back to me.
Even though I have tried multiple times to force openldap to install from the correct repo I believe it is still installing from the wrong repo and therefore misconfigured. I have now unistalled it removed repo files and re-installed. Finally slapd, arc-arex and arc-infosys-ldap now start.
Can you confirm which firewall ports need to be open on our perimeter firewall for our ARC CE to receive jobs from outside please? Alessandra said 443 or 2135 (for ldap) but are there any other ports required?
[root@grid-arc-01 ~]# rpm -qa |grep openldap
openldap-clients-2.4.44-21.el7_6.x86_64
openldap-servers-2.4.44-21.el7_6.x86_64
openldap-devel-2.4.44-21.el7_6.x86_64
openldap-2.4.44-21.el7_6.x86_64
[root@grid-arc-01 ~]# rpm -qa |grep nordugrid
nordugrid-arc-client-6.4.0-1.el7.x86_64
nordugrid-arc-hed-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-xrootd-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-needed-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-gfal-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-arcrest-6.4.0-1.el7.x86_64
nordugrid-arc-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-globus-6.4.0-1.el7.x86_64
nordugrid-arc-gridftpd-6.4.0-1.el7.x86_64
nordugrid-arc-plugins-s3-6.4.0-1.el7.x86_64
nordugrid-release-6-1.el7.noarch
nordugrid-arc-arex-6.4.0-1.el7.x86_64
nordugrid-arc-infosys-ldap-6.4.1-1.el7.noarch
nordugrid-arc-plugins-internal-6.4.0-1.el7.x86_64
----------------------------------------------------------------------------------------------------------------
[root@grid-arc-01 ~]# systemctl status slapd
● slapd.service - OpenLDAP Server Daemon
Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-12-02 09:39:44 GMT; 2s ago
Docs: man:slapd
man:slapd-config
man:slapd-hdb
man:slapd-mdb
file:///usr/share/doc/openldap-servers/guide.html
Process: 3823 ExecStart=/usr/sbin/slapd -u ldap -h ${SLAPD_URLS} $SLAPD_OPTIONS (code=exited, status=0/SUCCESS)
Process: 3786 ExecStartPre=/usr/libexec/openldap/check-config.sh (code=exited, status=0/SUCCESS)
Main PID: 3826 (slapd)
Tasks: 2
CGroup: /system.slice/slapd.service
└─3826 /usr/sbin/slapd -u ldap -h ldapi:/// ldap:///
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk runuser[3816]: pam_unix(runuser:session): session closed for user ldap
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk runuser[3818]: pam_unix(runuser:session): session opened for user ldap by (uid=0)
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk runuser[3818]: pam_unix(runuser:session): session closed for user ldap
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk runuser[3821]: pam_unix(runuser:session): session opened for user ldap by (uid=0)
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk runuser[3821]: pam_unix(runuser:session): session closed for user ldap
Dec 02 09:39:39 grid-arc-01.hpc.susx.ac.uk slapd[3823]: @(#) $OpenLDAP: slapd 2.4.44 (Jan 29 2019 17:42:45) $
[log in to unmask]:/builddir/build/BUILD/openldap-2.4.44/openldap-2.4.44/servers/slapd
Dec 02 09:39:44 grid-arc-01.hpc.susx.ac.uk slapd[3823]: tlsmc_get_pin: INFO: Please note the extracted key file will not be protected with a PIN any more...issions.
Dec 02 09:39:44 grid-arc-01.hpc.susx.ac.uk slapd[3826]: hdb_db_open: warning - no DB_CONFIG file found in directory /var/lib/ldap: (2).
Expect poor performance for suffix "dc=my-domain,dc=com".
Dec 02 09:39:44 grid-arc-01.hpc.susx.ac.uk slapd[3826]: slapd starting
Dec 02 09:39:44 grid-arc-01.hpc.susx.ac.uk systemd[1]: Started OpenLDAP Server Daemon.
----------------------------------------------------------------------------------------------------------------
root@grid-arc-01 ~]# systemctl status arc-arex
● arc-arex.service - ARC Resource-coupled EXecution service
Loaded: loaded (/usr/lib/systemd/system/arc-arex.service; enabled; vendor preset: disabled)
Active: active (running) since Fri 2019-11-29 15:40:27 GMT; 2 days ago
Main PID: 9982 (arched)
Status: "Processing requests..."
Tasks: 14
CGroup: /system.slice/arc-arex.service
├─4268 /bin/sh /usr/share/arc/scan-fork-job --config /var/run/arched-arex.cfg /var/spool/arc/jobstatus
├─4280 sleep 10
└─9982 /usr/sbin/arched -c /tmp/arex.xml.zMomlo
Nov 29 15:40:26 grid-arc-01.hpc.susx.ac.uk systemd[1]: Starting ARC Resource-coupled EXecution service...
Nov 29 15:40:27 grid-arc-01.hpc.susx.ac.uk systemd[1]: Started ARC Resource-coupled EXecution service.
----------------------------------------------------------------------------------------------------------------
[root@grid-arc-01 ~]# systemctl status arc-infosys-ldap
● arc-infosys-ldap.service - ARC LDAP-based information services - BDII-Update
Loaded: loaded (/usr/lib/systemd/system/arc-infosys-ldap.service; enabled; vendor preset: disabled)
Active: active (running) since Mon 2019-12-02 09:40:47 GMT; 12min ago
Process: 4257 ExecStartPost=/bin/sh /var/run/arc/infosys/bdii-update-post.cmd (code=exited, status=0/SUCCESS)
Process: 4252 ExecStart=/bin/sh /var/run/arc/infosys/bdii-update.cmd (code=exited, status=0/SUCCESS)
Process: 4198 ExecStartPre=/usr/share/arc/create-bdii-config (code=exited, status=0/SUCCESS)
Main PID: 4256 (bdii-update)
Tasks: 1
CGroup: /system.slice/arc-infosys-ldap.service
└─4256 /usr/bin/python2 /usr/sbin/bdii-update -c /var/run/arc/infosys/bdii.conf -d
Dec 02 09:40:46 grid-arc-01.hpc.susx.ac.uk systemd[1]: Starting ARC LDAP-based information services - BDII-Update...
Dec 02 09:40:46 grid-arc-01.hpc.susx.ac.uk runuser[4252]: pam_unix(runuser:session): session opened for user ldap by (uid=0)
Dec 02 09:40:47 grid-arc-01.hpc.susx.ac.uk systemd[1]: Started ARC LDAP-based information services - BDII-Update.
Thanks
Patrick
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Gareth Roy [[log in to unmask]]
Sent: 02 December 2019 09:18
To: [log in to unmask]
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Patrick,
See below, all the openldap stuff is from the base CentOS7 repo, along with updates. Do you have the nordugrid ldap stuff installed? I've listed all the nordugrid packages as well.
Thanks,
Gareth
[root@ce02 ~]# rpm -qa |grep openldap
openldap-2.4.44-21.el7_6.x86_64
openldap-clients-2.4.44-21.el7_6.x86_64
openldap-servers-2.4.44-21.el7_6.x86_64
[root@ce02 ~]# rpm -qa |grep nordugrid
nordugrid-arc-plugins-internal-6.4.1-1.el7.x86_64
nordugrid-arc-6.4.1-1.el7.x86_64
nordugrid-arc-plugins-s3-6.4.1-1.el7.x86_64
nordugrid-arc-hed-6.4.1-1.el7.x86_64
nordugrid-arc-gridftpd-6.4.1-1.el7.x86_64
nordugrid-arc-infosys-ldap-6.4.1-1.el7.noarch
nordugrid-release-6-1.el7.noarch
nordugrid-arc-plugins-needed-6.4.1-1.el7.x86_64
nordugrid-arc-plugins-arcrest-6.4.1-1.el7.x86_64
nordugrid-arc-plugins-xrootd-6.4.1-1.el7.x86_64
nordugrid-arc-plugins-globus-6.4.1-1.el7.x86_64
nordugrid-arc-plugins-gfal-6.4.1-1.el7.x86_64
nordugrid-arc-arex-6.4.1-1.el7.x86_64
Also this is what we have for the slapd service, which for some reason looks different from yours.
[root@ce02 ~]# cat /usr/lib/systemd/system/slapd.service
[Unit]
Description=OpenLDAP Server Daemon
After=syslog.target network-online.target
Documentation=man:slapd
Documentation=man:slapd-config
Documentation=man:slapd-hdb
Documentation=man:slapd-mdb
Documentation=file:///usr/share/doc/openldap-servers/guide.html
[Service]
Type=forking
PIDFile=/var/run/openldap/slapd.pid
Environment="SLAPD_URLS=ldap:/// ldapi:///" "SLAPD_OPTIONS="
EnvironmentFile=/etc/sysconfig/slapd
ExecStartPre=/usr/libexec/openldap/check-config.sh
ExecStart=/usr/sbin/slapd -u ldap -h ${SLAPD_URLS} $SLAPD_OPTIONS
[Install]
WantedBy=multi-user.target
________________________________
From: Testbed Support for GridPP member institutes <[log in to unmask]> on behalf of Patrick Smith <[log in to unmask]>
Sent: 29 November 2019 17:23
To: [log in to unmask] <[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Gareth,
I had SELinux set to permissive but since turned it off.
Could you let me know which version of openlap, openlap-clients and openldap-servers you have and which repo they were installed from please?
Thanks
Patrick
-------- Original message --------
From: Gareth Roy <[log in to unmask]>
Date: 29/11/2019 15:49 (GMT+00:00)
To: [log in to unmask]
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Patrick,
Do you have SELinux on and enforcing? If so you may find Grid software _really_ doesn’t like that… no ones done the work to provide SELinux profiles.
Check /etc/selinux/config is set to SELINUX=permissive
Thanks,
Gareth
From: Testbed Support for GridPP member institutes <[log in to unmask]> On Behalf Of Patrick Smith
Sent: 29 November 2019 15:46
To: [log in to unmask]
Subject: Re: ARC CE6/LCMAPS/BDII
Thanks Gareth,
I have removed the [grid-infosys] section and change lrms=sge to fork but still can't start slapd, get the same error
Nov 29 15:41:08 grid-arc-01.hpc.susx.ac.uk polkitd[955]: Unregistered Authentication Agent for unix-process:10424:8820672 (system bus
[root@grid-arc-01 ~]# rpm -qa openldap
openldap-2.4.44-21.el7_6.x86_64
[root@grid-arc-01 ~]# rpm -qa openldap-servers
openldap-servers-2.4.45-172_cm8.1.x86_64
[root@grid-arc-01 ~]# rpm -qa openldap-clients
openldap-clients-2.4.44-21.el7_6.x86_64
[root@grid-arc-01 ~]# rpm -qa bdii
bdii-5.2.25-2.el7.noarch
Thanks
Patrick
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Gareth Roy [[log in to unmask]]
Sent: 29 November 2019 15:37
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Okay,
Two things, I don’t think you need to have the [grid-infosys] section as I believe that is an ARC5 thing not ARC6… can’t see that in the manual list of stanzas:
http://www.nordugrid.org/arc/arc6/admins/reference.html
Second that looks like it’s breaking trying to query the SGE batch farm for job information, I’m assuming you’re using SGE…. What happens of you try just using a simple fork queue rather than SGE.
When we set it up the first time here we got jobs running as fork on the CE to work out kinks with Auth/Access and then got it talking to HTCondor.
Thanks,
Gareth
From: Testbed Support for GridPP member institutes <[log in to unmask]<mailto:[log in to unmask]>> On Behalf Of Patrick Smith
Sent: 29 November 2019 15:21
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Gareth,
The output of /var/log/arc/infoprovider.log
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] ############## A-REX infoprovider started ##############
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{delegationdb} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{delegationdb} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{fixdirectories} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{fixdirectories} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{helperlog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{helperlog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{joblog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{joblog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{mail} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{mail} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{maxrerun} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{maxrerun} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{norootpower} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{norootpower} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{pidfile} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{pidfile} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{tmpdir} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{tmpdir} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{user} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{user} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{watchdog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{watchdog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{infosys}{glue1}{enabled} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{gnu_time} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{gnu_time} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{slurm_use_sacct} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{slurm_use_sacct} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{xenvs}{atlas}{NodeSelection} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{xenvs}{gridpp}{NodeSelection} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] Some required config options are missing or not used by infosys
[2019-11-29 15:08:57] [CEInfo] [VERBOSE] [30796] Time spent in ConfigCentral: 0.045s
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Start data collection...
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Reading grid-mapfiles
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] gridmap not configured
[2019-11-29 15:08:57] [CEInfo] [WARNING] [30796] Cannot determine local users
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Fetching job information from control directory (GMJobsInfo.pm)
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/accepting
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/processing
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/finished
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Number of jobs to scan: 0 ; Number of jobs skipped: 0
[2019-11-29 15:08:57] [CEInfo] [VERBOSE] [30796] Time spent in GMJobsInfo: 0.000s
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Updating job status information
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Updating frontend information (HostInfo.pm)
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] Time spent collecting certificates info: 1.414s
[2019-11-29 15:08:59] [Sysinfo] [WARNING] [30796] No such directory: /var/spool/arc/sessiondir
[2019-11-29 15:08:59] [HostInfo] [WARNING] [30796] Failed checking disk space available in session directories
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{EMIversion} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{cache_free} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{cache_total} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{session_free} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{session_total} is missing
[2019-11-29 15:08:59] [CEInfo] [VERBOSE] [30796] Time spent in HostInfo: 1.469s
[2019-11-29 15:08:59] [CEInfo] [INFO] [30796] Updating RTE information (RTEInfo.pm)
[2019-11-29 15:08:59] [RTEInfo] [DEBUG] [30796] Can't access runtimedir: /var/spool/arc/jobstatus/rte/enabled/: No such file or directory
[2019-11-29 15:08:59] [CEInfo] [VERBOSE] [30796] Time spent in RTEInfo: 0.000s
[2019-11-29 15:08:59] [CEInfo] [INFO] [30796] Updating LRMS information (LRMSInfo.pm)
[2019-11-29 15:08:59] [SGEmod] [ERROR] [30796] SGE executables not found
The output of /var/log/arc/bdii/bdii-update.log
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] ############## A-REX infoprovider started ##############
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{delegationdb} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{delegationdb} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{fixdirectories} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{fixdirectories} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{helperlog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{helperlog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{joblog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{joblog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{mail} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{mail} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{maxrerun} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{maxrerun} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{norootpower} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{norootpower} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{pidfile} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{pidfile} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{tmpdir} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{tmpdir} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{user} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{user} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{watchdog} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{arex}{watchdog} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{infosys}{glue1}{enabled} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{gnu_time} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{gnu_time} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{slurm_use_sacct} is not recognized or not used by infoproviders
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{lrms}{slurm_use_sacct} deleting it
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{xenvs}{atlas}{NodeSelection} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] config key config->{xenvs}{gridpp}{NodeSelection} is missing
[2019-11-29 15:08:57] [ConfigCentral] [VERBOSE] [30796] Some required config options are missing or not used by infosys
[2019-11-29 15:08:57] [CEInfo] [VERBOSE] [30796] Time spent in ConfigCentral: 0.045s
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Start data collection...
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Reading grid-mapfiles
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] gridmap not configured
[2019-11-29 15:08:57] [CEInfo] [WARNING] [30796] Cannot determine local users
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Fetching job information from control directory (GMJobsInfo.pm)
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/accepting
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/processing
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Found 0 jobs in /var/spool/arc/jobstatus/finished
[2019-11-29 15:08:57] [GMJobsInfo] [VERBOSE] [30796] Number of jobs to scan: 0 ; Number of jobs skipped: 0
[2019-11-29 15:08:57] [CEInfo] [VERBOSE] [30796] Time spent in GMJobsInfo: 0.000s
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Updating job status information
[2019-11-29 15:08:57] [CEInfo] [INFO] [30796] Updating frontend information (HostInfo.pm)
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] Time spent collecting certificates info: 1.414s
[2019-11-29 15:08:59] [Sysinfo] [WARNING] [30796] No such directory: /var/spool/arc/sessiondir
[2019-11-29 15:08:59] [HostInfo] [WARNING] [30796] Failed checking disk space available in session directories
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{EMIversion} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{cache_free} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{cache_total} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{session_free} is missing
[2019-11-29 15:08:59] [HostInfo] [DEBUG] [30796] SelfCheck: result key hostinfo->{session_total} is missing
[2019-11-29 15:08:59] [CEInfo] [VERBOSE] [30796] Time spent in HostInfo: 1.469s
[2019-11-29 15:08:59] [CEInfo] [INFO] [30796] Updating RTE information (RTEInfo.pm)
[2019-11-29 15:08:59] [RTEInfo] [DEBUG] [30796] Can't access runtimedir: /var/spool/arc/jobstatus/rte/enabled/: No such file or directory
[2019-11-29 15:08:59] [CEInfo] [VERBOSE] [30796] Time spent in RTEInfo: 0.000s
[2019-11-29 15:08:59] [CEInfo] [INFO] [30796] Updating LRMS information (LRMSInfo.pm)
[2019-11-29 15:08:59] [SGEmod] [ERROR] [30796] SGE executables not found
[root@grid-arc-01 ~]# tail /var/log/arc/bdii/bdii-update.log
2019-11-29 15:12:44,973: [INFO] FailedAdds: 0
2019-11-29 15:12:44,973: [INFO] FailedModifies: 0
2019-11-29 15:12:44,974: [INFO] TotalEntries: 4
2019-11-29 15:12:44,974: [INFO] QueryTime: 0
2019-11-29 15:12:44,974: [INFO] NewEntries: 0
2019-11-29 15:12:44,974: [INFO] DBUpdateTime: 0
2019-11-29 15:12:44,974: [INFO] ReadTime: 0
2019-11-29 15:12:44,974: [INFO] PluginsTime: 0
2019-11-29 15:12:44,974: [INFO] ProvidersTime: 0
2019-11-29 15:12:44,987: [INFO] Sleeping for 10 seconds
My /etc/arc.conf for [infosys]
[infosys]
loglevel = INFO
[grid-infosys]
logfile=/var/log/arc/grid-manager.log
[infosys/ldap]
bdii_debug_level = INFO
[infosys/nordugrid]
[infosys/glue2]
admindomain_name = UKI-SOUTHGRID-SUSX
[infosys/glue2/ldap]
#user=slapd
#slapd=/usr/lib/systemd/system/slapd
#infosys_ldap_run_dir=/var/run/arc/infosys
#ldap_schema_dir=/etc/ladap/schema/
[infosys/cluster]
advertisedvo = ops
advertisedvo = dteam
advertisedvo = atlas
alias = SouthGrid Susx
hostname = grid-arc-01.hpc.susx.ac.uk
cluster_location = UK-BN19RH
cluster_owner = University_of_Sussex
clustersupport = [log in to unmask]<mailto:[log in to unmask]>
nodememory = 6000
defaultmemory = 2048
nodeaccess = outbound
Thanks
Patrick
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Gareth Roy [[log in to unmask]]
Sent: 29 November 2019 15:08
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Patrick,
Okay if your ldap does not work then there may be issues doing job submission with grid interfaces.
Interestingly you slapd.service doesn’t match what we have here on a freshly installed ARC 6.4, what does:
/var/log/arc/infoprovider.log
/var/log/arc/bdii/bdii-update.log
have to say? You should see some logging… you can also up the verbosity:
[infosys]
loglevel = INFO
[infosys/ldap]
Bdii_debug_loglevel = INFO
And then restart all the services
arcctl service restart -a
Thanks
Gareth
From: Testbed Support for GridPP member institutes <[log in to unmask]<mailto:[log in to unmask]>> On Behalf Of Patrick Smith
Sent: 29 November 2019 14:47
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Gareth,
Okay thanks, yes the IPv6 is by design. The result of the command is:
[pjs32@grid-arc-01 ~]$ ldapsearch -x -H ldap:// grid-arc-01.hpc.susx.ac.uk:2135 -b ‘o=Grid’
ldap_sasl_bind(SIMPLE): Can't contact LDAP server (-1)
I've been having an issue with the openldap not starting but can't work out why.
[pjs32@grid-arc-01 ~]$ systemctl status -l slapd
● slapd.service - OpenLDAP Server Daemon
Loaded: loaded (/usr/lib/systemd/system/slapd.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Thu 2019-11-28 16:11:37 GMT; 22h ago
Nov 28 16:11:35 grid-arc-01.hpc.susx.ac.uk systemd[1]: Starting OpenLDAP Server Daemon...
Nov 28 16:11:36 grid-arc-01.hpc.susx.ac.uk slapd[1846]: @(#) $OpenLDAP: slapd 2.4.45 (Dec 10 2017 23:15:45) $
root@0b51f3d31da0:/root/rpmbuild/BUILD/openldap-2.4.45/servers/slapd<mailto:root@0b51f3d31da0:/root/rpmbuild/BUILD/openldap-2.4.45/servers/slapd>
Nov 28 16:11:37 grid-arc-01.hpc.susx.ac.uk systemd[1]: slapd.service: control process exited, code=exited status=1
Nov 28 16:11:37 grid-arc-01.hpc.susx.ac.uk systemd[1]: Failed to start OpenLDAP Server Daemon.
Nov 28 16:11:37 grid-arc-01.hpc.susx.ac.uk systemd[1]: Unit slapd.service entered failed state.
Nov 28 16:11:37 grid-arc-01.hpc.susx.ac.uk systemd[1]: slapd.service failed.
----------------------------------------------------------------------------------------------------------------
[pjs32@grid-arc-01 ~]$ journalctl -xe
Nov 29 14:30:25 grid-arc-01.hpc.susx.ac.uk polkitd[955]: Registered Authentication Agent for unix-process:18426:8396360 (system bus name :1.99 [/usr/bin/pkttyagent --notify-fd 5 --fallback], object path /org/freedesktop/PolicyKit1/Authent
Nov 29 14:30:26 grid-arc-01.hpc.susx.ac.uk polkitd[955]: Unregistered Authentication Agent for unix-process:18426:8396360 (system bus name :1.99, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_GB.UTF-8) (disconnect
----------------------------------------------------------------------------------------------------------------
/usr/lib/systemd/system/slapd.service
[Unit]
Description=OpenLDAP Server Daemon
After=syslog.target network.target
[Service]
LimitMEMLOCK=infinity
LimitNOFILE=16384
LimitSTACK=infinity
Type=forking
PIDFile=/var/run/openldap/slapd.pid
Environment="SLAPD_URLS=ldaps:/// ldapi:///" "SLAPD_OPTIONS="
EnvironmentFile=/etc/sysconfig/slapd
ExecStart=/usr/sbin/slapd -u ldap -h ${SLAPD_URLS} $SLAPD_OPTIONS
[Install]
WantedBy=multi-user.target
----------------------------------------------------------------------------------------------------------------
Regards
Patrick
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Gareth Roy [[log in to unmask]]
Sent: 29 November 2019 14:13
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Patrick,
The empty interface is fine, it’s because it hasn’t decided on what type of connection it wants, it then creates a bunch LDAP, WS, EMI-ES to test against… it still thinks the secure connection is wrong, not sure why but I note it always appears to be talking to an IPv6 endpoint… is that by design?
What happens when you do a:
ldapsearch -x -H ldap:// grid-arc-01.hpc.susx.ac.uk:2135 -b ‘o=Grid’
You should be able to see that endpoint publicly but I can’t from here.
Thanks,
Gareth
From: Testbed Support for GridPP member institutes <[log in to unmask]<mailto:[log in to unmask]>> On Behalf Of Patrick Smith
Sent: 29 November 2019 13:41
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Gareth,
Thank you for responding. The results of the 'arcinfo -d DEBUG -c grid-arc-01.hpc.susx.ac.uk' command are below.
Not sure if it is the problem but I can see 'setting endpoint.. <empty InterfaceName>' several times so do I need to somehow binf the endpoint to an interface?
----------------------------------------------------------------------------------------------------------------
My networking/routing are:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:3f:0f:43 brd ff:ff:ff:ff:ff:ff
inet 139.184.80.44/23 brd 139.184.81.255 scope global noprefixroute dynamic eth0
valid_lft 38096sec preferred_lft 38096sec
inet6 2001:678:718:80::80:44/64 scope global noprefixroute
valid_lft forever preferred_lft forever
inet6 fe80::5054:ff:fe3f:f43/64 scope link noprefixroute
valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
link/ether 52:54:00:88:e2:36 brd ff:ff:ff:ff:ff:ff
inet 10.141.161.233/16 brd 10.141.255.255 scope global noprefixroute dynamic eth1
valid_lft 66235sec preferred_lft 66235sec
inet6 fe80::e688:20a6:a8bc:6efe/64 scope link noprefixroute
valid_lft forever preferred_lft forever
----------------------------------------------------------------------------------------------------------------
default via 139.184.80.1 dev eth0 proto dhcp metric 100
default via 10.141.255.254 dev eth1 proto dhcp metric 101
10.141.0.0/16 dev eth1 proto kernel scope link src 10.141.161.233 metric 101
10.141.0.0/16 via 10.141.0.254 dev eth1 proto static metric 101
139.184.80.0/23 dev eth0 proto kernel scope link src 139.184.80.44 metric 100
----------------------------------------------------------------------------------------------------------------
VERBOSE: Running command: arcinfo -d DEBUG -c grid-arc-01.hpc.susx.ac.uk
DEBUG: Loading configuration (/etc/arc/client.conf)
INFO: Configuration (/etc/arc/client.conf) loaded
DEBUG: Loading configuration (/home/pjs32/.arc/client.conf)
INFO: Configuration (/home/pjs32/.arc/client.conf) loaded
INFO: Using proxy file: /tmp/x509up_u1000
INFO: Using certificate file: /home/pjs32/.arc/usercert.pem
INFO: Using key file: /home/pjs32/.arc/userkey.pem
INFO: Using CA certificate directory: /etc/grid-security/certificates
DEBUG: Module Manager Init
DEBUG: Module Manager Init
DEBUG: Loaded /usr/lib64/arc/libaccARCHERY.so
DEBUG: Loaded /usr/lib64/arc/libaccLDAP.so
DEBUG: Loaded HED:ServiceEndpointRetrieverPlugin ARCHERY
DEBUG: Loaded HED:ServiceEndpointRetrieverPlugin EGIIS
DEBUG: Module Manager Init
DEBUG: Module Manager Init
DEBUG: Loaded /usr/lib64/arc/libaccINTERNAL.so
DEBUG: accINTERNAL made persistent
DEBUG: Loaded /usr/lib64/arc/libaccARCREST.so
DEBUG: Loaded /usr/lib64/arc/libaccEMIES.so
DEBUG: Loaded /usr/lib64/arc/libaccLDAP.so
DEBUG: Loaded HED:TargetInformationRetrieverPlugin INTERNAL
DEBUG: Loaded HED:TargetInformationRetrieverPlugin REST
DEBUG: Loaded HED:TargetInformationRetrieverPlugin EMIES
DEBUG: Loaded HED:TargetInformationRetrieverPlugin LDAPGLUE2
DEBUG: Loaded HED:TargetInformationRetrieverPlugin LDAPNG
DEBUG: Adding endpoint (grid-arc-01.hpc.susx.ac.uk) to TargetInformationRetriever
DEBUG: Setting status (STARTED) for endpoint: grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource)
DEBUG: Starting thread to query the endpoint on grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource)
DEBUG: The interface of this endpoint (grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource)) is unspecified, will try all possible plugins
DEBUG: Found HED:TargetInformationRetrieverPlugin INTERNAL (it was loaded already)
DEBUG: The endpoint (grid-arc-01.hpc.susx.ac.uk) is not supported by this plugin (INTERNAL)
DEBUG: Found HED:TargetInformationRetrieverPlugin REST (it was loaded already)
DEBUG: New endpoint is created (grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)) from the one with the unspecified interface (grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource))
DEBUG: Setting status (STARTED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)
DEBUG: Starting sub-thread to query the endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)
DEBUG: Found HED:TargetInformationRetrieverPlugin EMIES (it was loaded already)
DEBUG: New endpoint is created (grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)) from the one with the unspecified interface (grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource))
DEBUG: Setting status (STARTED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)
DEBUG: Starting sub-thread to query the endpoint on grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)
DEBUG: Found HED:TargetInformationRetrieverPlugin REST (it was loaded already)
DEBUG: Calling plugin REST to query endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)
DEBUG: Querying WSRF GLUE2 computing info endpoint.
DEBUG: Found HED:TargetInformationRetrieverPlugin LDAPGLUE2 (it was loaded already)
DEBUG: Setting status (STARTED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapglue2, capabilities: information.discovery.resource)
DEBUG: Starting sub-thread to query the endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapglue2, capabilities: information.discovery.resource)
DEBUG: Found HED:TargetInformationRetrieverPlugin EMIES (it was loaded already)
DEBUG: Calling plugin EMIES to query endpoint on grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)
DEBUG: Found HED:TargetInformationRetrieverPlugin LDAPNG (it was loaded already)
DEBUG: Module Manager Init
DEBUG: New endpoint is created (grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)) from the one with the unspecified interface (grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource))
DEBUG: Setting status (STARTED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)
DEBUG: Collecting EMI-ES GLUE2 computing info endpoint information.
DEBUG: Found HED:TargetInformationRetrieverPlugin LDAPGLUE2 (it was loaded already)
DEBUG: Calling plugin LDAPGLUE2 to query endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapglue2, capabilities: information.discovery.resource)
DEBUG: Loaded /usr/lib64/arc/libmcchttp.so
DEBUG: Starting sub-thread to query the endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)
DEBUG: Module Manager Init
DEBUG: Creating an EMI ES client
DEBUG: Found HED:TargetInformationRetrieverPlugin LDAPNG (it was loaded already)
DEBUG: Calling plugin LDAPNG to query endpoint on grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)
VERBOSE: Creating and sending service information request to https://grid-arc-01.hpc.susx.ac.uk:443/arex
VERBOSE: Processing a esrinfo:GetResourceInfo request
DEBUG: Loaded /usr/lib64/arc/libmccmsgvalidator.so
DEBUG: Module Manager Init
DEBUG: Loaded /usr/lib64/arc/libdmcs3.so
DEBUG: Loaded /usr/lib64/arc/libmccsoap.so
DEBUG: Loaded /usr/lib64/arc/libmcchttp.so
DEBUG: Loaded /usr/lib64/arc/libmcctcp.so
DEBUG: Loaded /usr/lib64/arc/libdmcgridftpdeleg.so
DEBUG: Loaded /usr/lib64/arc/libdmcgfaldeleg.so
DEBUG: Loaded /usr/lib64/arc/libmcctls.so
DEBUG: Loaded /usr/lib64/arc/libmccmsgvalidator.so
DEBUG: Certificate format is PEM
DEBUG: Trying to check X509 cert with check_cert_type
VERBOSE: Trying to connect grid-arc-01.hpc.susx.ac.uk(IPv6):443
DEBUG: Loaded MCC tcp.client(tcp)
DEBUG: Loaded MCC tls.client(tls)
DEBUG: Loaded MCC http.client(http)
DEBUG: TCP client process called
DEBUG: No security processing/check requested for 'outgoing'
DEBUG: No security processing/check requested for 'incoming'
DEBUG: Loaded /usr/lib64/arc/libdmcsrm.so
DEBUG: Loaded /usr/lib64/arc/libmccsoap.so
DEBUG: Loaded /usr/lib64/arc/libdmcxrootddeleg.so
DEBUG: Loaded /usr/lib64/arc/libmcctcp.so
DEBUG: Loaded /usr/lib64/arc/libmcctls.so
VERBOSE: Trying to connect grid-arc-01.hpc.susx.ac.uk(IPv6):443
DEBUG: Loaded MCC tcp.client(tcp)
DEBUG: Loaded MCC tls.client(tls)
DEBUG: Loaded MCC http.client(http)
DEBUG: Loaded MCC soap.client(soap)
DEBUG: TCP client process called
DEBUG: No security processing/check requested for 'outgoing'
DEBUG: No security processing/check requested for 'incoming'
VERBOSE: Failed to establish SSL connection
DEBUG: Loaded /usr/lib64/arc/libdmcacix.so
DEBUG: Linking MCC tls.client(tls) to MCC (tcp) under (empty)
DEBUG: Linking MCC http.client(http) to MCC (tls) under (empty)
DEBUG: Loaded /usr/lib64/arc/libdmcfile.so
DEBUG: Loaded /usr/lib64/arc/libdmchttp.so
VERBOSE: Failed to establish SSL connection
DEBUG: Loaded /usr/lib64/arc/libdmcldap.so
DEBUG: Setting status (FAILED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)
DEBUG: dmcldap made persistent
DEBUG: Checking for suspended endpoints which should be started.
DEBUG: Status of endpoint (grid-arc-01.hpc.susx.ac.uk (org.nordugrid.arcrest, capabilities: information.discovery.resource)) is FAILED
DEBUG: Linking MCC tls.client(tls) to MCC (tcp) under (empty)
DEBUG: Linking MCC http.client(http) to MCC (tls) under (empty)
DEBUG: Linking MCC soap.client(soap) to MCC (http) under (empty)
DEBUG: No security processing/check requested for 'outgoing'
VERBOSE: esrinfo:GetResourceInfo request to https://grid-arc-01.hpc.susx.ac.uk:443/arex failed with response: Failed to send SOAP message: TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert): TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert)
DEBUG: XML response: <soap-env:Envelope xmlns:soap-enc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap-env:Body><soap-env:Fault><soap-env:faultcode>soap-env:Server</soap-env:faultcode><soap-env:faultstring>Failed to send SOAP message: TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert): TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert)</soap-env:faultstring></soap-env:Fault></soap-env:Body></soap-env:Envelope>
DEBUG: Loaded /usr/lib64/arc/libdmcrucio.so
DEBUG: Re-creating an EMI ES client
VERBOSE: Processing a esrinfo:GetResourceInfo request
DEBUG: Module Manager Init
DEBUG: dmcldap made persistent
DEBUG: dmcldap made persistent
VERBOSE: LDAPQuery: Initializing connection to grid-arc-01.hpc.susx.ac.uk:2135
VERBOSE: LDAPQuery: Initializing connection to grid-arc-01.hpc.susx.ac.uk:2135
DEBUG: Loaded /usr/lib64/arc/libmcchttp.so
DEBUG: Loaded /usr/lib64/arc/libmccmsgvalidator.so
DEBUG: Loaded /usr/lib64/arc/libmccsoap.so
DEBUG: Loaded /usr/lib64/arc/libmcctcp.so
DEBUG: Loaded /usr/lib64/arc/libmcctls.so
VERBOSE: Trying to connect grid-arc-01.hpc.susx.ac.uk(IPv6):443
DEBUG: Loaded MCC tcp.client(tcp)
DEBUG: Loaded MCC tls.client(tls)
DEBUG: Loaded MCC http.client(http)
DEBUG: Loaded MCC soap.client(soap)
DEBUG: TCP client process called
DEBUG: No security processing/check requested for 'outgoing'
DEBUG: No security processing/check requested for 'incoming'
VERBOSE: LDAPQuery: Querying grid-arc-01.hpc.susx.ac.uk
DEBUG: base dn: Mds-Vo-name=local,o=Grid
DEBUG: filter: (|(objectclass=nordugrid-cluster)(objectclass=nordugrid-queue))
VERBOSE: LDAPQuery: Getting results from grid-arc-01.hpc.susx.ac.uk
VERBOSE: LDAPQuery: Querying grid-arc-01.hpc.susx.ac.uk
DEBUG: base dn: o=glue
DEBUG: filter: (&(!(GLUE2GroupID=ComputingActivities))(!(ObjectClass=GLUE2ComputingActivity)))
VERBOSE: LDAPQuery: Getting results from grid-arc-01.hpc.susx.ac.uk
VERBOSE: Failed to establish SSL connection
DEBUG: Linking MCC tls.client(tls) to MCC (tcp) under (empty)
DEBUG: Linking MCC http.client(http) to MCC (tls) under (empty)
DEBUG: Linking MCC soap.client(soap) to MCC (http) under (empty)
DEBUG: Setting status (FAILED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)
DEBUG: No security processing/check requested for 'outgoing'
DEBUG: Checking for suspended endpoints which should be started.
DEBUG: Status of endpoint (grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapng, capabilities: information.discovery.resource)) is FAILED
DEBUG: Setting status (FAILED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapglue2, capabilities: information.discovery.resource)
DEBUG: Checking for suspended endpoints which should be started.
DEBUG: Status of endpoint (grid-arc-01.hpc.susx.ac.uk (org.nordugrid.ldapglue2, capabilities: information.discovery.resource)) is FAILED
VERBOSE: esrinfo:GetResourceInfo request to https://grid-arc-01.hpc.susx.ac.uk:443/arex failed with response: Failed to send SOAP message: TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert): TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert)
DEBUG: XML response: <soap-env:Envelope xmlns:soap-enc="http://schemas.xmlsoap.org/soap/encoding/" xmlns:soap-env="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soap-env:Body><soap-env:Fault><soap-env:faultcode>soap-env:Server</soap-env:faultcode><soap-env:faultstring>Failed to send SOAP message: TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert): TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "ssl3_read_bytes" function, at "SSL routines" library, with "decryption failed" alert)</soap-env:faultstring></soap-env:Fault></soap-env:Body></soap-env:Envelope>
DEBUG: Setting status (FAILED) for endpoint: grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)
DEBUG: Checking for suspended endpoints which should be started.
DEBUG: Status of endpoint (grid-arc-01.hpc.susx.ac.uk (org.ogf.glue.emies.resourceinfo, capabilities: information.discovery.resource)) is FAILED
DEBUG: Setting status (FAILED) for endpoint: grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource)
DEBUG: Checking for suspended endpoints which should be started.
DEBUG: Status of endpoint (grid-arc-01.hpc.susx.ac.uk (<empty InterfaceName>, capabilities: information.discovery.resource)) is FAILED
ERROR: Failed to retrieve information from the following endpoints:
grid-arc-01.hpc.susx.ac.uk
________________________________
From: Testbed Support for GridPP member institutes [[log in to unmask]] on behalf of Gareth Roy [[log in to unmask]]
Sent: 29 November 2019 13:31
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: Re: ARC CE6/LCMAPS/BDII
Hi Partrick,
I’ve not seen that error before, if I try and contact your CE from externally I can’t get access to it on 443 or 2135 (for ldap) so there may still be firewall issues somewhere.
If you try a:
arcinfo -d DEBUG -c grid-arc-01.hpc.susx.ac.uk
It will try and contact all of the standard endpoints to gather info and print out a large amount of information about it… We don't actually have a 443 endpoint up at Glasgow, for instance if you try:
[vagrant@localhost vagrant]$ arcinfo -c ce01.gla.scotgrid.ac.uk:443/arex
ERROR: Failed to retrieve information from the following endpoints:
ce01.gla.scotgrid.ac.uk:443/arex
but:
[vagrant@localhost vagrant]$ arcinfo -c ce01.gla.scotgrid.ac.uk
Computing service: (production)
Information endpoint: ldap://ce01.gla.scotgrid.ac.uk:2135/Mds-Vo-Name=local,o=grid
Information endpoint: ldap://ce01.gla.scotgrid.ac.uk:2135/o=glue
Information endpoint: https://ce01.gla.scotgrid.ac.uk:443/arex
Information endpoint: https://ce01.gla.scotgrid.ac.uk:443/arex
Submission endpoint: https://ce01.gla.scotgrid.ac.uk:443/arex (status: critical, interface: org.nordugrid.arcrest)
Submission endpoint: https://ce01.gla.scotgrid.ac.uk:443/arex (status: critical, interface: org.ogf.glue.emies.activitycreation)
Submission endpoint: gsiftp://ce01.gla.scotgrid.ac.uk:2811/jobs (status: ok, interface: org.nordugrid.gridftpjob)
As it's actually scraping data from the ldap endpoint.
The -d DEBUG flag should hopefully give you more info to try and see what's going on. You could also see if there is an upstream filter blocking traffic.
Thanks,
Gareth
From: Testbed Support for GridPP member institutes <[log in to unmask]<mailto:[log in to unmask]>> On Behalf Of Patrick Smith
Sent: 29 November 2019 12:41
To: [log in to unmask]<mailto:[log in to unmask]>
Subject: ARC CE6/LCMAPS/BDII
Hello,
I have setup our ARC CE6/LCMAPS/BDII but get the following error when I try to test it remotely. Has anyone seen this before? I don't appear to have any SSL certificates installed in the usual places.
Thanks
Patrick
$ arcinfo -c grid-arc-01.hpc.susx.ac.uk/arex
ERROR: Failed to retrieve information from the following endpoints:
grid-arc-01.hpc.susx.ac.uk/arex (Fault received from https://grid-arc-01.hpc.susx.ac.uk:443/arex: Failed to send SOAP message: TLS: GENERIC_ERROR (SSL error, "sslv3 alert certificate expired", in "SSL3_READ_BYTES" function, at "SSL routines" library, with "decryption failed" alert))
$ arcinfo -c grid-arc-01.hpc.susx.ac.uk/arex
ERROR: Failed to retrieve information from the following endpoints:
grid-arc-01.hpc.susx.ac.uk/arex (Fault received from https://grid-arc-01.hpc.susx.ac.uk:443/arex: Not authorized: GENERIC_ERROR (Security error: 1))
on grid-arc-01.hpc.susx.ac.uk:
----------------------------------------------------------------------------------------------------------------
# arcctl service list
arc-acix-index (Not installed, Disabled, Stopped)
arc-acix-scanner (Not installed, Disabled, Stopped)
arc-arex (Installed, Enabled, Running)
arc-datadelivery-service (Not installed, Disabled, Stopped)
arc-gridftpd (Installed, Enabled, Running)
arc-infosys-ldap (Installed, Enabled, Running)
----------------------------------------------------------------------------------------------------------------
● arc-gridftpd.service - ARC gridftpd
Loaded: loaded (/usr/lib/systemd/system/arc-gridftpd.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-11-28 16:11:27 GMT; 19h ago
● arc-arex.service - ARC Resource-coupled EXecution service
Loaded: loaded (/usr/lib/systemd/system/arc-arex.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2019-11-28 16:11:32 GMT; 19h ago
----------------------------------------------------------------------------------------------------------------
rich rules:
rule family="ipv4" port port="6445" protocol="tcp" accept
rule family="ipv4" port port="2135" protocol="tcp" accept
rule family="ipv4" port port="2811" protocol="tcp" accept
rule family="ipv4" port port="443" protocol="tcp" accept
rule family="ipv4" port port="9000-9300" protocol="tcp" accept
rule family="ipv4" port port="9000-9300" protocol="udp" accept
rule family="ipv4" port port="8443" protocol="tcp" accept
rule family="ipv4" port port="2170" protocol="tcp" accept
----------------------------------------------------------------------------------------------------------------
/etc/grid-security/hostcert.pem:
Certificate:
Data:
Version: 3 (0x2)
Serial Number: 56252 (0xdbbc)
Signature Algorithm: sha256WithRSAEncryption
Issuer: C=UK, O=eScienceCA, OU=Authority, CN=UK e-Science CA 2B
Validity
Not Before: Sep 30 14:46:02 2019 GMT
Not After : Oct 29 14:46:02 2020 GMT
Subject: C=UK, O=eScience, OU=Sussex, L=PhysicsAndAstronomy, CN=grid-arc-01.hpc.susx.ac.uk
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
________________________________
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
########################################################################
To unsubscribe from the TB-SUPPORT list, click the following link:
https://www.jiscmail.ac.uk/cgi-bin/webadmin?SUBED1=TB-SUPPORT&A=1
