On Thu, May 04, 2006 at 10:31:14AM +0100 or thereabouts, Kostas Georgiou wrote:
> On Thu, May 04, 2006 at 09:37:05AM +0100, Steve Traylen wrote:
>
> > On Sat, Apr 29, 2006 at 06:37:47PM +0100 or thereabouts, Kostas Georgiou wrote:
> > >
> > > Correct me If I am wrong but shouldn't everything else read the data from
> > > the ldap server? The ldap server gets the ldif from lcg-info-generic but
> > > the attributes are stored as OIDs internally so the search will still give
> > > you the same results as the old script right?
> >
> > Of course you are correct again.
> >
> > After testing on a couple of other nodes not in production or where the
> > information is non critical it now deployed on our CE
> > lcgce01.gridpp.rl.ac.uk.
> >
> > Kostas, this clearly makes a difference in a good direction and should
> > be submitted up the tree for certification and deployment. I'm happy
> > to do that giving you credit of course if you want me to.
>
> I'll be happy if you can submit it upstream, I suspect that it will
> reach the right people a lot sooner if you do it instead of me.
>
> Here is a version that uses Net::LDAP::LDIF which makes the code a lot
> easier to read and it should be able to handle LDIF files a lot better
> than hand written regular expressions. The downside is that it requires
> yet another perl module but perl-Net-LDAP is already required by other
> packages so it's not that bad.
Comparing the outputs through a gris of the origional and your regex
one then the significant difference is the original showed
dn: GlueCEUniqueID=lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-S,mds-vo-nam
e=local,o=grid
objectClass: GlueCETop
objectClass: GlueCE
objectClass: GlueSchemaVersion
objectClass: GlueCEAccessControlBase
objectClass: GlueCEInfo
objectClass: GlueCEPolicy
objectClass: GlueCEState
objectClass: GlueInformationService
objectClass: GlueKey
GlueCEHostingCluster: lcgce01.gridpp.rl.ac.uk
GlueCEName: S
GlueCEUniqueID: lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-S
GlueCEInfoGatekeeperPort: 2119
GlueCEInfoHostName: lcgce01.gridpp.rl.ac.uk
GlueCEInfoLRMSType: torque
GlueCEInfoLRMSVersion: 2.0.0p7
and your one shows
dn: glueceuniqueid=lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-s,mds-vo-nam
e=local,o=grid
GlueSchemaVersionMajor: 1
GlueCEPolicyAssignedJobSlots: 0
GlueCEInfoApplicationDir: /stage/sl3-lcg-exp
objectClass: GlueCETop
objectClass: GlueCE
objectClass: GlueSchemaVersion
objectClass: GlueCEAccessControlBase
objectClass: GlueCEInfo
objectClass: GlueCEPolicy
in particular this instance is lower case , are these both the same
instance in LDIF terms.
Now looking at the your perl('LDAP') version it shows it as
before with camel case instances.
# lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-S, local, grid
dn: GlueCEUniqueID=lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-S,mds-vo-nam
e=local,o=grid
objectClass: GlueCETop
objectClass: GlueCE
objectClass: GlueSchemaVersion
objectClass: GlueCEAccessControlBase
objectClass: GlueCEInfo
objectClass: GlueCEPolicy
objectClass: GlueCEState
objectClass: GlueInformationService
objectClass: GlueKey
GlueCEHostingCluster: lcgce01.gridpp.rl.ac.uk
GlueCEName: S
GlueCEUniqueID: lcgce01.gridpp.rl.ac.uk:2119/jobmanager-lcgpbs-S
GlueCEInfoGatekeeperPort: 2119
GlueCEInfoHostName: lcgce01.gridpp.rl.ac.uk
GlueCEInfoLRMSType: torque
GlueCEInfoLRMSVersion: 2.0.0p7
GlueCEInfoTotalCPUs: 856
GlueCEInfoJobManager: lcgpbs
so my suspician is that the 1st and 2nd outputs are not equivalent though
it does not matter since perl-Ldap one is probably less error prone
to the previous white space problems and such like.
Steve
>
> Cheers,
> Kostas
> #!/usr/bin/perl -w
> use strict;
> use POSIX;
> use File::stat;
> use FileHandle;
> use Net::LDAP::LDIF;
>
> # Read an ldif file and populate a hash with Net::LDAP::Entry objects.
> sub read_ldif_file($%) {
> my $file = shift;
> my $href = shift;
> my $ldif = Net::LDAP::LDIF->new( $file, "r", onerror => 'die');
> while( not $ldif->eof ( ) ) {
> my $entry = $ldif->read_entry ( );
> if ( ! defined( $href->{lc($entry->dn())} ) ) {
> $href->{lc($entry->dn())} = $entry;
> } else {
> my $old_entry = $href->{lc($entry->dn())};
> foreach my $attr ( $entry->attributes ) {
> $old_entry->add( $attr => $entry->get_value( $attr, asref => 1 ) );
> }
> }
> }
> $ldif->done();
> }
>
> # Set our process group to a distinct value
> setpgrp();
>
> #Read in the configuration file.
> my %config = (
> static_dir => '',
> plugin_dir => '',
> provider_dir => '',
> cache_ttl => '',
> freshness => '',
> timeout => '',
> response => '',
> temp_dir => '',
> );
>
> if ($ARGV[0]){
> my $fh = new FileHandle $ARGV[0]
> or die "Error: Can't open config file: $ARGV[0]\n";
> foreach (<$fh>){
> if ((! m/^\#/) & (m/=/)){
> m/^(.*)=(.*)$/;
> my $key=$1;
> my $value=$2;
> $key=~s/\s+//g;
> $value=~s/\s+//g;
> $config{$key}=$value;
> }
> }
> }else{
> print "Usage: lcg-info-generic <config file>\n";
> exit 1;
> }
>
> while ( my ($key, $value) = each(%config) ) {
> if ( ! $config{$key} ){
> print "Error: configuration parameter $key not set.\n";
> exit 2;
> }
> }
>
> my @files=glob("$config{static_dir}/*.ldif");
>
> my $dynamic_entries = {};
> my $static_entries = {};
> foreach(@files){
> #Reads the static ldif.
> read_ldif_file($_, $static_entries);
> }
>
> chomp($config{temp_dir});
> mkdir($config{temp_dir});
>
> my @dynamic;
> my @provider;
> my $pid;
> my @pid;
> my $file_time;
> my $current_time=time;
>
> #Remove junk files.
> my @junk;
> push @junk, glob("$config{plugin_dir}/*~");
> push @junk, glob("$config{provider_dir}/*~");
> push @junk, glob("$config{plugin_dir}/\#*");
> push @junk, glob("$config{provider_dir}/\#*");
>
> foreach(@junk){
> remove($_);
> }
>
> @dynamic=glob("$config{plugin_dir}/*");
> @provider=glob("$config{provider_dir}/*");
>
> for(@dynamic, @provider){
> my $chksum = unpack("%64C*", $_);
> my @file= split ( );
> $file[0]=~s/.*\///;
>
> if(! -e "$config{temp_dir}/$file[0].ldif.$chksum"){
> system("touch $config{temp_dir}/$file[0].ldif.$chksum");
> $file_time = $current_time - ($config{freshness}+1);
> }
> else{
> $file_time=stat("$config{temp_dir}/$file[0].ldif.$chksum");
> if($file_time){
> $file_time=$file_time->mtime;
> }else{
> $file_time=0;
> }
> }
>
> if($file_time + $config{freshness} < $current_time ){
>
> # Fork the search.
> unless ($pid=fork){
>
> close STDOUT;
>
> # Set our process group to a distinct value.
> setpgrp();
> my $PGRP=getpgrp();
>
> # Eval will kill the process if it times out.
> eval {
> local $SIG{ALRM} = sub { die "GOT TIRED OF WAITING" };
> alarm ($config{timeout}); #Will call alarm after the timeout.
> system("$_ > $config{temp_dir}/$file[0].ldif.$chksum.$PGRP");
> `mv $config{temp_dir}/$file[0].ldif.$chksum.$PGRP $config{temp_dir}/$file[0].ldif.$chksum`;
> alarm(0); # Cancel the pending alarm if responds.
> };
>
> # This sections is executed if the process times out.
> if ($@ =~ /GOT TIRED OF WAITING/) {
> `rm -f $config{temp_dir}/$file[0].ldif.$chksum.$PGRP`;
> kill (-SIGKILL(), $PGRP);
> exit 1;
> }
> exit 0;
> }
> push @pid, $pid;
> }
> }
>
> # Eval will kill the process if it times out.
> eval {
> local $SIG{ALRM} = sub { die "GOT TIRED OF WAITING" };
> alarm ($config{response}); #Will call alarm after the timeout.
> foreach(@pid){
> waitpid($_,0);
> }
> alarm(0); # Cancel the pending alarm if responds.
> };
>
> for(@provider){
> my $chksum = unpack("%64C*", $_);
> my @file= split ( );
> $file[0]=~s/.*\///;
> $file_time=stat("$config{temp_dir}/$file[0].ldif.$chksum");
> if($file_time){
> $file_time=$file_time->mtime;
> }else{
> $file_time=$current_time;
> }
> if($file_time + $config{cache_ttl} > $current_time ){
> #Reads the output from the provider script.
> read_ldif_file("$config{temp_dir}/$file[0].ldif.$chksum", $static_entries);
> }
> }
>
> for(@dynamic){
> my $chksum = unpack("%64C*", $_);
> my @file= split ( );
> $file[0]=~s/.*\///;
> $file_time=stat("$config{temp_dir}/$file[0].ldif.$chksum");
> if($file_time){
> $file_time=$file_time->mtime;
> }else{
> $file_time=$current_time;
> }
> if($file_time + $config{cache_ttl} > $current_time ){
> #Reads the output from the dynamic script.
> read_ldif_file("$config{temp_dir}/$file[0].ldif.$chksum", $dynamic_entries);
> }
> }
>
> # Merge dynamic values to the static ones
> # Dynamic entries without corresponding static entries are ignored
> foreach my $dn ( keys %$static_entries ) {
> if ( defined( $dynamic_entries->{$dn} ) ) {
> my $dynamic_entry = $dynamic_entries->{$dn};
> my $static_entry = $static_entries->{$dn};
> foreach my $attr ( $dynamic_entry->attributes() ) {
> if( $static_entry->exists($attr) ) {
> $static_entry->replace( $attr => $dynamic_entry->get_value( $attr, asref => 1 ) );
> }
> }
> }
> }
>
> # Write the ldif in STDOUT
> print "\n";
> my $ldif = new Net::LDAP::LDIF("-", "w",
> lowercase => 0,
> encode => 'none',
> wrap => 0);
> foreach my $entry ( values %$static_entries ) {
> $ldif->write( $entry );
> }
> $ldif->done();
> print "\n";
>
> exit;
--
Steve Traylen
[log in to unmask]
http://www.gridpp.ac.uk/
