 |
 |
Yes, I think this is your only option now. The developers are being less than helpful. As for tidying things up, I think it's going to be messy. WE should be able to use something like this to find the pnfsids that are not related to any file in the namespace. http://www.sysadmin.hep.ac.uk/svn/fabric-management/dcache/pnfs/remove-orphan-files.sh Greig On 07/12/07 11:21, Matt Doidge wrote: > Hello, > > I've made extensive use of the single-user mode of postgres whilst > attempting to repair our databases, and I couldn't find any glaring > errors or problems looking into these things-but then again I wouldn't > nessicerily have done as the postgres backend to pnfs has always been > something opaque and mysterious to me. > > After much deliberation we've decided to go with the rollback option, > our best "customers" would rather have us back in action and needing > to be reseeded then sitting around being useless. I'm going to keep > the bust database we have and see if something can be salvaged from > it, maybe some entries can be translated over? Maybe I'm just being > overly optimisitic. I'll do the rollback after lunch, to give time for > people to object if they think it's a very very bad idea or present me > with other options, but right now I don't think we have any. > > One thing's for certain, I'll never blindly assume that my backups are > working fine again. > > Thanks for the help Greig, > > Matt > > On 07/12/2007, Greig A Cowan <[log in to unmask]> wrote: >> Matt, >> >> Have you considered the approach of running in single-user mode with just >> one of the broken databases? >> >> http://www.postgresql.org/docs/8.1/static/app-postgres.html >> >> postgres -D /var/lib/pgsql/data/ data1 >> >> This might give you some ability to try and fix things. >> >> As you will know, I've also submitted another ticket to dCache support. >> >> Greig >> >> >> >> On Fri, 7 Dec 2007, Matt Doidge wrote: >> >>> Hello, >>> >>> Just in case the worst happens and we can't salvage pnfs from what's >>> our current postgres and have to use the 3.5 month old backup, would >>> there be guidelines as to how to go about sorting out the horrid mess >>> that would leave. There's guidelines to sort things out from the point >>> of view of a pool snuffing it, but nothing for database failures >>> (largely as such extreme database related errors shouldn't happen due >>> to regular backups being in place). >>> >>> The other piece of advice is, at what point do you should I just give >>> up and start dusting off the old back-up? We've been down since >>> Tuesday. How much downtime is a 3.5 month data rollback worth? Maybe I >>> should put this question to the major VOs we support (aka atlas)? >>> >>> As you can tell I'm a little confused and overwhelmed, and mightily frustrated. >>> >>> cheers, >>> Matt >>> >>> On 06/12/2007, Matt Doidge <[log in to unmask]> wrote: >>>> Hello, >>>> >>>> All my empty files are in place. Postgres is back up and running- I >>>> can connect to it and poke around. However there could have been >>>> dataloss, and so when pnfs looks into postgres for its gubbins all it >>>> is perhaps seeing is gobblygook and thus not be able to initialise >>>> properly. However that's just a theory that I so hope is wrong. >>>> >>>> I'll see if I can dig up a spare node to see if I can get it to work >>>> elsewhere, but I'm not sure we've got any spare machines laying about >>>> the place. It's worth a try, at the moment I'm just banging my head >>>> against a wall, which isn't getting the job done sadly. >>>> >>>> If anyone knows of any postgres queries I could issue that would test >>>> how postgres is looking to pnfs then that would be great. >>>> >>>> cheers, >>>> Matt >>>> On 06/12/2007, Greig Alan Cowan <[log in to unmask]> wrote: >>>>> You've got empty file corresponding to each database in this directory? >>>>> >>>>> /opt/pnfsdb/pnfs/databases >>>>> >>>>> Are you sure that postgres is back up and running? Can you really >>>>> connect to it? >>>>> >>>>> I don't know what causes the enabled (x) output, but it probably implies >>>>> a problem with postgres. I've only seen it once before. >>>>> >>>>> Greig >>>>> >>>>> On 06/12/07 08:35, Matt Doidge wrote: >>>>>> Thanks for the reply Greig, >>>>>> >>>>>> The /opt/pnfsdb/pnfs/info files all seem present and correct. The >>>>>> output of a mdb show gives >>>>>> much the same as usual, except the status column for each reads >>>>>> "enabled (x)". It should be noted that all the bust databases are also >>>>>> all the larger ones, of the databases that work only babar has any >>>>>> significant amount of data. >>>>>> >>>>>> Posting to both users and support was a little cheeky, but I'm trying >>>>>> to maximise coverage in the hopes of maximising my chances of finding >>>>>> a solution that doesn't involve losing 3 months of data, desperate >>>>>> times call for desperate postings! >>>>>> >>>>>> cheers, >>>>>> Matt >>>>>> >>>>>> On 06/12/2007, Greig A Cowan <[log in to unmask]> wrote: >>>>>>> Hi Matt, >>>>>>> >>>>>>> You've probably checked this already, but what are the contents of >>>>>>> >>>>>>> /opt/pnfsdb/pnfs/info >>>>>>> >>>>>>> There should be a file for each database, with contents like: >>>>>>> >>>>>>> $ cat D-0000 >>>>>>> admin:0:r:enabled:/opt/pnfsdb/pnfs/databases/admin >>>>>>> >>>>>>> Also what does this command give you? It should be something like that >>>>>>> below. >>>>>>> >>>>>>> $ /opt/pnfs/tools/mdb show >>>>>>> ID Name Type Status Path >>>>>>> ---------------------------------------------- >>>>>>> 0 admin r enabled (r) /opt/pnfsdb/pnfs/databases/admin >>>>>>> 1 data1 r enabled (r) /opt/pnfsdb/pnfs/databases/data1 >>>>>>> ... >>>>>>> >>>>>>> Also, you should note that you should typically just post to the >>>>>>> user-forum or support@dcache (the developers get a bit narky when people >>>>>>> post to both ;) ) >>>>>>> >>>>>>> Cheers, >>>>>>> Greig >>>>>>> >> -- >> ======================================================================== >> Dr Greig A Cowan http://www.ph.ed.ac.uk/~gcowan1 >> School of Physics, University of Edinburgh, James Clerk Maxwell Building >> >> TIER-2 STORAGE SUPPORT PAGES: http://wiki.gridpp.ac.uk/wiki/Grid_Storage >> ======================================================================== >>