Hi all,
After restarting a bunch of our services as part of the glibc
vulnerability I now have a problem with our DPM server. The symptoms
are that our head node is having trouble connecting to out DPM MySQL
server node and is therefore now doing file transactions at all.
Looking at the /var/log/mysqld.log file on the DPM MySQL server after
a mysqld restart I see:
160225 12:10:23 mysqld_safe Starting mysqld daemon with databases from
/var/lib/mysql
160225 12:10:23 [Warning] No argument was provided to --log-bin, and
--log-bin-index was not used; so replication may break when this MySQL
server acts as a master and has his hostname changed!! Please use
'--log-bin=mysqld-bin' to avoid this problem.
160225 12:10:24 InnoDB: Initializing buffer pool, size = 5.9G
160225 12:10:24 InnoDB: Completed initialization of buffer pool
160225 12:10:25 InnoDB: Started; log sequence number 87 4138095865
160225 12:10:25 [Warning] Failed to setup SSL
160225 12:10:25 [Warning] SSL error: Failed to set ciphers to use
160225 12:10:25 [ERROR] Column count of mysql.db is wrong. Expected
22, found 20. The table is probably corrupted
160225 12:10:25 [ERROR] mysql.user has no `Event_priv` column at position 29
160225 12:10:25 [ERROR] Event Scheduler: An error occurred when
initializing system tables. Disabling the Event Scheduler.
160225 12:10:25 [Note] /usr/libexec/mysqld: ready for connections.
Version: '5.1.69-log' socket: '/var/lib/mysql/mysql.sock' port: 3306
Source distribution
160225 12:26:56 [ERROR] /usr/libexec/mysqld: Lock wait timeout
exceeded; try restarting transaction
160225 12:26:57 [ERROR] /usr/libexec/mysqld: Sort aborted
with those last two line being repeated every minute or so and this is
causing loads averages of "9.88, 8.37, 5.76" or more.
I have tried restarting the mysql server and the head node but neither
seems to have helped.
Now, the error in there "Column count of mysql.db is wrong. Expected
22, found 20. The table is probably corrupted" looks pretty bad. Has
anyone else seen this error before and does anyone have any ideas
about possible fixes? We have backups of the database so a restore
from yesterday would be possible but I'm also unsure about the best
way to do that.
Cheers,
Matt
|