本文分享自华为云社区《记一次MySQL崩溃修复案例，再也不用删库跑路了》，作者： 冰 河。
问题描述
研究 MySQL 源代码，调试并压测 MySQL 源代码时，MySQL 崩溃了！问题是它竟然崩溃了！而且还损坏了 InnoDB 文件！！还好是在调试环境下发生的，赶紧看看如何解决这个问题，经过一系列的查阅资料、验证、对比、MySQL 源码调试跟踪、修复损坏的 InnoDB 文件、总结等流程，整理成此文，如果以后真的发生在线上的生产坏境，也不用担心是不是要跑路的问题了，可以分分钟搞定 MySQL 的崩溃问题了！！查看错误日志，如下：
-----------------------------------------161108 23:36:45 mysqld_safe Starting mysqld daemon with databases from /usr/local/mysql/var2022-08-25 23:36:46 0 [Warning] TIMESTAMP with implicit DEFAULT value is deprecated. Please use --explicit_defaults_for_timestamp server option (see documentation for more details).2022-08-25 23:36:46 5497 [Note] Plugin 'FEDERATED' is disabled.2022-08-25 23:36:46 7f11c48e1720 InnoDB: Warning: Using innodb_additional_mem_pool_size is DEPRECATED. This option may be removed in future releases, together with the option innodb_use_sys_malloc and with the InnoDB's internal memory allocator.2022-08-25 23:36:46 5497 [Note] InnoDB: Using atomics to ref count buffer pool pages2022-08-25 23:36:46 5497 [Note] InnoDB: The InnoDB memory heap is disabled2022-08-25 23:36:46 5497 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins2022-08-25 23:36:46 5497 [Note] InnoDB: Memory barrier is not used2022-08-25 23:36:46 5497 [Note] InnoDB: Compressed tables use zlib 1.2.32022-08-25 23:36:46 5497 [Note] InnoDB: Using CPU crc32 instructions2022-08-25 23:36:46 5497 [Note] InnoDB: Initializing buffer pool, size = 16.0M2022-08-25 23:36:46 5497 [Note] InnoDB: Completed initialization of buffer poolInnoDB: Database page corruption on disk or a failedInnoDB: file read of page 5.InnoDB: You may have to recover from a backup.2022-08-25 23:36:46 7f11c48e1720 InnoDB: Page dump in ascii and hex (16384 bytes): len 16384; hex 7478d078000000050000000000000000000000000f271f4d000700000000000000000000000000000000001b4000000000000000000200f20000000000000006000000000000002d000000000000002e000000000000002f0000000000000030000000000（省略很多类似代码）InnoDB: End of page dump2022-08-25 23:36:46 7f11c48e1720 InnoDB: uncompressed page, stored checksum in field1 1954074744, calculated checksums for field1: crc32 993334256, innodb 2046145943, none 3735928559, stored checksum in field2 1139795846, calculated checksums for field2: crc32 993334256, innodb 1606613742, none 3735928559, page LSN 0 254222157, low 4 bytes of LSN at page end 254221236, page number (if stored to page already) 5, space id (if created with >= MySQL-4.1.1 and stored already) 0InnoDB: Page may be a transaction system pageInnoDB: Database page corruption on disk or a failedInnoDB: file read of page 5.InnoDB: You may have to recover from a backup.InnoDB: It is also possible that your operatingInnoDB: system has corrupted its own file cacheInnoDB: and rebooting your computer removes theInnoDB: error.InnoDB: If the corrupt page is an index pageInnoDB: you can also try to fix the corruptionInnoDB: by dumping, dropping, and reimportingInnoDB: the corrupt table. You can use CHECKInnoDB: TABLE to scan your table for corruption.InnoDB: See also http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.InnoDB: Ending processing because of a corrupt database page.2022-08-25 23:36:46 7f11c48e1720 InnoDB: Assertion failure in thread 139714288817952 in file buf0buf.cc line 4201InnoDB: We intentionally generate a memory trap.InnoDB: Submit a detailed bug report to http://bugs.mysql.com.InnoDB: If you get repeated assertion failures or crashes, evenInnoDB: immediately after the mysqld startup, there may beInnoDB: corruption in the InnoDB tablespace. Please refer toInnoDB: http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.htmlInnoDB: about forcing recovery.03:36:46 UTC - mysqld got signal 6 ;This could be because you hit a bug. It is also possible that this binaryor one of the libraries it was linked against is corrupt, improperly built,or misconfigured. This error can also be caused by malfunctioning hardware.We will try our best to scrape up some info that will hopefully helpdiagnose the problem, but since we have already crashed,something is definitely wrong and this may fail.key_buffer_size=16777216read_buffer_size=262144max_used_connections=0max_threads=1000thread_count=0connection_count=0It is possible that mysqld could use up tokey_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 798063 K bytes of memoryHope that's ok; if not, decrease some variables in the equation.Thread pointer: 0x0Attempting backtrace. You can use the following information to find outwhere mysqld died. If you see no messages after this, something wentterribly wrong...stack_bottom = 0 thread_stack 0x40000/usr/local/mysql/bin/mysqld(my_print_stacktrace+0x35)[0x8e64b5]/usr/local/mysql/bin/mysqld(handle_fatal_signal+0x41b)[0x652fbb]/lib64/libpthread.so.0(+0xf7e0)[0x7f11c44c77e0]/lib64/libc.so.6(gsignal+0x35)[0x7f11c315d625]/lib64/libc.so.6(abort+0x175)[0x7f11c315ee05]/usr/local/mysql/bin/mysqld[0xa585c5]/usr/local/mysql/bin/mysqld[0xa6c7b4]/usr/local/mysql/bin/mysqld[0xa6cbc7]/usr/local/mysql/bin/mysqld[0xa5bce2]/usr/local/mysql/bin/mysqld[0xa1e2ba]/usr/local/mysql/bin/mysqld[0xa0bf60]/usr/local/mysql/bin/mysqld[0x95a427]/usr/local/mysql/bin/mysqld(_Z24ha_initialize_handlertonP13st_plugin_int+0x48)[0x58f788]/usr/local/mysql/bin/mysqld[0x6e4a36]/usr/local/mysql/bin/mysqld(_Z11plugin_initPiPPci+0xb3e)[0x6e826e]/usr/local/mysql/bin/mysqld[0x582d85]/usr/local/mysql/bin/mysqld(_Z11mysqld_mainiPPc+0x4d8)[0x587d18]/lib64/libc.so.6(__libc_start_main+0xfd)[0x7f11c3149d5d]/usr/local/mysql/bin/mysqld[0x57a019]The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html containsinformation that should help you find out what is causing the crash.161108 23:36:46 mysqld_safe mysqld from pid file /usr/local/mysql/var/VM_241_49_centos.pid ended------------------------------------------------------------------------------
复制代码
问题分析
从日志中可以看出是 innodb 引擎出了问题。日志里提示到 http://dev.mysql.com/doc/refman/5.6/en/forcing-innodb-recovery.html查看强制恢复的方法。在 mysql 的配置文件 my.cnf 里找到 [mysqld]字段下，添加 innodb_force_recovery=1：
[mysqld]innodb_force_recovery = 1
复制代码
如果 innodb_force_recovery = 1 不生效，则可尝试 2——6 几个数字
然后重启 mysql，重启成功。然后使用 mysqldump 或 pma 导出数据，执行修复操作等。修复完成后，把该参数注释掉，还原默认值 0。
配置文件的参数：innodb_force_recovery
innodb_force_recovery 影响整个 InnoDB 存储引擎的恢复状况。默认为 0，表示当需要恢复时执行所有的恢复操作（即校验数据页/purge undo/insert buffer merge/rolling back&forward），当不能进行有效的恢复操作时，mysql 有可能无法启动，并记录错误日志；
innodb_force_recovery 可以设置为 1-6,大的数字包含前面所有数字的影响。当设置参数值大于 0 后，可以对表进行 select,create,drop 操作,但 insert,update 或者 delete 这类操作是不允许的。
(SRV_FORCE_IGNORE_CORRUPT):忽略检查到的 corrupt 页。
(SRV_FORCE_NO_BACKGROUND):阻止主线程的运行，如主线程需要执行 full purge 操作，会导致 crash。
(SRV_FORCE_NO_TRX_UNDO):不执行事务回滚操作。
(SRV_FORCE_NO_IBUF_MERGE):不执行插入缓冲的合并操作。
(SRV_FORCE_NO_UNDO_LOG_SCAN):不查看重做日志，InnoDB 存储引擎会将未提交的事务视为已提交。
(SRV_FORCE_NO_LOG_REDO):不执行前滚的操作。
解决方案
一般修复方法参考：
第一种方法
建立一张新表：
create table demo_bak #和原表结构一样，只是把INNODB改成了MYISAM。
复制代码
把数据导进去
insert into demo_bak select * from demo;
复制代码
删除掉原表：
注释掉 innodb_force_recovery 之后，重启。
重命名：
rename table demo_bak to demo;
复制代码
最后改回存储引擎：
alter table demo engine = innodb
复制代码
第二种方法
另一个方法是使用 mysqldump 将表格导出，然后再导回到 InnoDB 表中。这两种方法的结果是相同的。
备份导出(包括结构和数据)：
mysqldump -uroot -p123 test > test.sql
复制代码
还原方法 1：
还原方法 2（系统命令行）：
mysql -uroot -p123 test < test.sql;
复制代码
注意，CHECK TABLE 命令在 InnoDB 数据库中基本上是没有用的。
第三种方法
1、配置
配置 innodb_force_recovery = 1 或 2——6 几个数字，重启 MySQL
2、导出数据脚本
mysqldump -uroot -p123 test > test.sql
复制代码
导出 SQL 脚本。或者用 Navicat 将所有数据库/表导入到其他服务器的数据库中。
注意：这里的数据一定要备份成功。然后删除原数据库中的数据。
3、删除 ib_logfile0、ib_logfile1、ibdata1
备份 MySQL 数据目录下的 ib_logfile0、ib_logfile1、ibdata1 三个文件，然后将这三个文件删除
4、配置
将 my.cnf 中 innodb_force_recovery = 1 或 2——6 几个数字这行配置删除或者配置为 innodb_force_recovery = 0，重启 MySQL 服务
5、将数据导入 MySQL 数据库
mysql -uroot -p123 test < test.sql; 或者用Navicat将备份的数据导入到数据库中。
复制代码
此种方法下要注意的问题：
点击关注，第一时间了解华为云新鲜技术~
评论