mysql表文件或者数据损坏 异常问题 mysql运行异常、宕机，日志如下： InnoDB: End of page dump 2019-05-26T05:44:41.901012Z 44 [Note] InnoDB: Uncompressed page, stored checksum in field1 851205953, calculated checksums for field1: crc32 4086194213/29424421, innodb 2478445248, none 3735928559, stored checksum in field2 1958202064, calculated checksums for field2: crc32 4086194213/29424421, innodb 4236257487, none 3735928559, page LSN 61 1536263792, low 4 bytes of LSN at page end 2321078681, page number (if stored to page already) 1, space id (if created with >= MySQL-4.1.1 and stored already) 181012 InnoDB: Page may be an insert buffer bitmap page 2019-05-26T05:44:41.901034Z 44 [Note] InnoDB: It is also possible that your operating system has corrupted its own file cache and rebooting your computer removes the error. If the corrupt page is an index page. You can also try to fix the corruption by dumping, dropping, and reimporting the corrupt table. You can use CHECK TABLE to scan your table for corruption. Please refer to http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html for information about forcing recovery. 2019-05-26T05:44:41.901047Z 44 [ERROR] [FATAL] InnoDB: Unable to read page [page id: space=181012, page number=1] into the buffer pool after 100 attempts. The most probable cause of this error may be that the table has been corrupted. Or, the table was compressed with with an algorithm that is not supported by this instance. If it is not a decompress failure, you can try to fix this problem by using innodb_force_recovery. Please see http://dev.mysql.com/doc/refman/5.7/en/ for more details. Aborting... 2019-05-26 13:44:41 0x7f5053596700 InnoDB: Assertion failure in thread 139982972479232 in file ut0ut.cc line 942 InnoDB: We intentionally generate a memory trap. InnoDB: Submit a detailed bug report to http://bugs.mysql.com. InnoDB: If you get repeated assertion failures or crashes, even InnoDB: immediately after the mysqld startup, there may be InnoDB: corruption in the InnoDB tablespace. Please refer to InnoDB: http://dev.mysql.com/doc/refman/5.7/en/forcing-innodb-recovery.html InnoDB: about forcing recovery. 05:44:41 UTC - mysqld got signal 6 ; This could be because you hit a bug. It is also possible that this binary or one of the libraries it was linked against is corrupt, improperly built, or misconfigured. This error can also be caused by malfunctioning hardware. Attempting to collect some information that could help diagnose the problem. As this is a crash and something is definitely wrong, the information collection process might fail. key_buffer_size=268435456 read_buffer_size=2097152 max_used_connections=71 max_threads=1024 thread_count=71 connection_count=71 It is possible that mysqld could use up to key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 4470152 K bytes of memory Hope that's ok; if not, decrease some variables in the equation. Thread pointer: 0x7f4f58000ae0 Attempting backtrace. You can use the following information to find out where mysqld died. If you see no messages after this, something went terribly wrong... stack_bottom = 7f5053595e28 thread_stack 0x40000 /usr/sbin/mysqld(my_print_stacktrace+0x35)[0xf54535] /usr/sbin/mysqld(handle_fatal_signal+0x4a4)[0x7d3b54] /lib64/libpthread.so.0(+0xf7e0)[0x7f5618fce7e0] /lib64/libc.so.6(gsignal+0x35)[0x7f5617a6e4f5] /lib64/libc.so.6(abort+0x175)[0x7f5617a6fcd5] /usr/sbin/mysqld(_Z18ut_print_timestampP8_IO_FILE+0x0)[0x7c2c78] /usr/sbin/mysqld(_ZN2ib5fatalD1Ev+0xb3)[0x10f4cc3] /usr/sbin/mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0x1606)[0x11363d6] /usr/sbin/mysqld[0xfc4a87] /usr/sbin/mysqld(_Z29ibuf_merge_or_delete_for_pageP11buf_block_tRK9page_id_tPK11page_size_tm+0x7dd)[0xfcbedd] /usr/sbin/mysqld(_Z20buf_page_io_completeP10buf_page_tb+0x4c9)[0x1130619] /usr/sbin/mysqld[0x11534fc] /usr/sbin/mysqld(_Z13buf_read_pageRK9page_id_tRK11page_size_t+0x40)[0x1153c80] /usr/sbin/mysqld(_Z16buf_page_get_genRK9page_id_tRK11page_size_tmP11buf_block_tmPKcmP5mtr_tb+0xd21)[0x1135af1] /usr/sbin/mysqld(_Z18btr_root_block_getPK12dict_index_tmP5mtr_t+0xd8)[0x10fd0e8] /usr/sbin/mysqld(_Z12btr_root_getPK12dict_index_tP5mtr_t+0x11)[0x10fd201] /usr/sbin/mysqld(_Z18btr_validate_indexP12dict_index_tPK5trx_tb+0x1a4)[0x110a264] /usr/sbin/mysqld(_ZN11ha_innobase5checkEP3THDP15st_ha_check_opt+0x19a)[0xf8bc7a] /usr/sbin/mysqld(_ZN7handler8ha_checkEP3THDP15st_ha_check_opt+0x79)[0x824089] /usr/sbin/mysqld[0xe8c3b3] /usr/sbin/mysqld(_ZN19Sql_cmd_check_table7executeEP3THD+0xcf)[0xe8ceef] /usr/sbin/mysqld(_Z21mysql_execute_commandP3THDb+0xdb9)[0xd137a9] /usr/sbin/mysqld(_Z11mysql_parseP3THDP12Parser_state+0x40d)[0xd17e1d] /usr/sbin/mysqld(_Z16dispatch_commandP3THDPK8COM_DATA19enum_server_command+0x174e)[0xd195ee] /usr/sbin/mysqld(_Z10do_commandP3THD+0x194)[0xd19f54] /usr/sbin/mysqld(handle_connection+0x2b4)[0xdef2c4] /usr/sbin/mysqld(pfs_spawn_thread+0x174)[0x126ac94] /lib64/libpthread.so.0(+0x7aa1)[0x7f5618fc6aa1] /lib64/libc.so.6(clone+0x6d)[0x7f5617b24c4d] Trying to get some variables. Some pointers may be invalid and cause the dump to abort. Query (7f4f58005640): is an invalid pointer Connection ID (thread ID): 44 Status: NOT_KILLED The manual page at http://dev.mysql.com/doc/mysql/en/crashing.html contains information that should help you find out what is causing the crash. 表损坏，备份也会失败，操作到这个表的时候会导致数据库立即崩溃 定位目标表： mysqlcheck -u root -p --auto-repair --all-databases 检测到指定表会导致数据库挂掉，仍然不会显示表名，但是能够显示最后一个正常的表，再进入 show tables 看这个表的下一个表就可以了 损坏恢复 mysql表数据文件frm，ibd损坏后的解决办法 现象：数据库无法正常启动，提示表空间或者表其他错误，需要重binlog恢复，如果此时没有对应的binlog，那么注释binlog日志即可。此时只能使用innodb_force_recovery来启动，这个参数有7个级别，其中0-3是允许使用drop，create等命令的，大于等于4级别innodb被设置为只读。所以设置为3后重启数据库，成功链接。 但此想在库中drop或者create表仍然不行，都提示table doesn't exists。 解决办法： 在以正常启动数据库为目标，无所谓数据丢失的前提下。停掉数据库，删除损坏表对应的frm和ibd，复制正常表的frm为原表frm名，然后启动数据库，可以成功drop此表，然后重建表即可。 PS. mysql如果某些bug或者问题导致数据库挂了，或无限重启，那么可以尝试修改innodb_force_recovery为3，如果3仍然不行，可以选择修改为4，此时虽然数据库不可写，但是能够启动库，接着再使用此方法来确定有问题的表

mysql批量转换表名大小写 转换方法 使用存储过程来批量转换 DELIMITER // DROP PROCEDURE IF EXISTS uppercase // CREATE PROCEDURE uppercase(IN dbname VARCHAR(200)) BEGIN DECLARE done INT DEFAULT 0; DECLARE oldname VARCHAR(200); DECLARE cur CURSOR FOR SELECT table_name FROM information_schema.TABLES WHERE table_schema = dbname; DECLARE CONTINUE HANDLER FOR NOT FOUND SET done = 1; OPEN cur; REPEAT FETCH cur INTO oldname; SET @newname = UPPER(oldname); #IF newname equals to oldname, do nothing; #select 'a' <> 'A'; -> 0 #select 'a' <> BINARY 'A'; -> 1 SET @isNotSame = @newname <> BINARY oldname; IF NOT done && @isNotSame THEN SET @SQL = CONCAT('rename table ',oldname,' to ',@newname); PREPARE tmpstmt FROM @SQL; EXECUTE tmpstmt; DEALLOCATE PREPARE tmpstmt; END IF; UNTIL done END REPEAT; CLOSE cur; END // DELIMITER ; 调用该存储过程: call uppercase('xxx'); 接库名

mysql误删恢复 恢复方案 有以下几种数据恢复方式： 方法一: 恢复到上次的备份，无二进制日志（丢失上次备份到当前的数据） 方法二: 恢复到上次的备份，把上次备份到现在的二进制日志中相关表到删除语句之前的日志进行回放（可以恢复到删除语句之前的数据） 方法三: 如果数据量比较小，但是整体库很大，从压缩中提取出来比较慢，那么可以解析出二进制日志的delete sql，手动改写为insert（数据量小的表也等于是全量恢复） Tips. 查询binlog的时候带的参数-d xxx --base64-output=DECODE-ROWS --set-charset=UTF8，但是恢复数据的时候不要带这些参数 如果恢复过程中如果主键数据有问题或者有其他错误，但仍然想恢复这个表数据，可以mysql -uroot -p'xxx' -f 来出现错误的时候强制继续 如何恢复 场景1 如果表创建以及各种操作的binlog都还存在，那么直接拿出这些binlog，指定start时间和stop时间进行恢复，比如： mysqlbinlog --start-datetime="2021-08-08 01:00:00" --stop-datetime="2021-08-08 12:11:00" --database=hsjc_bi /tmp/binlog.000005 | mysql -uroot -p'xxx' 实际的操作应该是先从定时备份中恢复表，然后找到 binlog中表备份之后的时间到数据被删之前的时间段日志，进行应用 场景2 表已经用过一段时间，表被truncate。那么需要一个最近有效的备份，并且备份到最近时间的binlog日志。需要找到truncate的语句，从binlog中，然后truncate语句上下的pos，回放这两个pos前后所有的sql mysqlbinlog --start-position="126577910" --stop-position="126649749" --database=xxx /tmp/binlog.000004 | mysql -uroot -p'xxx' mysqlbinlog --start-position="126649944" --stop-position="126678302" --database=xxx /tmp/binlog.000004 | mysql -uroot -p'xxx' 场景3 没有备份，或者不使用备份的情况下。当然表也没有被truncate，只是被修改或者删除 提取出当前binlog的sql语句 /opt/mysql/8/bin/mysqlbinlog -vv -d oa --base64-output=DECODE-ROWS --set-charset=UTF8 /opt/mysql/8/data/binlog.000063 > fact_xxx.t 把提取出来的全部sql语句，仅过滤出所需表的delete语句 cat fact_ypdlb.t | awk '/DELETE FROM/ && (/oa.fact_ypdlb/ || /`oa`.`fact_ypdlb`/){ while(1){ print $0; getline; if($0 !~ /^###/){ break; }; } }' > oa.fact_ypdlb.delete.txt 处理语句 sed -i 's/^### //g' oa.fact_ypdlb.delete.txt sed -i "s/^DELETE FROM/INSERT INTO/g" oa.fact_ypdlb.delete.txt sed -i "s/^WHERE/VALUES(/g" oa.fact_ypdlb.delete.txt 4.处理语句 sed -i '/@13=.*/a );' oa.fact_ypdlb.delete.txt cat oa.fact_ypdlb.delete.txt | awk -F"=|/*" '{ if($0 ~ /^INSERT|^VALUES|^);/){ print $0; }else{ printf $2","; }; }' > oa.fact_ypdlb.delete.sql sed -i "s/,);$/);/g" oa.fact_ypdlb.delete.sql sed -i 's/,INSERT/);INSERT/g' oa.fact_ypdlb.delete.sql 以上的思路： 因为是没有备份，只能通过binlog的delete语句还原数据 拿到需要恢复的binlog 提取出来相关的delete语句 cat 111.sql | awk '/DELETE FROM/ { while(1){ print $0; getline; if($0 !~ /^###/){ break; }; } }' > 111.txt 处理语句大致为insert into格式 比较有局限性，因为中间可能有不同表。每个表字段数量不同，所以指定处理哪个字段不是很好。所以从这里开始思路有改变 要给每一个insert into语句末尾加上分号(;)，可以匹配INSERT INTO关键字，在上面插入即可 sed -i '/INSERT INTO/i );' 002.txt # 第一行和最后一行不对，需要手动修改下。其实第一行无所谓，主要是最后一行少这一部分 语句内的/ / 这一部分是注释，拿掉 sed -i 's#/\* .* \*/#,#g' 002.txt 语句内的开头@1这样的关键是是不能有的，实测拿去执行的话sql显示成功，但数据其实没插入，所以这些关键字拿掉 sed -i 's/@.*=//g' 002.txt 在第六步把字段末尾都替换成了逗号，这是正常的字段分隔符，但是SQL最后一个字段的末尾是不能有的，所以拿掉 sed -i ":label;N;s/,\n);/\n);/g;b label" 002.txt 这里用到了sed匹配换行符，简单搜索了下，sed原理跟换行有些关系，所以默认匹配不到换行，增加: label;N;xxxxx;b label 这一部分即可匹配成功

mysql xtrabackup备份恢复 备份脚本 #!/bin/sh #on xtrabackup 8.0.14 # 第一次执行时会检查是否有全备,否则先创建一个全库备份 #/app/percona-xtrabackup-2.4.19/bin/xtrabackup --defaults-file=/app/mysql/my.cnf #-ucgvonline -pcjcgv2019! --socket=/app/mysql/mysql.sock #-P3306 --backup --all-databases --target-dir=/tmp/tmpcheck user="root" pass="xxxx" port="3361" MY_CNF="/etc/my5.cnf" #mysql的配置文件 MYSQL_BASE="/var/lib/mysql80_5/bin" MYSQL="${MYSQL_BASE}/mysql" MYSQL_ADMIN="${MYSQL_BASE}/mysqladmin" TMPPATH="/tmp/tmpcheck" TMPLOG="${TMPPATH}/xtrabackup.$$.log" BACKUP_DIR=/data/backup # 备份的主目录 Xtrabackup_PATH=xtrabackup #INNOBACKUPEX的命令 Xtrabackup_FULLPATH=/usr/bin/$Xtrabackup_PATH #INNOBACKUPEX的命令路径 #mysql目标服务器以及用户名和密码 MYSQL_CMD=" --defaults-file=${MY_CNF} -S /var/lib/mysql80_5/mysql.sock --user=${user} --password=${pass} --port=${port}" MYSQL_UP=" --user=${user} --password=${pass} --port=${port} " #mysqladmin的用户名和密码 time=$(date +'%Y%m%d%H%M%S') FULLBACKUP_IN=$BACKUP_DIR/${port}/ FULLBACKUP_DIR=$BACKUP_DIR/${port}/"$time" # 全库备份的目录 FULLBACKUP_INTERVAL=86400 # 全库备份的间隔周期，时间：秒 ，86400为一日 KEEP_FULLBACKUP=3 # 至少保留几个全库备份 logfiledate=backup_${port}.`date +%Y%m%d%H%M`.txt #开始时间 STARTED_TIME=`date +%s` if [ ! -d "${BACKUP_DIR}" ];then mkdir -pv ${BACKUP_DIR} &> /dev/null fi if [ ! -d "${FULLBACKUP_IN}" ];then mkdir -pv ${FULLBACKUP_IN} &> /dev/null fi if [ ! -d "${TMPPATH}" ];then mkdir -pv ${TMPPATH} &> /dev/null fi ############################################################################# # 显示错误并退出 ############################################################################# error() { echo "$1" 1>&2 exit 1 } # 检查执行环境 if [ ! -x "$Xtrabackup_FULLPATH" ]; then error "$Xtrabackup未安装或未链接到/usr/bin." fi if [ ! -d $BACKUP_DIR ]; then error "备份目标文件夹:$BACKUP_DIR不存在." fi #第一句的判断不是很准确，可能造成误判 #第二句的lsof在crontab使用时候没有环境变量，需要写全路径，或者加载环境变量 #mysql_status=`netstat -nl | awk 'NR>2{if ($4 ~ /.*:3306/) {print "Yes";exit 0}}'` if ! /usr/sbin/lsof -n -i:${port} &> /dev/null;then error "MySQL 没有启动运行." fi if ! `echo 'exit' | $MYSQL $MYSQL_CMD -s` ; then error "提供的数据库用户名或密码不正确!" fi # 备份的头部信息 echo "----------------------------" echo echo "$0: MySQL备份脚本" echo "开始于: `date +%F' '%T' '%w`" echo #新建全备目录 mkdir -p $FULLBACKUP_DIR #查找最新的完全备份 LATEST_FULL_BACKUP=`find $FULLBACKUP_DIR -mindepth 1 -maxdepth 1 -iname "202*" -type d -printf "%P\n" | sort -nr | head -1` # 查找最近修改的最新备份时间 LATEST_FULL_BACKUP_CREATED_TIME=`stat -c %Y $FULLBACKUP_DIR/$LATEST_FULL_BACKUP` #如果全备有效进行增量备份否则执行完全备份 echo "*********************************" echo -e "正在执行全新的完全备份...请稍等..." echo "*********************************" $Xtrabackup_FULLPATH ${MYSQL_CMD} --backup --parallel=4 --compress --compress-threads=4 --use-memory=2G --target-dir=$FULLBACKUP_DIR > $TMPLOG 2>&1 #保留一份备份的详细日志 cat $TMPLOG>/data/backup/${port}/$logfiledate if [ -z "`tail -1 $TMPLOG | grep 'innobackupex: completed OK!'`" ] ; then echo "$INNOBACKUPEX命令执行失败:"; echo echo -e "---------- $Xtrabackup_PATH错误 ----------" cat $TMPLOG rm -f $TMPLOG #exit 1 fi THISBACKUP=`awk -- "/Backup created in directory/ { split( \\\$0, p, \"'\\'" ) ; print p[2] }" $TMPLOG` rm -f $TMPLOG echo -n "数据库成功备份到:$THISBACKUP" # 提示应该保留的备份文件起点 LATEST_FULL_BACKUP=`find $FULLBACKUP_DIR -mindepth 1 -maxdepth 1 -iname "202*" -type d -printf "%P\n" | sort -nr | head -1` RES_FULL_BACKUP=${FULLBACKUP_DIR}/${LATEST_FULL_BACKUP} echo "完成于: `date +%F' '%T' '%w`" exit 0 恢复方法 xtrabackup --prepare --target-dir=/dat/20210813111418 #指定好datadir即可，等恢复执行完后使用此datadir目录启动mysql xtrabackup --move-back --target-dir=/dat/20210813111418 --datadir=/dat/mysql80/data5

mysql源码编译安装 编译安装 编译参数 cmake .. -DCMAKE_INSTALL_PREFIX=/opt/mysql2 \ -DMYSQL_DATADIR=/opt/mysql2/data \ -DSYSCONFDIR=/etc \ -DWITH_MYISAM_STORAGE_ENGINE=1 \ -DWITH_INNOBASE_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DENABLED_LOCAL_INFILE=1 \ -DENABLE_DOWNLOADS=1 \ -DDEFAULT_CHARSET=utf8mb4 \ -DWITH_DEBUG=0 \ -DMYSQL_MAINTAINER_MODE=0 \ -DWITH_BOOST=../boost \ -DDEFAULT_COLLAION=utf8mb4_general_ci \ -DWITH_MEMORY_STORAGE_ENGINE=1 \ -DWITH_PARTITION_STORAGE_ENGINE=1 \ -DCMAKE_BUILD_TYPE=mysql_release 最小化编译 cmake .. -DCMAKE_INSTALL_PREFIX=/opt/mysql3 \ -DMYSQL_DATADIR=/opt/mysql3/data \ -DSYSCONFDIR=/etc \ -DWITH_MYISAM_STORAGE_ENGINE=1 \ -DWITH_INNOBASE_STORAGE_ENGINE=1 \ -DWITH_ARCHIVE_STORAGE_ENGINE=1 \ -DENABLED_LOCAL_INFILE=1 \ -DENABLE_DOWNLOADS=1 \ -DDEFAULT_CHARSET=utf8mb4 \ -DWITH_DEBUG=0 \ -DMYSQL_MAINTAINER_MODE=0 \ -DWITH_BOOST=../boost \ -DDEFAULT_COLLAION=utf8mb4_general_ci \ -DWITH_MEMORY_STORAGE_ENGINE=1 \ -DWITH_PARTITION_STORAGE_ENGINE=1 \ -DCMAKE_BUILD_TYPE=MinSizeRel