1、查看日志
tail -f /var/log/ceph/ceph-osd.2.log
-6> 2024-05-23 09:29:20.734328 7f837d847d00 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1716452960734313, "job": 1, "event": "recovery_started", "log_files": [1064066, 1064069]}-5> 2024-05-23 09:29:20.734340 7f837d847d00 4 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:482] Recovering log #1064066 mode 0-4> 2024-05-23 09:29:52.503818 7f837d847d00 5 rocksdb: [/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/rocksdb/db/db_impl_open.cc:815] [default] [WriteLevel0TableForRecovery] Level-0 table #1064067: started-3> 2024-05-23 09:30:08.239889 7f837d847d00 1 bluefs _allocate failed to allocate 0x47d00000 on bdev 1, free 0x1b300000; fallback to bdev 2-2> 2024-05-23 09:30:08.239910 7f837d847d00 -1 bluefs _allocate failed to allocate 0x47d00000 on bdev 2, dne-1> 2024-05-23 09:30:08.239912 7f837d847d00 -1 bluefs _flush_range allocated: 0x0 offset: 0x0 length: 0x47c548200> 2024-05-23 09:30:08.246076 7f837d847d00 -1 /home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueFS.cc: In function 'int BlueFS::_flush_range(BlueFS::FileWriter*, uint64_t, uint64_t)' thread 7f837d847d00 time 2024-05-23 09:30:08.239921
/home/jenkins-build/build/workspace/ceph-build/ARCH/x86_64/AVAILABLE_ARCH/x86_64/AVAILABLE_DIST/centos7/DIST/centos7/MACHINE_SIZE/huge/release/12.2.7/rpm/el7/BUILD/ceph-12.2.7/src/os/bluestore/BlueFS.cc: 1525: FAILED assert(0 == "bluefs enospc")ceph version 12.2.7 (3ec878d1e53e1aeb47a9f619c49d9e7c0aa384d5) luminous (stable)1: (ceph::__ceph_assert_fail(char const*, char const*, int, char const*)+0x110) [0x7f837e2f6850]2: (BlueFS::_flush_range(BlueFS::FileWriter*, unsigned long, unsigned long)+0x1b1b) [0x7f837e27501b]3: (BlueFS::_flush(BlueFS::FileWriter*, bool)+0x188) [0x7f837e275588]4: (BlueRocksWritableFile::Flush()+0x3c) [0x7f837e29113c]5: (rocksdb::WritableFileWriter::Flush()+0xaf) [0x7f837e662a6f]6: (rocksdb::WritableFileWriter::Sync(bool)+0x33) [0x7f837e662c83]7: (rocksdb::BuildTable(std::string const&, rocksdb::Env*, rocksdb::ImmutableCFOptions const&, rocksdb::MutableCFOptions const&, rocksdb::EnvOptions const&, rocksdb::TableCache*, rocksdb::InternalIterator*, std::unique_ptr<rocksdb::InternalIterator, std::default_delete<rocksdb::InternalIterator> >, rocksdb::FileMetaData*, rocksdb::InternalKeyComparator const&, std::vector<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> >, std::allocator<std::unique_ptr<rocksdb::IntTblPropCollectorFactory, std::default_delete<rocksdb::IntTblPropCollectorFactory> > > > const*, unsigned int, std::string const&, std::vector<unsigned long, std::allocator<unsigned long> >, unsigned long, rocksdb::CompressionType, rocksdb::CompressionOptions const&, bool, rocksdb::InternalStats*, rocksdb::TableFileCreationReason, rocksdb::EventLogger*, int, rocksdb::Env::IOPriority, rocksdb::TableProperties*, int)+0x1004) [0x7f837e679d54]8: (rocksdb::DBImpl::WriteLevel0TableForRecovery(int, rocksdb::ColumnFamilyData*, rocksdb::MemTable*, rocksdb::VersionEdit*)+0x90c) [0x7f837e5c621c]9: (rocksdb::DBImpl::RecoverLogFiles(std::vector<unsigned long, std::allocator<unsigned long> > const&, unsigned long*, bool)+0x1430) [0x7f837e5c7e90]10: (rocksdb::DBImpl::Recover(std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, bool, bool, bool)+0x7e6) [0x7f837e5c92e6]11: (rocksdb::DB::Open(rocksdb::DBOptions const&, std::string const&, std::vector<rocksdb::ColumnFamilyDescriptor, std::allocator<rocksdb::ColumnFamilyDescriptor> > const&, std::vector<rocksdb::ColumnFamilyHandle*, std::allocator<rocksdb::ColumnFamilyHandle*> >*, rocksdb::DB**)+0xed3) [0x7f837e5ca5a3]12: (rocksdb::DB::Open(rocksdb::Options const&, std::string const&, rocksdb::DB**)+0x186) [0x7f837e5cb7e6]13: (RocksDBStore::do_open(std::ostream&, bool)+0x8e8) [0x7f837e212368]14: (BlueStore::_open_db(bool)+0xdb3) [0x7f837e19d313]15: (BlueStore::_mount(bool)+0x40e) [0x7f837e1cf06e]16: (OSD::init()+0x3bd) [0x7f837dd8434d]17: (main()+0x2d07) [0x7f837dc89947]18: (__libc_start_main()+0xf5) [0x7f837a0c6b35]19: (()+0x4b9003) [0x7f837dd28003]NOTE: a copy of the executable, or `objdump -rdS <executable>` is needed to interpret this
使用ceph-bluestore-tool
ceph-bluestore-tool 是一个对 BlueStore 实例执行低级管理操作的实用工具。
使用前，关闭对应 OSD 服务
[root@node-1 ceph-objectstore-tool-test]# systemctl stop ceph-osd@0
2. 示例
元数据、数据校验（–deep true，开启数据校验）
这实际是调用bluestore中的fsck()函数，有deep、repair等选项，支持不同程度的检查工作。具体工作有：更新kvdb中super前缀信息（ondisk_format、min_alloc_size等，具体可查阅BlueStore::_upgrade_super()函数）、重放deferred_transaction、删除失效blob、删除错误pextent、更新bluefs文件系统信息等。
所以fsck主要针对bluestore中的元数据信息的修复和检查。
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ fsck [--deep true][root@localhost build]# ./bin/ceph-bluestore-tool --log-level 10 --log-file /root/bluestore-tool.log --path /var/lib/ceph/osd/ceph-2/ fsck --deep true
fsck success修复
调用bluestore中的fsck()函数实现功能。
ceph-bluestore-tool --path /var/lib/ceph/osd/ceph-0/ [repair|quick-fix] [--deep true][root@localhost build]# ./bin/ceph-bluestore-tool --log-level 10 --log-file /root/bluestore-tool.log --path /var/lib/ceph/osd/ceph-2/ repair --deep true
repair success