|
Versity Software 已经把 ScoutFS 存档文件系统开源了，该项目最大的特性是能储存1万亿文件，并且支持跨平台。ScoutFS 项目始创于 2016 年，旨在解决对更大的 POSIX 命名空间和更快的元数据处理的日益增长的要求，它设计能在单一的命名空间储存最多一万亿文件。
背景 由于世界各地的大型归档站点都在规划未来的数据存储系统，因此命名空间的可扩展性成了关注重点领域。当前技术通常依赖于中央服务器进行元数据管理，超过10亿个文件的数据收集变得难以管理。在 2016 年，Versity 开始研究一种新的归档文件系统，该系统专门用于通过使用节点集群来扩展元数据处理来管理超大型命名空间。然后“scale out filesystem（扩展文件系统）”即 ScoutFS 就诞生了。
ScoutFS简介 ScoutFS 是第一个在 GPL 下发布的存档文件系统，兼容 POSIX，设计和实现将庞大数据集归档到低成本的外部储存源如磁带、磁盘、对象和云端。
ScoutFS主要特性及优势 1.主要特性： 集成一致的索引以加速归档维护操作 共享 LSM 索引结构，用于根据存储带宽扩展元数据速率 从序列化设备写入解耦逻辑锁定以减少争用 2.优势： 节点之间完全一致的 POSIX 语义 丰富的元数据，以确保元数据引用的完整性 原子事务维护一致的持久性结构 一流的内核实现，实现高性能和低延迟 基于开源的 GPLv2 实现
相关链接
附：Quick Start This following a very rough example of the procedure to get up and running, experience will be needed to fill in the gaps. We're happy to help on the mailing list. Some software components (pacemaker?) may be packaged seperately by distributions. The requirements for running scoutfs on a small cluster are: 1.One or more nodes running x86-64 CentOS/RHEL 7.4 (or 7.3) 2.Access to a single shared block device 3.IPv4 connectivity between the nodes The steps for getting scoutfs mounted and operational are: 1.Configure pacemaker clustering and the kernel DLM for locking 2.Get the kernel module running on the nodes 3.Make a new filesystem on the device with the userspace utilities 4.Mount the device on all the nodes In this example we run all of these commands on two nodes. The block device name is the same on all the nodes. The listen= mount option is given the local IP address of each node. 1.Configure and Start the DLM yum install pcs pacemaker fence-agents-all firewall-cmd --permanent --add-service=high-availability firewall-cmd --add-service=high-availability passwd hacluster systemctl start pcsd.service systemctl enable pcsd.service pcs cluster auth node1 node2 pcs cluster setup --start --name scoutfs node1 node2 pcs cluster enable yum install dlm systemctl enable dlm systemctl start dlm 2.Get the Kernel Module and Userspace Binaries 1].Either use snapshot RPMs built from git by Versity: rpm -i https://scoutfs.s3-us-west-2.amazonaws.com/scoutfs-repo-0.0.1-1.el7_4.noarch.rpm yum install scoutfs-utils kmod-scoutfs 2].Or use the binaries built from checked out git repositories: yum install kernel-devel git clone git@github.com:versity/scoutfs-kmod-dev.git make -C scoutfs-kmod-dev module modprobe libcrc32c insmod scoutfs-kmod-dev/src/scoutfs.ko git clone git@github.com:versity/scoutfs-utils-dev.git make -C scoutfs-utils-dev alias scoutfs=$PWD/scoutfs-utils-dev/src/scoutfs 3.Make a New Filesystem (destroys contents, no questions asked) scoutfs mkfs /dev/shared_block_device 4.Mount the Filesystem mkdir /mnt/scoutfs mount -t scoutfs -o cluster=scoutfs,listen=node_ip_address \ /dev/shared_block_device /mnt/scoutfs 5.For Kicks, Observe the Metadata Change Index The meta_seq index tracks the inodes that are changed in each transaction. scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/two; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs touch /mnt/scoutfs/one; sync scoutfs walk-inodes meta_seq 0 -1 /mnt/scoutfs
相关主题 |
