搭建分布式文件系统FastDFS集群

FastDFS是为互联网应用量身定做的一套分布式文件存储系统，非常适合用来存储用户图片、视频、文档等文件。对于互联网应用，和其他分布式文件系统相比，优势非常明显。出于简洁考虑，FastDFS没有对文件做分块存储，因此不太适合分布式计算场景。

在生产环境中往往数据存储量比较大，因此会大部分会选择分布式存储来解决，主要解决以下几个问题

  • 海量数据存储
  • 数据高可用(冗余备份)
  • 较高读写性能和负载均衡
  • 支持多平台多语言
  • 高并发问题

常见分布式存储对比

image_1ebvdh20b1guo13gibct1bkcl5k9

FastDFS 相关组件及原理

FastDFS介绍

FastDFS是一个C语言实现的开源轻量级分布式文件系统，支持Linux、FreeBSD、AID等Linux系统，解决了大量数据存储和读写负载等问题，适合存储4KB~500MB之间的小文件，如图片网站，短视频网站，文档，APP下载站等，UC，京东，支付宝，迅雷，酷狗等都有使用，其中UC基于FastDFS向用户提供网盘，广告和应用下载的业务。FastDFS与MogileFS、HDFS、TFS等都不是系统级的分布式文件系统，而是应用级的分布式文件存储服务

FastDFS架构

FastDFS服务有三个角色: 跟踪服务(tracker server)、存储服务(storage server)和客户端(client)

tracker server: 跟踪服务，主要做调度工作，起到均衡的作用；负责管理所有的storage server和group，每个storage在启动后会连接Tracker，告知自己所属group等信息，并保持周期性心跳，tracker根据storage心跳信息，建立group --> [storage server list]的映射表；tracker管理的元数据很少，会直接存放在内存；tracker上的元信息都是由storage汇报的信息生成的，本身不需要持久化任何数据，tracker之间是对等关系，因此扩展tracker访问非常容器，之间增加tracker访问即可，所有tracker都接受storage心跳信息，生成元数据信息来提供读写访问（与其他master-slave架构的优势是没有单点，tracker也不会成为瓶颈，最终数据是和一个可用的storage server进行传输）

storage server: 存储服务器，主要提供容量和备份访问；以group为单位，每个group内可以包含多个storage server，数据互为备份，存储容量空间以group内容量最小的storage为准；建议group内的storage server配置相同；以group为单位组织存储能够方便的进行引用隔离、负载均衡和副本数定制；

缺点: group的容量受单机存储容量的限制，同时group内机器坏掉，数据恢复只能依赖group内其他机器重新同步(硬盘替换，重新挂载重启fdfs_storaged即可)

group存储策略

  • round robin (轮训)
  • load balance (选择最大剩余空间的组上传文件)
  • specify group (指定group上传)

group中storage存储依赖本地文件系统，storage可配置多个数据存储目录，磁盘不做raid，直接分别挂在到多个目录，将这些目录配置为storage的数据目录即可

storage接受写请求时，会根据配置好的规则，选择其中一个存储目录来存储文件；为避免单个目录下文件过多，storage第一次启动，会在每个数据存储目录里创建2级子目录，每级256个，总共65536个，新写的文件会以hash的方式被路由到其中某个子目录下，然后将文件数据直接作为一个本地文件存储到该目录中

image_1ebvil7hskr11tv1l9m1j7p1agjm

FastDFS工作流程

上传

image_1ebviv30o1v5p16geisa18gsp6f13

FastDFS提供基本的文件访问接口，如upload、download、append、delete等

选择tracker server

集群中tracker之间是对等关系，客户端在上传文件时可用任意选择一个tracker

选择存储group

当tracker接受到upload file的请求时，会为该文件分配一个可以存储的group，目前支持选择group的规则为

1.Round Robin (所有group轮训使用)

2.Specified group (指定某个确定的group)

3.Load balance (剩余存储空间较多的group优先)

选择storage server

当选定group后，tracker会在group内选择一个storage server给客户端，目前支持选择server的规则为

1.Round Robin (所有server轮训使用)默认规则

2.根据IP地质进行排序选择第一个服务器 (IP地址最小者)

3.根据优先级进行排序 (上传优先级由storage server来设置，参数为upload_priority)

选择storage path (磁盘或者挂载点)

当分配好storage server后，客户端将向storage发送写文件请求，storage会将文件分配一个数据存储目录，目前支持选择存储路径的规则为:

1.round robin (轮训)默认

2.load balance 选择使用剩余空间最大的存储路径

选择下载服务器

目前支持的规则为

1.轮训方式，可以下载当前文件的任一storage server

2.从源storage server下载

生成file_id

选择存储目录后，storage会生成一个file_id，采用Base64编码，包含字段包括: storage server ip、文件创建时间、文件大小、文件CRC32校验码和随机数；每个存储目录下有两个256*256个子目录，storage会按文件file_id进行两次hash，路由到其中一个子目录，然后将文件file_id为文件名存储在该子目录下，最后生成文件路径: group名称、虚拟磁盘路径、数据两级目录、file_id

  1. group1 /M00/02/44/wkgDRe348wAAAAGKYJK42378.sh

其中，组名: 上传文件后所在的存储组的名称，在文件上传成功后由存储服务器返回，需要客户端自行保存

虚拟磁盘路径: 存储服务器配置的虚拟路径，与磁盘选项store_path*参数对应

数据两级目录: 存储服务器在每个虚拟磁盘路径下创建的两级目录，用于存储数据文件

同步机制

1.新增tracker服务器数据同步

由于storage server上配置了所有的tracker server，storage server和tracker server之间的通信是由storage server主动发起的，storage server为每台tracker server 启动一个线程进行通信；在通信过程中，若发现tracker server返回的本组storage server列表比本机记录少，就会将该tracker server上没有的storage server同步给该tracker，这样的机制使得tracker之间是对等的关系，数据保持一致

2.组内新增storage数据同步

若新增storage server或者其状态发生变化，tracker server都会将storage server列表同步给该组内所有storage server；以新增storage server为例，新加入storage server会主动连接tracker server，tracker server发现有新的storage server加入，就会将该组内所有的storage server返回给新加入的storage server，并重新将该组的storage server列表返回给该组内的其他storage server

3.组内storage数据同步

组内storage server之间是对等的，文件上传，删除等操作可以在组内任意一台storage server上进行。文件同步只能在同组内的storage server之间进行，采用push方式，即源服务器同步到目标服务器

A. 只在同组内的storage server之间同步

B. 源数据才需要同步，备份数据不再同步

C. 特例: 新增storage server时，由其中一台将已有的所有数据(包括源数据和备份数据)同步到新增服务器

storage server 7种状态

通过命令fdfs_monitor /etc/fdfs/client.conf可以查看ip_addr选项显示storage server当前状态

  1. INIT 初始化，尚未得到同步已有数据的源服务器
  2. WAIT_SYNC 等待同步，已得到同步已有数据的源服务器
  3. SYNCING 同步中
  4. DELETE 已删除，该服务器从本组中摘除
  5. OFFLINE 离线
  6. ONLINE 在线，尚不能提供服务
  7. ACTIVE 在线，可以提供服务

组内增加storage server A状态变化过程:

1.storage server A主动连接tracker server，此时tracker server将storage serverA状态设置为INIT

2.storage server A向tracker server询问追加同步的源服务器和追加同步截止时间点(当前时间)，若组内只有storage server A或者上传文件数为0，则告诉新主机不需要同步数据，storage serverA状态为ONLINE；若组内没有active状态及其，就返回错误给新机器，新机器重新尝试；否则tracker将其状态设置为WAIT_SYNC

3.假如分配了storage server B为同步源服务器和截止时间点，那么storage serverB将会截止时间点之前的所有数据同步给storage server A，并请求tracker设置storage server A状态为SYNCING；到了截止时间后，storage server B向storage server A的同步将由追加同步切换为正常binlog增量同步，当获取不到更多binlog时，请求tracker将storage server A同步完所有数据，暂时没有数据要同步时，storage server B请求tracker server将storage server A的状态设置为ONLINE

4.storage server B向storage server A同步完所有数据，暂时没有数据要同步时，storage server B请求tracker server将 storage server A的状态设置为ONLINE

5.当storage server A向tracker server发起心跳时，tracker server将其状态更改为ACTIVE，之后就是增量同步(binlog)

image_1ec2fv4og23cn1j1m58pvr12rk1g

注: 整个源同步班过程是源机器启动弄一个同步线程，将数据Push到新机器，最大达到一个磁盘的IO，不能并发；由于源同步截止条件是获取不到binlog，系统繁忙，不断有新数据写入的情况，将会导致一直无法完成源同步

下载

image_1ec2g5nompfd7e815vq1u0a1g491t

client发送下载请求给某个tracker，必须带上文件名信息，tracker从文件名中解析出文件的group、大小、创建时间等信息，然后为该请求选择一个storage用于读请求；由于group内的文件同步是异步进行，可能出现文件没有同步到其他storage server上或者延迟的问题，可以使用nginx_fastdfs_module模块解决

image_1ec2gftjjh28idtrvv1reqd792a

关于文件去重

由于FastDFS本身不能对重复上传的文件进行去重，而FastDHT可以做到去重。FastDHT是一个高性能的分布式哈希系统，它是基于键值对存储的，而且它需要依赖于Berkeley DB作为数据存储的媒介，同时需要依赖于libfastcommon

由于业务需要，目前不存在文件去重的时候，如果需要可以自己简单了解一下FastDHT

FastDFS-Nginx扩展模块源码分析

安装FastDFS集群

本次环境架构

image_1ec84b7b91s9hrhr1gdk1m0o1pg79

针对tracker.conf && storage.conf && mod_fastdfs.conf有一篇单独的文章介绍相关参数。有兴趣的可以看一下，也可以直接看默认的配置文件，对每个参数都有介绍

fastdfs111

fastdfs111

 

环境说明

  1. #nginx这里可以部署2台，加上keepliveed作高可用，由于我这里机器不足，就使用单台nginx进行代理
  2. nginx 192.168.31.100 nginx
  3.  
  4.  
  5. tracker 节点
  6. tracker 01:192.168.31.101 FastDFS,libfastcommon,nginx,ngx_cache_purge
  7. tracker 02:192.168.31.102 FastDFS,libfastcommon,nginx,ngx_cache_purge
  8. #其中tracker不提供存储
  9.  
  10.  
  11. Storage 节点
  12. [group1]
  13. storage 01:192.168.31.103 FastDFS,libfastcommon,nginx,fastdfs–nginx–module
  14. storage 02:192.168.31.104 FastDFS,libfastcommon,nginx,fastdfs–nginx–module
  15.  
  16. [group2]
  17. storage 03:192.168.31.105 FastDFS,libfastcommon,nginx,fastdfs–nginx–module
  18. storage 04:192.168.31.106 FastDFS,libfastcommon,nginx,fastdfs–nginx–module

1.所有的服务器都需要安装nginx，主要是用于访问和上传无关；

2.tracker安装nginx主要为了提供http反向代理、负载均衡以及缓存服务

3.每一台storage服务器部署Nginx及FastDFS扩展模块，主要用于对storage存储的文件提供http下载访问，仅当前storage节点找不到文件时会向源storage主机发送rediect或者proxy动作

所有节点安装

关闭防火墙，selinux

  1. systemctl stop firewalld
  2. systemctl disable firewalld
  3. iptables –F && iptables –X && iptables –F –t nat && iptables –X –t nat
  4. iptables –P FORWARD ACCEPT
  5. setenforce 0
  6. sed –i ‘s/^SELINUX=.*/SELINUX=disabled/’ /etc/selinux/config

设置yum源

  1. yum install –y wget
  2. wget –O /etc/yum.repos.d/CentOS–Base.repo https://mirrors.aliyun.com/repo/Centos-7.repo
  3. wget –O /etc/yum.repos.d/epel.repo http://mirrors.aliyun.com/repo/epel-7.repo
  4.  
  5. yum clean all
  6. yum makecache

温馨提示：除了nginx节点(192.168.31.100)，其他节点都需要执行安装fastdfs和nginx

安装依赖包 (可解决99%的依赖问题)

  1. yum –y install gcc gcc–c++ make autoconf libtool–ltdl–devel gd–devel freetype–devel libxml2–devel libjpeg–devel libpng–devel openssh–clients openssl–devel curl–devel bison patch libmcrypt–devel libmhash–devel ncurses–devel binutils compat–libstdc++-33 elfutils–libelf elfutils–libelf–devel glibc glibc–common glibc–devel libgcj libtiff pam–devel libicu libicu–devel gettext–devel libaio–devel libaio libgcc libstdc++ libstdc++-devel unixODBC unixODBC–devel numactl–devel glibc–headers sudo bzip2 mlocate flex lrzsz sysstat lsof setuptool system–config–network–tui system–config–firewall–tui ntsysv ntp pv lz4 dos2unix unix2dos rsync dstat iotop innotop mytop telnet iftop expect cmake nc gnuplot screen xorg–x11–utils xorg–x11–xinit rdate bc expat–devel compat–expat1 tcpdump sysstat man nmap curl lrzsz elinks finger bind–utils traceroute mtr ntpdate zip unzip vim wget net–tools

下载依赖包 （除了nginx节点，其他节点都要安装）

  1. mkdir /root/tools/
  2. cd /root/tools
  3. wget http://nginx.org/download/nginx-1.18.0.tar.gz
  4. wget https://github.com/happyfish100/libfastcommon/archive/V1.0.43.tar.gz
  5. wget https://github.com/happyfish100/fastdfs/archive/V6.06.tar.gz
  6. wget https://github.com/happyfish100/fastdfs-nginx-module/archive/V1.22.tar.gz
  7.  
  8.  
  9. #为了保证文章可用性，本次软件包已经进行备份，下载地址如下
  10. mkdir /root/tools/
  11. cd /root/tools
  12. wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz
  13. wget http://down.i4t.com/fdfs/v6.6/V1.0.43.tar.gz
  14. wget http://down.i4t.com/fdfs/v6.6/V6.06.tar.gz
  15. wget http://down.i4t.com/fdfs/v6.6/V1.22.tar.gz
  16.  
  17.  
  18.  
  19. #解压
  20. cd /root/tools
  21. tar xf nginx–1.18.0.tar.gz
  22. tar xf V1.0.43.tar.gz
  23. tar xf V1.22.tar.gz
  24. tar xf V6.06.tar.gz

安装libfastcommon (除了nginx节点，其他节点都要安装)

  1. cd /root/tools/libfastcommon–1.0.43
  2. ./make.sh
  3. ./make.sh install

安装FastDFS (除了nginx节点，其他节点都要安装)

  1. cd /root/tools/fastdfs–6.06/
  2. ./make.sh
  3. ./make.sh install

拷贝配置文件 （tracker01 02节点）

  1. [root@tracker01 fastdfs–6.06]# cp /etc/fdfs/tracker.conf.sample /etc/fdfs/tracker.conf #tracker节点
  2. [root@01 fastdfs–6.06]# cp /etc/fdfs/client.conf.sample /etc/fdfs/client.conf #客户端文件（测试使用）
  3. [root@01 fastdfs–6.06]# cp /root/tools/fastdfs–6.06/conf/http.conf /etc/fdfs/ #nginx配置文件
  4. [root@01 fastdfs–6.06]# cp /root/tools/fastdfs–6.06/conf/mime.types /etc/fdfs/ #nginx配置文件

配置tracker 01节点

这里可以先配置一台节点，没有问题在启动另外的节点

创建tracker数据存储及日志目录 （需要在tracker节点执行）

  1. mkdir /data/tracker/ –p

修改配置文件 （tracker 01节点执行）

  1. cat >/etc/fdfs/tracker.conf <<EOF
  2. disabled = false
  3. bind_addr =
  4. port = 22122
  5. connect_timeout = 5
  6. network_timeout = 60
  7. base_path = /data/tracker
  8. max_connections = 1024
  9. accept_threads = 1
  10. work_threads = 4
  11. min_buff_size = 8KB
  12. max_buff_size = 128KB
  13. store_lookup = 0
  14. store_server = 0
  15. store_path = 0
  16. download_server = 0
  17. reserved_storage_space = 20%
  18. log_level = info
  19. run_by_group=
  20. run_by_user =
  21. allow_hosts = *
  22. sync_log_buff_interval = 1
  23. check_active_interval = 120
  24. thread_stack_size = 256KB
  25. storage_ip_changed_auto_adjust = true
  26. storage_sync_file_max_delay = 86400
  27. storage_sync_file_max_time = 300
  28. use_trunk_file = false
  29. slot_min_size = 256
  30. slot_max_size = 1MB
  31. trunk_alloc_alignment_size = 256
  32. trunk_free_space_merge = true
  33. delete_unused_trunk_files = false
  34. trunk_file_size = 64MB
  35. trunk_create_file_advance = false
  36. trunk_create_file_time_base = 02:00
  37. trunk_create_file_interval = 86400
  38. trunk_create_file_space_threshold = 20G
  39. trunk_init_check_occupying = false
  40. trunk_init_reload_from_binlog = false
  41. trunk_compress_binlog_min_interval = 86400
  42. trunk_compress_binlog_interval = 86400
  43. trunk_compress_binlog_time_base = 03:00
  44. trunk_binlog_max_backups = 7
  45. use_storage_id = false
  46. storage_ids_filename = storage_ids.conf
  47. id_type_in_filename = id
  48. store_slave_file_use_link = false
  49. rotate_error_log = false
  50. error_log_rotate_time = 00:00
  51. compress_old_error_log = false
  52. compress_error_log_days_before = 7
  53. rotate_error_log_size = 0
  54. log_file_keep_days = 0
  55. use_connection_pool = true
  56. connection_pool_max_idle_time = 3600
  57. http.server_port = 8080
  58. http.check_alive_interval = 30
  59. http.check_alive_type = tcp
  60. http.check_alive_uri = /status.html
  61. EOF

启动tracker

  1. [root@01 fastdfs–6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

配置tracker02节点

拷贝配置文件

  1. scp –r /etc/fdfs/tracker.conf root@192.168.31.102:/etc/fdfs/
  2. ssh root@192.168.31.102 mkdir /data/tracker/ –p

tracker02启动tracker

  1. [root@02 fastdfs–6.06]# /usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start

检查启动状态

  1. netstat –lntup|grep 22122
  2. tcp 0 0 0.0.0.0:22122 0.0.0.0:* LISTEN 108126/fdfs_tracker

如果启动失败可以查看tracker报错

  1. tail –f /data/tracker/logs/trackerd.log

接下来编辑启动脚本

  1. cat > /usr/lib/systemd/system/tracker.service <<EOF
  2. [Unit]
  3. Description=The FastDFS File server
  4. After=network.target remote–fs.target nss–lookup.target
  5.  
  6. [Service]
  7. Type=forking
  8. ExecStart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf start
  9. ExecStop=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf stop
  10. ExecRestart=/usr/bin/fdfs_trackerd /etc/fdfs/tracker.conf restart
  11.  
  12. [Install]
  13. WantedBy=multi–user.target
  14. EOF
  15.  
  16.  
  17. $ systemctl daemon–reload
  18. $ systemctl start tracker
  19. $ systemctl enable tracker
  20. $ systemctl status tracker
  21.  
  22. #需要先手动kill 掉tracker

storage 01-02节点配置

storage01节点和02节点配置相同，storage03和storage04配置相同

storage01和storage02节点属于group1组

创建storage 数据存储目录

  1. mkdir /data/fastdfs_data –p

修改配置文件

  1. cat >/etc/fdfs/storage.conf<<EOF
  2. disabled = false
  3. group_name = group1
  4. bind_addr =
  5. client_bind = true
  6. port = 23000
  7. connect_timeout = 5
  8. network_timeout = 60
  9. heart_beat_interval = 30
  10. stat_report_interval = 60
  11. base_path = /data/fastdfs_data
  12. max_connections = 1024
  13. buff_size = 256KB
  14. accept_threads = 1
  15. work_threads = 4
  16. disk_rw_separated = true
  17. disk_reader_threads = 1
  18. disk_writer_threads = 1
  19. sync_wait_msec = 50
  20. sync_interval = 0
  21. sync_start_time = 00:00
  22. sync_end_time = 23:59
  23. write_mark_file_freq = 500
  24. disk_recovery_threads = 3
  25. store_path_count = 1
  26. store_path0 = /data/fastdfs_data
  27. subdir_count_per_path = 256
  28. tracker_server = 192.168.31.101:22122
  29. tracker_server = 192.168.31.102:22122
  30. log_level = info
  31. run_by_group =
  32. run_by_user =
  33. allow_hosts = *
  34. file_distribute_path_mode = 0
  35. file_distribute_rotate_count = 100
  36. fsync_after_written_bytes = 0
  37. sync_log_buff_interval = 1
  38. sync_binlog_buff_interval = 1
  39. sync_stat_file_interval = 300
  40. thread_stack_size = 512KB
  41. upload_priority = 10
  42. if_alias_prefix =
  43. check_file_duplicate = 0
  44. file_signature_method = hash
  45. key_namespace = FastDFS
  46. keep_alive = 0
  47. use_access_log = false
  48. rotate_access_log = false
  49. access_log_rotate_time = 00:00
  50. compress_old_access_log = false
  51. compress_access_log_days_before = 7
  52. rotate_error_log = false
  53. error_log_rotate_time = 00:00
  54. compress_old_error_log = false
  55. compress_error_log_days_before = 7
  56. rotate_access_log_size = 0
  57. rotate_error_log_size = 0
  58. log_file_keep_days = 0
  59. file_sync_skip_invalid_record = false
  60. use_connection_pool = true
  61. connection_pool_max_idle_time = 3600
  62. compress_binlog = true
  63. compress_binlog_time = 01:30
  64. check_store_path_mark = true
  65. http.domain_name =
  66. http.server_port = 80
  67. EOF
  68.  
  69. #注意: 需要修改tracker_server地址，多个节点多复制几行，一个节点写一行就可以。 不建议单节点使用localhost

配置启动文件

  1. cat >/usr/lib/systemd/system/storage.service <<EOF
  2.  
  3. [Unit]
  4. Description=The FastDFS File server
  5. After=network.target remote–fs.target nss–lookup.target
  6.  
  7. [Service]
  8. Type=forking
  9. ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
  10. ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
  11. ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
  12.  
  13. [Install]
  14. WantedBy=multi–user.target
  15. EOF
  16.  
  17. systemctl daemon–reload
  18. systemctl start storage
  19. systemctl status storage
  20. systemctl enable storage

检查启动状态

  1. netstat –lntup|grep 23000

如果出现启动失败，可以到设置的目录查看一下log

  1. tail –f /data/fastdfs_data/logs/storaged.log

storage 02-03节点配置

storage03和storage04节点属于group2组

基本流程不变这里只说明需要修改的地方

在storage03-storage-04节点同步执行

  1. 创建storage 数据存储目录
  2. mkdir /data/fastdfs_data –p
  3.  
  4. 修改配置文件
  5. cat >/etc/fdfs/storage.conf<<EOF
  6. disabled = false
  7. group_name = group2
  8. bind_addr =
  9. client_bind = true
  10. port = 23000
  11. connect_timeout = 5
  12. network_timeout = 60
  13. heart_beat_interval = 30
  14. stat_report_interval = 60
  15. base_path = /data/fastdfs_data
  16. max_connections = 1024
  17. buff_size = 256KB
  18. accept_threads = 1
  19. work_threads = 4
  20. disk_rw_separated = true
  21. disk_reader_threads = 1
  22. disk_writer_threads = 1
  23. sync_wait_msec = 50
  24. sync_interval = 0
  25. sync_start_time = 00:00
  26. sync_end_time = 23:59
  27. write_mark_file_freq = 500
  28. disk_recovery_threads = 3
  29. store_path_count = 1
  30. store_path0 = /data/fastdfs_data
  31. subdir_count_per_path = 256
  32. tracker_server = 192.168.31.101:22122
  33. tracker_server = 192.168.31.102:22122
  34. log_level = info
  35. run_by_group =
  36. run_by_user =
  37. allow_hosts = *
  38. file_distribute_path_mode = 0
  39. file_distribute_rotate_count = 100
  40. fsync_after_written_bytes = 0
  41. sync_log_buff_interval = 1
  42. sync_binlog_buff_interval = 1
  43. sync_stat_file_interval = 300
  44. thread_stack_size = 512KB
  45. upload_priority = 10
  46. if_alias_prefix =
  47. check_file_duplicate = 0
  48. file_signature_method = hash
  49. key_namespace = FastDFS
  50. keep_alive = 0
  51. use_access_log = false
  52. rotate_access_log = false
  53. access_log_rotate_time = 00:00
  54. compress_old_access_log = false
  55. compress_access_log_days_before = 7
  56. rotate_error_log = false
  57. error_log_rotate_time = 00:00
  58. compress_old_error_log = false
  59. compress_error_log_days_before = 7
  60. rotate_access_log_size = 0
  61. rotate_error_log_size = 0
  62. log_file_keep_days = 0
  63. file_sync_skip_invalid_record = false
  64. use_connection_pool = true
  65. connection_pool_max_idle_time = 3600
  66. compress_binlog = true
  67. compress_binlog_time = 01:30
  68. check_store_path_mark = true
  69. http.domain_name =
  70. http.server_port = 80
  71. EOF
  72.  
  73.  
  74. #配置启动文件
  75. cat >/usr/lib/systemd/system/storage.service <<EOF
  76.  
  77. [Unit]
  78. Description=The FastDFS File server
  79. After=network.target remote–fs.target nss–lookup.target
  80.  
  81. [Service]
  82. Type=forking
  83. ExecStart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
  84. ExecStop=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
  85. ExecRestart=/usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart
  86.  
  87. [Install]
  88. WantedBy=multi–user.target
  89. EOF
  90.  
  91. systemctl daemon–reload
  92. systemctl start storage
  93. systemctl status storage
  94. systemctl enable storage
  95.  
  96.  
  97. #检查启动状态
  98. netstat –lntup|grep 23000

如果出现systemctl启动失败，可以使用命令启动，在根据日志进行查看。 大概启动时间为10s

  1. #storage启动、停止、重启命令
  2. /usr/bin/fdfs_storaged /etc/fdfs/storage.conf start
  3. /usr/bin/fdfs_storaged /etc/fdfs/storage.conf stop
  4. /usr/bin/fdfs_storaged /etc/fdfs/storage.conf restart

所有节点storage启动完毕后进行检查，是否可以获取到集群信息 (刚创建的集群比较慢，稍等一会。需要等待状态为ACTIVE即可)

  1. #在任意节点storage节点执行命令都可以，获取结果应该如下
  2. [root@storage01 fdfs]# fdfs_monitor /etc/fdfs/storage.conf list
  3.  
  4. [2020–07–03 01:15:25] DEBUG – base_path=/data/fastdfs_data, connect_timeout=5, network_timeout=60, tracker_server_count=2, anti_steal_token=0, anti_steal_secret_key length=0, use_connection_pool=1, g_connection_pool_max_idle_time=3600s, use_storage_id=0, storage server id count: 0
  5.  
  6. server_count=2, server_index=1
  7.  
  8. tracker server is 192.168.31.102:22122 #tracker server处理本次命令的节点
  9.  
  10. group count: 2 #group组数量
  11.  
  12. Group 1: #group1组信息
  13. group name = group1
  14. disk total space = 17,394 MB
  15. disk free space = 13,758 MB
  16. trunk free space = 0 MB
  17. storage server count = 2
  18. active server count = 2
  19. storage server port = 23000
  20. storage HTTP port = 80
  21. store path count = 1
  22. subdir count per path = 256
  23. current write server index = 0
  24. current trunk file id = 0
  25.  
  26. Storage 1: #storage1节点信息
  27. id = 192.168.31.103
  28. ip_addr = 192.168.31.103 ACTIVE #storage节点状态
  29. http domain =
  30. version = 6.06 #fdfs 版本
  31. join time = 2020–07–03 01:08:29 #加入集群时间
  32. up time = 2020–07–03 01:08:29
  33. total storage = 17,394 MB
  34. free storage = 14,098 MB
  35. upload priority = 10
  36. store_path_count = 1
  37. subdir_count_per_path = 256
  38. storage_port = 23000
  39. storage_http_port = 80
  40. current_write_path = 0
  41. source storage id = 192.168.31.104
  42. if_trunk_server = 0
  43. connection.alloc_count = 256
  44. connection.current_count = 1
  45. ……………省略号……………………….
  46. last_heart_beat_time = 2020–07–03 01:15:18
  47. last_source_update = 1970–01–01 08:00:00
  48. last_sync_update = 1970–01–01 08:00:00
  49. last_synced_timestamp = 1970–01–01 08:00:00
  50. Storage 2: #storage2节点信息
  51. id = 192.168.31.104 #storage 节点IP
  52. ip_addr = 192.168.31.104 ACTIVE #storage 节点状态
  53. http domain =
  54. version = 6.06 #storage 节点版本
  55. join time = 2020–07–03 01:08:26 #加入时间
  56. up time = 2020–07–03 01:08:26
  57. total storage = 17,394 MB
  58. free storage = 13,758 MB
  59. upload priority = 10
  60. store_path_count = 1
  61. ……………省略号……………………….
  62. last_heart_beat_time = 2020–07–03 01:15:17
  63. last_source_update = 1970–01–01 08:00:00
  64. last_sync_update = 1970–01–01 08:00:00
  65. last_synced_timestamp = 1970–01–01 08:00:00
  66.  
  67. Group 2: #group2集群信息
  68. group name = group2
  69. disk total space = 17,394 MB
  70. disk free space = 15,538 MB
  71. trunk free space = 0 MB
  72. storage server count = 2
  73. active server count = 2
  74. storage server port = 23000
  75. storage HTTP port = 80
  76. store path count = 1
  77. subdir count per path = 256
  78. current write server index = 0
  79. current trunk file id = 0
  80.  
  81. Storage 1: #storage1节点信息
  82. id = 192.168.31.105
  83. ip_addr = 192.168.31.105 ACTIVE
  84. http domain =
  85. version = 6.06
  86. join time = 2020–07–03 01:13:42
  87. up time = 2020–07–03 01:13:42
  88. total storage = 17,394 MB
  89. free storage = 15,538 MB
  90. upload priority = 10
  91. store_path_count = 1
  92. subdir_count_per_path = 256
  93. storage_port = 23000 #storage端口
  94. storage_http_port = 80
  95. current_write_path = 0
  96. source storage id =
  97. if_trunk_server = 0
  98. connection.alloc_count = 256
  99. connection.current_count = 1
  100. ……………省略号……………………….
  101. last_heart_beat_time = 2020–07–03 01:15:22
  102. last_source_update = 1970–01–01 08:00:00
  103. last_sync_update = 1970–01–01 08:00:00
  104. last_synced_timestamp = 1970–01–01 08:00:00
  105. Storage 2:
  106. id = 192.168.31.106
  107. ip_addr = 192.168.31.106 ACTIVE
  108. http domain =
  109. version = 6.06
  110. join time = 2020–07–03 01:14:05
  111. up time = 2020–07–03 01:14:05
  112. total storage = 17,394 MB
  113. free storage = 15,538 MB
  114. upload priority = 10
  115. store_path_count = 1
  116. subdir_count_per_path = 256
  117. storage_port = 23000
  118. storage_http_port = 80
  119. current_write_path = 0
  120. source storage id = 192.168.31.105
  121. if_trunk_server = 0
  122. connection.alloc_count = 256
  123. connection.current_count = 1
  124. connection.max_count = 1
  125. total_upload_count = 0
  126. ……………省略号……………………….
  127. total_file_write_count = 0
  128. success_file_write_count = 0
  129. last_heart_beat_time = 2020–07–03 01:15:10
  130. last_source_update = 1970–01–01 08:00:00
  131. last_sync_update = 1970–01–01 08:00:00
  132. last_synced_timestamp = 1970–01–01 08:00:00

配置client

这里在tracker01节点配置celient客户端 (其他节点可不配置,client.conf为可选配置)

  1. mkdir –p /data/fdfs_client/logs #日志存放路径
  2.  
  3. cat >/etc/fdfs/client.conf <<EOF
  4. connect_timeout = 5
  5. network_timeout = 60
  6. base_path = /data/fdfs_client/logs
  7. tracker_server = 192.168.31.101:22122
  8. tracker_server = 192.168.31.102:22122
  9. log_level = info
  10. use_connection_pool = false
  11. connection_pool_max_idle_time = 3600
  12. load_fdfs_parameters_from_tracker = false
  13. use_storage_id = false
  14. storage_ids_filename = storage_ids.conf
  15. http.tracker_server_port = 80
  16. EOF
  17.  
  18. #需要修改tracker_server地址

上传文件测试，这里的文件是init.yaml

  1. [root@01 ~]# echo “test” >init.yaml
  2. [root@01 ~]# fdfs_upload_file /etc/fdfs/client.conf init.yaml
  3. group2/M00/00/00/wKgfaV7–GG2AQcpMAAAABTu5NcY98.yaml

Storage节点安装Nginx

所有storage节点mod_fastdfs.conf配置如下

  1. cat >/etc/fdfs/mod_fastdfs.conf <<EOF
  2. connect_timeout=2
  3. network_timeout=30
  4. base_path=/tmp
  5. load_fdfs_parameters_from_tracker=true
  6. storage_sync_file_max_delay = 86400
  7. use_storage_id = false
  8. storage_ids_filename = storage_ids.conf
  9. tracker_server=192.168.31.101:22122
  10. tracker_server=192.168.31.102:22122
  11. storage_server_port=23000
  12. url_have_group_name = true
  13. store_path_count=1
  14. log_level=info
  15. log_filename=
  16. response_mode=proxy
  17. if_alias_prefix=
  18. flv_support = true
  19. flv_extension = flv
  20. group_count = 2
  21. #include http.conf
  22.  
  23. [group1]
  24. group_name=group1
  25. storage_server_port=23000
  26. store_path_count=1
  27. store_path0=/data/fastdfs_data
  28.  
  29. [group2]
  30. group_name=group2
  31. storage_server_port=23000
  32. store_path_count=1
  33. store_path0=/data/fastdfs_data
  34. EOF

拷贝相关依赖 (以下是所有storage节点安装)

  1. cp /root/tools/fastdfs–6.06/conf/http.conf /etc/fdfs/
  2. cp /root/tools/fastdfs–6.06/conf/mime.types /etc/fdfs/

安装Nginx依赖包

  1. yum install –y gcc glibc gcc–c++ prce–devel openssl–devel pcre–devel lua–devel libxml2 libxml2–devel libxslt–devel perl–ExtUtils–Embed GeoIP GeoIP–devel GeoIP–data zlib zlib–devel openssl pcre pcre–devel gcc g++ gcc–c++ gd–devel

创建nginx用户

  1. useradd –s /sbin/nologin nginx –M

编译nginx

  1. cd /root/tools/nginx–1.18.0
  2. ./configure —prefix=/usr/local/nginx–1.18 —with–http_ssl_module —user=nginx —group=nginx —with–http_sub_module —add–module=/root/tools/fastdfs–nginx–module–1.22/src
  3.  
  4. make && make install
  5.  
  6. ln –s /usr/local/nginx–1.18 /usr/local/nginx

修改Nginx配置文件

  1. cat > /usr/local/nginx/conf/nginx.conf <<EOF
  2. worker_processes 1;
  3. events {
  4. worker_connections 1024;
  5. }
  6.  
  7. http {
  8. include mime.types;
  9. default_type application/octet–stream;
  10. sendfile on;
  11. keepalive_timeout 65;
  12.  
  13.  
  14. server {
  15. listen 8888;
  16. server_name localhost;
  17.  
  18. location ~/group[0-9]/M00 {
  19. root /data/fastdfs_data;
  20. ngx_fastdfs_module;
  21. }
  22. }
  23.  
  24. }
  25. EOF

启动nginx

  1. /usr/local/nginx/sbin/nginx –t
  2. /usr/local/nginx/sbin/nginx

在storage节点上8888的请求且有group的都转给ngx_fastdfs_module插件处理

接下来我们手动上传两张图片进行测试

  1. [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf abcdocker.png
  2. group1/M00/00/00/wKgfZ17–JkKAYX–SAABc0HR4eEs313.png
  3.  
  4. [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg
  5. group2/M00/00/00/wKgfal7–JySABmgLAABdMoE–LPo504.jpg

目前我们tracker属于轮训机制，会轮训group1和group2；具体使用参数可以参考下面的文章

接下来我们可以通过浏览器访问,不同的组对应不同的项目，FastDFS集群可以有多个组，但是每台机器只可以有一个storage

  1. http://storage1节点:8888/group1/M00/00/00/wKgfZ17-JkKAYX-SAABc0HR4eEs313.png
  2. http://storage1节点:8888/group2/M00/00/00/wKgfal7-JySABmgLAABdMoE-LPo504.jpg

经过我的测试，即使我们把图片上传到group1中，在group2上面直接访问也可以访问成功，但是在group2的存储目录并没有找到图片文件。原因如下

  1. #Nginx日志
  2. 192.168.31.174 – – [03/Jul/2020:03:03:31 +0800] “GET /group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg HTTP/1.1” 304 0 “-“ “Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:77.0) Gecko/20100101 Firefox/77.0”

为何我们其他storage节点也可以访问，原因是nginx中fastdfs-nginx-module模块可以重定向文件连接到源服务器取文件

补充: FastDFS常用命令参数

  1. #查看集群状态
  2. fdfs_monitor /etc/fdfs/storage.conf
  3.  
  4. #上传
  5. fdfs_upload_file /etc/fdfs/client.conf abcdocker.png
  6. #下载
  7. fdfs_download_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17–JkKAYX–SAABc0HR4eEs313.png
  8. #查看文件属性
  9. fdfs_file_info /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17–JkKAYX–SAABc0HR4eEs313.png
  10.  
  11. #删除文件
  12. fdfs_delete_file /etc/fdfs/client.conf group1/M00/00/00/wKgfZ17–JkKAYX–SAABc0HR4eEs313.png
  13.  
  14. #删除一个storage
  15. /usr/local/bin/fdfs_monitor /etc/fdfs/storage.conf delete group2 192.168.31.105

Tracker配置高可用

在tracker上安装的nginx主要为了提供http访问的反向代理、负载均衡和缓存服务

这里我们tracker01 02同时进行安装即可

  1. #下载nginx依赖包
  2. mkdir /root/tools –p
  3. cd /root/tools
  4. wget http://down.i4t.com/ngx_cache_purge-2.3.tar.gz
  5. wget http://down.i4t.com/fdfs/v6.6/nginx-1.18.0.tar.gz
  6. tar xf ngx_cache_purge–2.3.tar.gz
  7. tar xf nginx–1.18.0.tar.gz
  8.  
  9. #创建nginx用户
  10. useradd –s /sbin/nologin –M nginx
  11.  
  12. #编译nginx
  13. cd /root/tools/nginx–1.18.0
  14. ./configure —prefix=/usr/local/nginx–1.18 —with–http_ssl_module —user=nginx —group=nginx —with–http_sub_module —add–module=/root/tools/ngx_cache_purge–2.3
  15.  
  16. make && make install
  17.  
  18. ln –s /usr/local/nginx–1.18 /usr/local/nginx

nginx安装完毕，接下来配置nginx.conf

tracker 中nginx节点可以不是80，我这里以80位代表

mkdir /data/nginx_cache -p

 

$ vim /usr/local/nginx/conf/nginx.conf

worker_processes 1;

events {

 worker_connections 1024;

}

 

 

http {

 include mime.types;

 default_type application/octet-stream;

 sendfile on;

 keepalive_timeout 65;

 server_names_hash_bucket_size 128;

 client_header_buffer_size 32k;

 large_client_header_buffers 4 32k;

 client_max_body_size 300m;

 proxy_redirect off;

 proxy_set_header Host $http_host;

 proxy_set_header X-Real-IP $remote_addr;

 proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_connect_timeout 90;

 proxy_send_timeout 90;

 proxy_read_timeout 90;

 proxy_buffer_size 16k;

 proxy_buffers 4 64k;

 proxy_busy_buffers_size 128k;

 proxy_temp_file_write_size 128k;

 proxy_cache_path /data/nginx_cache keys_zone=http-cache:100m;

 

upstream fdfs_group1 {

 server 192.168.31.103:8888 weight=1 max_fails=2 fail_timeout=30s;

 server 192.168.80.104:8888 weight=1 max_fails=2 fail_timeout=30s;

}

upstream fdfs_group2 {

 server 192.168.31.105:8888 weight=1 max_fails=2 fail_timeout=30s;

 server 192.168.31.106:8888 weight=1 max_fails=2 fail_timeout=30s;

}

 

 

 server {

 listen 80;

 server_name localhost;

 location /group1/M00 {

 proxy_next_upstream http_502 http_504 error timeout invalid_header;

 proxy_cache http-cache;

 proxy_cache_valid 200 304 12h;

 proxy_cache_key $uri$is_args$args;

 proxy_pass http://fdfs_group1;

 expires 30d;

}

 

 location /group2/M00 {

 proxy_next_upstream http_502 http_504 error timeout invalid_header; proxy_cache http-cache;

 proxy_cache_valid 200 304 12h;

 proxy_cache_key $uri$is_args$args;

 proxy_pass http://fdfs_group2;

 expires 30d;

}

 

}

}

 

/usr/local/nginx/sbin/nginx -t

/usr/local/nginx/sbin/nginx

此时访问tracker01节点和tracker02节点应该都没有问题

  1. http://192.168.31.101/group1/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
  2. http://192.168.31.101/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
  3.  
  4. http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg
  5. http://192.168.31.102/group2/M00/00/00/wKgfaF7-JyKAYde2AABdMoE-LPo633.jpg

效果图如下

image_1ec8ggt1o32s19besoncejeikm

Nginx代理安装

通过上面的步骤，已经可以使用storage节点和tracker节点进行访问，但是为了解决统一管理和tracker高可用，我们还需要使用nginx在去代理tracker

  1. #nginx安装和上面一样，我这里就只更改nginx.conf文件，nginx代理不需要缓存模块，普通安装即可
  2.  
  3. $ vim /usr/local/nginx/conf/nginx.conf
  4.  
  5. worker_processes 1;
  6. events {
  7. worker_connections 1024;
  8. }
  9.  
  10. http {
  11. include mime.types;
  12. default_type application/octet–stream;
  13. sendfile on;
  14. keepalive_timeout 65;
  15.  
  16. upstream fastdfs_tracker {
  17. server 192.168.31.101:80 weight=1 max_fails=2 fail_timeout=30s;
  18. server 192.168.31.102:80 weight=1 max_fails=2 fail_timeout=30s;
  19. }
  20. server {
  21. listen 80;
  22. server_name localhost;
  23.  
  24. location / {
  25. proxy_pass http://fastdfs_tracker/;
  26. }
  27.  
  28. }
  29. }

最后我们在tracker01节点上，测试nginx代理是否都可以访问成功

  1. [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg
  2. group1/M00/00/00/wKgfZ17–Oa2AMuRqAABdMoE–LPo686.jpg
  3.  
  4. [root@tracker01 ~]# fdfs_upload_file /etc/fdfs/client.conf i4t.jpg
  5. group2/M00/00/00/wKgfaV7–Oa6ANXGLAABdMoE–LPo066.jpg
  6.  
  7. 访问查看
  8. [root@tracker01 ~]# curl 192.168.31.100/group1/M00/00/00/wKgfZ17–Oa2AMuRqAABdMoE–LPo686.jpg –I
  9. HTTP/1.1 200 OK
  10. Server: nginx/1.18.0
  11. Date: Thu, 02 Jul 2020 19:49:49 GMT
  12. Content–Type: image/jpeg
  13. Content–Length: 23858
  14. Connection: keep–alive
  15. Last–Modified: Thu, 02 Jul 2020 19:46:53 GMT
  16. Expires: Sat, 01 Aug 2020 19:49:49 GMT
  17. Cache–Control: max–age=2592000
  18. Accept–Ranges: bytes
  19.  
  20. [root@tracker01 ~]# curl 192.168.31.100/group2/M00/00/00/wKgfaV7–Oa6ANXGLAABdMoE–LPo066.jpg –I
  21. HTTP/1.1 200 OK
  22. Server: nginx/1.18.0
  23. Date: Thu, 02 Jul 2020 19:50:17 GMT
  24. Content–Type: image/jpeg
  25. Content–Length: 23858
  26. Connection: keep–alive
  27. Last–Modified: Thu, 02 Jul 2020 19:46:54 GMT
  28. Expires: Sat, 01 Aug 2020 19:50:17 GMT
  29. Cache–Control: max–age=2592000
  30. Accept–Ranges: bytes

Related Posts:

  1. Kubernetes 1.14 二进制集群安装
  2. Kuerbernetes 1.11 集群二进制安装
  3. CentOS 7 ETCD集群配置大全
  4. 自动化运维工具之–Cobbler

Related Post

群晖/极空间无公网IP搭建Cloudflared-阿里云免费隧道穿透教程

2024年4月6日

群晖NAS网络服务 – 原生 DDNS 部署配置 Cloudflare DDNS 动态域名解析

2022年1月3日

群晖NAS设置Docker中国官方镜像加速教程

2023年11月24日

发表回复