gitlab高可用架構部署文檔

系統環境
操作系統：centos7.6
gitlab版本：12.6.1
數據庫版本：postgresql9.6
redis版本：3.2

架構圖

postgresql數據庫部署

安裝yum源
yum -y install https://download.postgresql.org/pub/repos/yum/9.6/redhat/rhel-7-x86_64/pgdg-redhat96-9.6-3.noarch.rpm

安裝數據庫
yum -y install postgresql96
yum -y install postgresql96-server postgresql96-devel

初始化數據庫
/usr/pgsql-9.6/bin/postgresql96-setup initdb

修改啟動文件名稱
mv /usr/lib/systemd/system/postgresql-9.6.service /usr/lib/systemd/system/postgresql.service

開機自啟動
systemctl enable postgresql

設置環境變量
export PATH=/usr/pgsql-9.6/bin:$PATH
source /etc/profile

啟動數據庫
systemctl start postgresql

修改配置文件
vim /var/lib/pgsql/9.6/data/pg_hba.conf
在最后增加網絡信任
host all all 0.0.0.0/0 trust

vim /var/lib/pgsql/9.6/data/postgresql.conf
修改偵聽地址
listen_addresses = '*'

安裝擴展插件
yum -y install postgresql96-contrib
注：不要使用默認yum install postgresql-contrib安裝，原因：默認安裝版本是9.2，版本太低

重啟服務
systemctl restart postgresql

登錄數據庫進行初始設置
[root@localhost ~]# su - postgres
-bash-4.2$ psql

查看信息
\l
退出數據庫
lq
查看用戶
\du

CREATE USER gitlab WITH PASSWORD 'pass';
CREATE DATABASE gitlabhq_production;
ALTER ROLE gitlab CREATEROLE CREATEDB;
ALTER ROLE gitlab CREATEROLE SUPERUSER;

退出登錄，重新登入
su - postgres

-bash-4.2$ psql gitlabhq_production
psql (9.6.6)
Type "help" for help.

postgres=# CREATE EXTENSION pg_trgm;

退出重啟數據庫服務
systemctl restart postgresql

注：數據庫高可用可參考https://www.cnblogs.com/linkenpark/p/8339936.html
或者官網https://docs.gitlab.com/ee/administration/high_availability/database.html#configure-using-omnibus-for-high-availability

redis安裝部署可參考其他文檔

NFS文件共享

[root@git-235 gitlab]# cat /etc/exports

/var/opt/gitlab/.ssh 172.28.13.0/24(rw,sync,no_root_squash)
/var/opt/gitlab/gitlab-rails/uploads 172.28.13.0/24(rw,sync,no_root_squash)
/var/opt/gitlab/gitlab-rails/shared 172.28.13.0/24(rw,sync,no_root_squash)
/var/opt/gitlab/gitlab-ci/builds 172.28.13.0/24(rw,sync,no_root_squash)
/var/opt/gitlab/git-data 172.28.13.0/24(rw,sync,no_root_squash)

服務端創建文件夾
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data

服務端文件授權
chmod 777 -R /var/opt/

在gitlab應用程序節點創建文件夾
mkdir -p /var/opt/gitlab/.ssh /var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/git-data

在gitlab應用程序節點掛載目錄
mount -t nfs 172.28.13.235:/var/opt/gitlab/.ssh /var/opt/gitlab/.ssh
mount -t nfs 172.28.13.235:/var/opt/gitlab/gitlab-rails/uploads /var/opt/gitlab/gitlab-rails/uploads
mount -t nfs 172.28.13.235:/var/opt/gitlab/gitlab-rails/shared /var/opt/gitlab/gitlab-rails/shared
mount -t nfs 172.28.13.235:/var/opt/gitlab/gitlab-ci/builds /var/opt/gitlab/gitlab-ci/builds
mount -t nfs 172.28.13.235:/var/opt/gitlab/git-data /var/opt/gitlab/git-data

在gitlab應用程序節點修改掛載文件
vim /etc/fstab

gitlab雙節點部署

git01配置
創建gitlab的yum倉庫文件
vim /etc/yum.repos.d/gitlab-ce.repo
[gitlab-ce]
name=Gitlab CE Repository
baseurl=https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el$releasever/
gpgcheck=0
enabled=1

安裝依賴
yum install -y curl policycoreutils-python openssh-server sudo systemctl enable sshd postfix
systemctl enable sshd
systemctl restart sshd
systemctl enable postfix
systemctl restart postfix

yum安裝gitlab-ce
yum install -y gitlab-ce
注：如果無法訪問外網，可以自行下載https://mirrors.tuna.tsinghua.edu.cn/gitlab-ce/yum/el7/gitlab-ce-12.6.1-ce.0.el7.x86_64.rpm

然后使用本地yum安裝
yum localinstall -y gitlab-ce-12.6.1-ce.0.el7.x86_64.rpm

修改配置文件
vim /etc/gitlab/gitlab.rb
external_url 'http://qijian.example.com'
gitlab_rails['time_zone'] = 'Asia/Shanghai'
roles ['application_role']
high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
postgresql['enable'] = false
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "utf8"
gitlab_rails['db_database'] = "gitlabhq_production"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "pass"
gitlab_rails['db_host'] = "172.28.13.235"
gitlab_rails['db_port'] = 5432
redis['enable'] = false
gitlab_rails['redis_host'] = "172.28.13.235"
gitlab_rails['redis_port'] = 6379
gitlab_rails['redis_password'] = "123456"
gitlab_rails['redis_database'] = 0
nginx['enable'] = true

創建啟動服務
gitlab啟動腳本
vim /etc/systemd/system/gitlab.service

[Unit]
Description=gitlab

[Service]
Type=oneshot
RemainAfterExit=yes
ExecStart=/bin/gitlab-ctl start
ExecStop=/bin/gitlab-ctl stop

[Install]
WantedBy=multi-user.target

開機自啟動
systemctl enable gitlab

使配置生效
gitlab-ctl reconfigure

注：如要再次初始化數據，運行 gitlab-rake gitlab:setup（一般前面執行了gitlab-ctl reconfigure已經初始化數據）
gitlab-ctl start

注：常用命令
查看日志：gitlab-ctl tail gitlab-rails
檢測環境：gitlab-rake gitlab:check

git02配置
從git01把/etc/gitlab/gitlab-secrets.json 復制到 git2的/etc/gitlab目錄下
touch /etc/gitlab/skip-auto-migrations

cat /etc/gitlab/gitlab.rb
external_url 'http://qijian.example.com'
gitlab_rails['time_zone'] = 'Asia/Shanghai'
roles ['application_role']
high_availability['mountpoint'] = '/var/opt/gitlab/git-data'
gitlab_rails['auto_migrate'] = false
postgresql['enable'] = false
gitlab_rails['db_adapter'] = "postgresql"
gitlab_rails['db_encoding'] = "utf8"
gitlab_rails['db_database'] = "gitlabhq_production"
gitlab_rails['db_username'] = "gitlab"
gitlab_rails['db_password'] = "pass"
gitlab_rails['db_host'] = "172.28.13.235"
gitlab_rails['db_port'] = 5432
redis['enable'] = false
gitlab_rails['redis_host'] = "172.28.13.235"
gitlab_rails['redis_port'] = 6379
gitlab_rails['redis_password'] = "123456"
gitlab_rails['redis_database'] = 0
nginx['enable'] = true

使配置生效
gitlab-ctl reconfigure

前端負載均衡

前端再分別對http負載（可用nginx反向代理）

[root@git-235 conf.d]# pwd
/etc/nginx/conf.d

[root@git-235 conf.d]# cat qijian.example.com.conf
upstream gitlab_backend {
ip_hash;
server 172.28.13.236:80;
server 172.28.13.237:80;
}
server {
server_name qijian.example.com;

location / {
    proxy_pass http://gitlab_backend;
    proxy_set_header Host       $http_host;
    proxy_set_header x-forwarded-for $proxy_add_x_forwarded_for;
}

}

網頁版訪問測試
http://qijian.example.com/

ssh（可用HAproxy代理，或nginx TCP代理）
yum install haproxy -y

[root@git-235 src]# cat /etc/haproxy/haproxy.cfg
global

log         127.0.0.1 local2

chroot      /var/lib/haproxy
pidfile     /var/run/haproxy.pid
maxconn     4000
user        haproxy
group       haproxy
daemon

# turn on stats unix socket
stats socket /var/lib/haproxy/stats level admin 

defaults
retries 3
timeout client 360s
timeout server 360s
timeout connect 360s
maxconn 32000
option redispatch
option abortonclose
log global
timeout queue 1m
timeout http-request 360s
timeout check 3s

listen admin_stats
bind 0.0.0.0:11011
mode http
maxconn 10
stats refresh 10s
stats uri /web/status
stats auth admin:example
stats hide-version

listen sshd_22
bind 0.0.0.0:22
mode tcp
balance source
server sshd_vxi01 172.28.13.236:22 maxconn 2048 check inter 3000 rise 2 fall 3 weight 1
server sshd_vxi02 172.28.13.237:22 maxconn 2048 check inter 3000 rise 2 fall 3 weight 1

測試驗證
將客戶端密鑰對公鑰添加到gitlab

git clone git@qijian.example.com:chanping/front.git

gitlab集成域控制器AD驗證

修改gitlab應用成程序節點配置文件
gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-'EOS'
main: # 'main' is the GitLab 'provider ID' of this LDAP server
label: 'LDAP'
host: '172.28.15.69'
port: 389
uid: 'sAMAccountName'
bind_dn: 'cn=admin001,cn=users,dc=vxiqijian,dc=com'
password: 'pwdpwd'
encryption: 'plain' # "start_tls" or "simple_tls" or "plain"
verify_certificates: true
active_directory: true
allow_username_or_email_login: false
lowercase_usernames: false
block_auto_created_users: false
base: 'dc=example,dc=com'
user_filter: ''
EOS

使配置生效
gitlab-ctl reconfigure

檢測gitlab與域控驗證
gitlab-rake gitlab:ldap:check

注：如檢測失敗可以重啟服務再次檢測
gitlab-ctl restart

登錄界面

至此，gitlab集群架構部署完成。