最近看到一篇公眾號(hào)講了nginx-ingress-controller的應(yīng)用。下面有人評(píng)論如何做日志持久化，剛好工作上遇到該問(wèn)題，整理一個(gè)方案，僅供參考。
nginx-ingress-controller的日志
nginx-ingress-controller的日志包括三個(gè)部分：
- controller日志： 輸出到stdout，通過(guò)啟動(dòng)參數(shù)中的–log_dir可已配置輸出到文件，重定向到文件后會(huì)自動(dòng)輪轉(zhuǎn)，但不會(huì)自動(dòng)清理
- accesslog：輸出到stdout，通過(guò)nginx-configuration中的字段可以配置輸出到哪個(gè)文件。輸出到文件后不會(huì)自動(dòng)輪轉(zhuǎn)或清理
- errorlog：輸出到stderr，配置方式與accesslog類似。
給controller日志落盤
- 給nginx-ingress-controller掛一個(gè)hostpath： /data/log/nginx_ingress_controller/ 映射到容器里的/var/log/nginx_ingress_controller/ ，
- 給nginx-ingress-controller配置log-dir和logtostderr參數(shù)，將日志重定向到/var/log/nginx_ingress_controller/中。
controller的日志需要做定時(shí)清理。由于controller的日志是通過(guò)klog(k8s.io/klog)輸出的，會(huì)進(jìn)行日志滾動(dòng)，所以我們通過(guò)腳本定時(shí)清理一定時(shí)間之前的日志文件即可。
給nginx日志落盤
修改configmap： nginx-configuration。配置accesslog和errorlog的輸出路徑，替換默認(rèn)的stdout和stderr。輸出路徑我們可以與controller一致，便于查找。
accesslog和errorlog都只有一個(gè)日志文件，我們可以使用logrotate進(jìn)行日志輪轉(zhuǎn)，將輸出到宿主機(jī)上的日志進(jìn)行輪轉(zhuǎn)和清理。配置如：
$ cat /etc/logrotate.d/nginx.log
/data/log/nginx_ingress_controller/access.log {
su root list
rotate 7
daily
maxsize 50M
copytruncate
missingok
create 0644 www-data root
}
官方提供的模板中，nginx-ingress-controller默認(rèn)都是以33這個(gè)用戶登錄啟動(dòng)容器的，因此掛載hostpath路徑時(shí)存在權(quán)限問(wèn)題。我們需要手動(dòng)在機(jī)器上執(zhí)行chown -R 33:33 /data/log/nginx_ingress_controller.
自動(dòng)化ops
nginx日志落盤中，第2、3兩點(diǎn)均需要人工運(yùn)維，有什么解決辦法嗎？
問(wèn)題的關(guān)鍵是：有什么辦法可以在nginx-ingress-controller容器啟動(dòng)之前加一個(gè)hook，將宿主機(jī)的指定目錄執(zhí)行chown呢？
可以用initContainer。initcontainer必須在containers中的容器運(yùn)行前運(yùn)行完畢并成功退出。利用這一k8s特性，我們開(kāi)發(fā)一個(gè)docker image，里面只執(zhí)行如下腳本：
#!/bin/bash
logdir=$LOG_DIR
userID=$USER_ID
echo "try to set dir: $logdir 's group as $userID"
chown -R $userID:$userID $logdir
腳本讀取一些環(huán)境變量， 確認(rèn)需要修改哪個(gè)目錄，改成怎樣的user group。
將腳本打包成dockerimage， 放在nginx-ingress-controller的deploy yaml中，作為initcontainers。 注意要對(duì)該initcontainer配置環(huán)境變量和volumeMount.
再說(shuō)第二點(diǎn)，我們注意到nginx-ingress-controller的基礎(chǔ)鏡像中就自帶了logrotate，那么問(wèn)題就簡(jiǎn)單了，我們將寫好的logrotate配置文件以configmap的形式掛載到容器中就可以了。
一個(gè)deploy yaml如下：
---
apiVersion: v1
kind: Service
metadata:
name: ingress-nginx
namespace: kube-system
spec:
type: ClusterIP
ports:
- name: http
port: 80
targetPort: 80
protocol: TCP
- name: https
port: 443
targetPort: 443
protocol: TCP
selector:
app: ingress-nginx
---
apiVersion: v1
kind: Service
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
ports:
- port: 80
targetPort: 8080
selector:
app: default-http-backend
---
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: default
namespace: kube-system
spec:
backend:
serviceName: default-http-backend
servicePort: 80
---
kind: ConfigMap
apiVersion: v1
metadata:
name: nginx-configuration
namespace: kube-system
labels:
app: ingress-nginx
data:
use-forwarded-headers: "true"
# 此處配置nginx日志的重定向目標(biāo)
access-log-path: /var/log/nginx_ingress_controller/access.log
error-log-path: /var/log/nginx_ingress_controller/error.log
---
# 創(chuàng)建一個(gè)configmap，配置nginx日志的輪轉(zhuǎn)策略，對(duì)應(yīng)的是nginx日志在容器內(nèi)的日志文件
apiVersion: v1
data:
nginx.log: |
{{ user_nginx_log.host_path }}/access.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
{{ user_nginx_log.host_path }}/error.log {
rotate {{ user_nginx_log.rotate_count }}
daily
maxsize {{ user_nginx_log.rotate_size }}
minsize 10M
copytruncate
missingok
create 0644 root root
}
kind: ConfigMap
metadata:
name: nginx-ingress-logrotate
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: tcp-services
namespace: kube-system
---
kind: ConfigMap
apiVersion: v1
metadata:
name: udp-services
namespace: kube-system
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRole
metadata:
name: nginx-ingress-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: Role
metadata:
name: nginx-ingress-role
namespace: kube-system
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: RoleBinding
metadata:
name: nginx-ingress-role-nisa-binding
namespace: kube-system
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: nginx-ingress-role
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: rbac.authorization.k8s.io/v1beta1
kind: ClusterRoleBinding
metadata:
name: nginx-ingress-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: nginx-ingress-clusterrole
subjects:
- kind: ServiceAccount
name: nginx-ingress-serviceaccount
namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: ingress-nginx
namespace: kube-system
spec:
selector:
matchLabels:
app: ingress-nginx
template:
metadata:
labels:
app: ingress-nginx
annotations:
prometheus.io/port: '10254'
prometheus.io/scrape: 'true'
spec:
serviceAccountName: nginx-ingress-serviceaccount
tolerations:
- key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "system/ingress"
operator: In
values:
- "true"
dnsPolicy: ClusterFirstWithHostNet
hostNetwork: true
# 配置initcontainer，確保在nginx-ingress-controller容器啟動(dòng)前將日志目錄的權(quán)限配置好
initContainers:
- name: adddirperm
image: "{{ image_registry.addr }}/{{ image.adddirperm }}"
env:
- name: LOG_DIR
value: /var/log/nginx_ingress_controller
- name: USER_ID
value: "33"
volumeMounts:
- name: logdir
mountPath: /var/log/nginx_ingress_controller
containers:
- name: nginx-ingress-controller
image: "{{ image_registry.addr }}/{{ image.ingress }}"
imagePullPolicy: IfNotPresent
args:
- /nginx-ingress-controller
- --default-backend-service=$(POD_NAMESPACE)/default-http-backend
- --configmap=$(POD_NAMESPACE)/nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/ingress-nginx
- --annotations-prefix=nginx.ingress.kubernetes.io
# 設(shè)置controller日志的輸出路徑和方式
- --log_dir=/var/log/nginx_ingress_controller
- --logtostderr=false
securityContext:
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: https
containerPort: 443
resources:
requests:
cpu: 100m
memory: 256Mi
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
volumeMounts:
# 配置掛載容器中控制器組件和nginx的日志輸出路徑
- name: logdir
mountPath: /var/log/nginx_ingress_controller
# 配置nginx日志的logrotate配置掛載路徑
- name: logrotateconf
mountPath: /etc/logrotate.d/nginx.log
subPath: nginx.log
volumes:
# 控制器組件和nginx的日志輸出路徑為宿主機(jī)的hostpath
- name: logdir
hostPath:
path: {{ user_nginx_log.host_path }}
type: ""
# nginx日志的輪轉(zhuǎn)配置文件來(lái)自于configmap
- name: logrotateconf
configMap:
name: nginx-ingress-logrotate
items:
- key: nginx.log
path: nginx.log
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: default-http-backend
namespace: kube-system
labels:
app: default-http-backend
spec:
selector:
matchLabels:
app: default-http-backend
template:
metadata:
labels:
app: default-http-backend
spec:
terminationGracePeriodSeconds: 60
tolerations:
- key: dedicated
value: ingress-nginx
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "system/ingress"
operator: In
values:
- "true"
containers:
- name: default-http-backend
# Any image is permissible as long as:
# 1. It serves a 404 page at /
# 2. It serves 200 on a /healthz endpoint
image: "{{ image_registry.addr }}/{{ image.http_backend }}"
imagePullPolicy: IfNotPresent
livenessProbe:
httpGet:
path: /healthz
port: 8080
scheme: HTTP
initialDelaySeconds: 30
timeoutSeconds: 5
ports:
- containerPort: 8080
resources:
limits:
cpu: 10m
memory: 20Mi
requests:
cpu: 10m
memory: 20Mi
---
最后，有的人建議將initcontainer去掉，改為基于原有的nginx-ingress-controller鏡像加一層layer，將配置路徑權(quán)限的腳本放在該層執(zhí)行。 個(gè)人認(rèn)為這種方法既不美觀，也不方便。唯一的好處僅在于deploy yaml仍然簡(jiǎn)潔（但少不了volumeMount之類的配置）。不過(guò)還是看個(gè)人使用感受吧~
到此這篇關(guān)于nginx-ingress-controller日志持久化方案的解決的文章就介紹到這了,更多相關(guān)nginx ingress controller日志持久化內(nèi)容請(qǐng)搜索腳本之家以前的文章或繼續(xù)瀏覽下面的相關(guān)文章希望大家以后多多支持腳本之家！
