Jenkins Pipeline為Kubernetes應用部署增加狀態(tài)檢測腳本優(yōu)化
更新時間：2023年12月21日 16:52:05 作者：對你無可奈何
這篇文章主要為大家介紹了Jenkins Pipeline為Kubernetes應用部署增加狀態(tài)檢測腳本優(yōu)化示例詳解,有需要的朋友可以借鑒參考下,希望能夠有所幫助,祝大家多多進步,早日升職加薪
引言
在軟件部署的世界中，Jenkins已經(jīng)成為自動化流程的代名詞。不斷變化的技術(shù)環(huán)境要求我們持續(xù)改進部署流程以滿足現(xiàn)代應用部署的需要。在本篇博客中，作為一位資深運維工程師，我將分享如何將Jenkins Pipeline進化至不僅能支持部署應用直至Running狀態(tài)檢測，同時也能兼顧Deployment和StatefulSet資源的輪詢更新，并詳細介紹滾動更新策略的配置方法。
初始Jenkins Pipeline分析
參照前文：Jenkins Pipeline 腳本優(yōu)化實踐：從繁瑣到簡潔，初始化pipeline如下：
pipeline {
agent none // Use none at the top level, each stage will define its own agent.
environment {
REGISTRY = "swr.cn-north-4.myhuaweicloud.com/master-metaspace"
KUBE_CONFIG = "--namespace=master-metaspace --context=master"
KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
// Assume that 'data' is defined elsewhere or injected as a parameter.
BASE_WORKSPACE = "xxxxxxx" // 定義一個基礎(chǔ)工作空間路徑
}
stages {
stage("GetCode") {
agent { label "build01" }
steps {
script {
checkout scm: [
$class: 'GitSCM',
branches: [[name: env.branchName]],
extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
userRemoteConfigs: [[credentialsId: 'xxxx', url: env.gitHttpURL]]
]
}
}
}
stage("Docker Builds") {
parallel {
stage('Build dataloader-game-ucenter') {
agent { label "build01" }
when { environment name: 'dataloader', value: 'true' }
steps {
buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build datawriter-game-ucenter') {
agent { label "build01" }
when { environment name: 'datawriter', value: 'true' }
steps {
buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build game-ucenter') {
agent { label "build01" }
when { environment name: 'game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
}
}
stage('Development Deployment') {
parallel {
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
}
}
stage("Deploy dataloader-game-ucenter") {
when { environment name: 'dataloader', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("dataloader-game-ucenter")
}
}
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "huaweiyun-xx" }
steps {
deployToKubernetes("game-ucenter")
}
}
}
}
}
}
// Define methods outside pipeline to avoid repetition
def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基礎(chǔ)工作空間變量
sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=$imageName -t $REGISTRY/$imageName:$tag ."
withCredentials([usernamePassword(credentialsId: 'hw-registry', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
sh "docker push $REGISTRY/$imageName:$tag"
}
}
def deployToKubernetes(String kubernetesComponent) {
String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}初始的Jenkins Pipeline定義了一個基本的CI/CD流程，涵蓋了代碼拉取、Docker鏡像構(gòu)建、推送及在Kubernetes環(huán)境中的部署。然而，流程中缺少了對部署狀態(tài)的檢查，這是在確保部署穩(wěn)定性方面至關(guān)重要的一個環(huán)節(jié)。
進化 I：探針引入Deployment部署
現(xiàn)代應用部署不僅僅需要一個“部署到Kubernetes”的指令，更需要在部署后進行健康檢查。對于Deployment類型的應用來說，我們需要在所有Pods運行并處于READY狀態(tài)后才認為部署成功。
狀態(tài)檢測方法介紹
為此，我們引入了checkKubernetesResourceStatus方法來檢查資源的狀態(tài)。該方法通過kubectl的get命令和jsonpath查詢輸出來輪詢檢查ready副本數(shù)。如果指定時間內(nèi)資源不達狀態(tài)，則流程失敗。
Jenkinsfile變更詳解：
引入checkKubernetesResourceStatus方法來檢測deployment各個階段部署的狀態(tài)。
def checkKubernetesResourceStatus(String deploymentName, String namespace) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the deployment status
String statusCheck = sh (
script: "kubectl get deployment ${deploymentName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and greater than 0
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "Deployment ${deploymentName} is ready."
return
} else {
echo "Waiting for Deployment ${deploymentName} to be ready. Attempt ${i}/${attempts}"
sleep sleepTime
}
}
error "Deployment ${deploymentName} did not become ready after ${attempts} attempts"
}以Deploy game-ucenter stage為例：
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("game-ucenter")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace")
}
}game-ucenter模板文件如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: game-ucenter
spec:
replicas: 1
selector:
matchLabels:
app: game-ucenter
template:
metadata:
labels:
app: game-ucenter
spec:
containers:
- name: game-ucenter
image: xxxxxx/xxxx/game-ucenter:{data}
envFrom:
- configMapRef:
name: deploy
ports:
- containerPort: 80
resources:
requests:
memory: "4096M"
cpu: "2000m"
limits:
memory: "4096M"
cpu: "2000m"
livenessProbe:
httpGet:
scheme: HTTP
path: /test.html
port: 80
initialDelaySeconds: 20
periodSeconds: 120
successThreshold: 1
failureThreshold: 3
readinessProbe:
httpGet:
scheme: HTTP
path: /test.html
port: 80
initialDelaySeconds: 20
periodSeconds: 120
imagePullSecrets:
- name: xxxx
---
apiVersion: v1
kind: Service
metadata:
name: game-ucenter
labels:
app: game-ucenter
spec:
ports:
- port: 80
protocol: TCP
targetPort: 80
selector:
app: game-ucenter
嘗試修改以下replicas: 3 重新運行以下pipeline：
也沒有問題，pipeline 腳本有效！
進化 II：兼容StatefulSet的健康檢查
考慮到某些應用可能采用StatefulSet作為工作負載類型，我們必須確保Jenkins Pipeline能夠針對不同的工作負載類型執(zhí)行健康檢查。
狀態(tài)檢測兼容性改進
為了適配StatefulSet，我們對checkKubernetesResourceStatus方法做了略微修改，使其可以接受一個resourceType參數(shù)來區(qū)分資源類型，進而查詢對應的狀態(tài)字段，代碼片段如下：
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the resource status
String statusCheck = sh (
script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and equal to the desired number
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "${resourceType} ${resourceName} is ready."
return
} else {
echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}修改game-ucenter stage:
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "k8s-node-06" }
steps {
deployToKubernetes("game-ucenter")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
}
}創(chuàng)建一個statefulset datawriter-game-ucenter stage:
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "xxxxx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
}
}
注意:我這里截圖還是用了game-ucenter做的測試，其實我想用我的datawriter-game-ucenter，but這個服務是一個node應用沒有沒有livenessProbe readinessProbe，所以截圖我還是使用了game-ucenter!
進化 III：引入滾動更新策略配置和檢測
當我們更新Deployment資源時，通常會采用滾動更新策略，以逐步替換舊Pods，最小化部署時的中斷。
更新策略檢測邏輯
def checkDeploymentUpdateStatus(String deploymentName, String namespace) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
echo "Checking the update status of Deployment: ${deploymentName}"
for (int i = 1; i <= attempts; i++) {
String updateStatus = sh (
script: "kubectl rollout status deployment/${deploymentName} --namespace=${namespace}",
returnStdout: true
).trim()
if (updateStatus.contains("successfully rolled out")) {
echo "Update status: ${updateStatus}"
return
} else {
echo "Waiting for Deployment ${deploymentName} to successfully roll out. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "Deployment ${deploymentName} did not successfully roll out after ${attempts} attempts"
}- 我們增加了
checkDeploymentUpdateStatus方法，該方法通過kubectl命令rollout status監(jiān)控Deployment的更新狀態(tài)。 - 當檢測到
successfully rolled out時，表示滾動更新成功。 - 如果在給定時間內(nèi)更新沒有成功，則流程將失敗。
繼續(xù)考慮一下如果statefulset多實例呢？不想寫兩個了整合成一個方法如下：
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
if (!(resourceType in ["deployment", "statefulset"])) {
error "Unknown resource type: ${resourceType}. Only 'deployment' and 'statefulset' are supported."
}
echo "Checking the update status of ${resourceType} '${resourceName}' in namespace '${namespace}'"
for (int i = 1; i <= attempts; i++) {
String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
String updateStatus = sh (
script: rolloutCommand,
returnStdout: true
).trim()
if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
echo "Update status: ${updateStatus}"
return
} else {
echo "Waiting for ${resourceType} '${resourceName}' to successfully roll out. Attempt ${i}/${attempts}."
sleep(sleepTime)
}
}
error "${resourceType} '${resourceName}' did not successfully roll out after ${attempts} attempts in namespace '${namespace}'"
}Jenkinsfile更新實現(xiàn)
經(jīng)過上述進化，Jenkinsfile中現(xiàn)在包含了完整的部署狀態(tài)檢查邏輯，以應對不同類型資源的部署監(jiān)控需求。
pipeline {
agent none // Use none at the top level, each stage will define its own agent.
environment {
REGISTRY = "ccr.ccs.tencentyun.com/xxxxx"
KUBE_CONFIG = "--namespace=master-metaspace"
KUBE_YAML_PATH = "/home/jenkins/workspace/yaml/master-metaspace"
// Assume that 'data' is defined elsewhere or injected as a parameter.
BASE_WORKSPACE = "xxxxxx" // 定義一個基礎(chǔ)工作空間路徑
}
stages {
stage("GetCode") {
agent { label "build01" }
steps {
script {
checkout scm: [
$class: 'GitSCM',
branches: [[name: env.branchName]],
extensions: [[$class: 'CloneOption', depth: 1, noTags: false, shallow: true]],
userRemoteConfigs: [[credentialsId: 'xxxxx', url: env.gitHttpURL]]
]
}
}
}
stage("Docker Builds") {
parallel {
stage('Build dataloader-game-ucenter') {
agent { label "build01" }
when { environment name: 'dataloader-game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("dataloader-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build datawriter-game-ucenter') {
agent { label "build01" }
when { environment name: 'datawriter-game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("datawriter-game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
stage('Build game-ucenter') {
agent { label "build01" }
when { environment name: 'game-ucenter', value: 'true' }
steps {
buildAndPushDockerImage("game-ucenter", env.data, env.BASE_WORKSPACE)
}
}
}
}
stage('Development Deployment') {
parallel {
stage("Deploy datawriter-game-ucenter") {
when { environment name: 'datawriter-game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("datawriter-game-ucenter")
checkKubernetesResourceStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
checkRolloutStatus("datawriter-game-ucenter", "master-metaspace", "statefulset")
}
}
stage("Deploy dataloader-game-ucenter") {
when { environment name: 'dataloader-game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("dataloader-game-ucenter")
checkKubernetesResourceStatus("dataloader-game-ucenter", "master-metaspace", "statefulset")
}
}
stage("Deploy game-ucenter") {
when { environment name: 'game-ucenter', value: 'true' }
agent { label "xxxx" }
steps {
deployToKubernetes("game-ucenter")
checkRolloutStatus("game-ucenter", "master-metaspace", "deployment")
checkKubernetesResourceStatus("game-ucenter", "master-metaspace", "deployment")
}
}
}
}
}
}
// Define methods outside pipeline to avoid repetition
def buildAndPushDockerImage(String imageName, String tag, String workspacePath) {
sh "cd ${workspacePath} && echo 'Current directory: \$(pwd)'" // 使用基礎(chǔ)工作空間變量
sh "cd ${workspacePath}/${imageName}&& docker build --build-arg NODE_ENV=game-ucenter -t $REGISTRY/$imageName:$tag ."
withCredentials([usernamePassword(credentialsId: 'xxxxx', passwordVariable: 'dockerPassword', usernameVariable: 'dockerUser')]) {
sh "docker login -u $dockerUser -p $dockerPassword $REGISTRY"
sh "docker push $REGISTRY/$imageName:$tag"
}
}
def deployToKubernetes(String kubernetesComponent) {
String templateFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.tpl"
String outputFile = "${KUBE_YAML_PATH}/${kubernetesComponent}.yaml"
sh "sed -e 's/{data}/$data/g' $templateFile > $outputFile"
sh "sudo kubectl apply -f $outputFile $KUBE_CONFIG"
}
def checkRolloutStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // 設(shè)置重試次數(shù)
int sleepTime = 10 // 設(shè)置重試間隔時間（秒）
if (!(resourceType in ["deployment", "statefulset"])) {
error "未知資源類型：${resourceType}。只支持 'deployment' 和 'statefulset' 。"
}
echo "正在檢查${resourceType} '${resourceName}' 在命名空間 '${namespace}' 的更新狀態(tài)"
for (int i = 1; i <= attempts; i++) {
String rolloutCommand = "kubectl rollout status ${resourceType}/${resourceName} --namespace=${namespace}"
try {
String updateStatus = sh (
script: rolloutCommand,
returnStdout: true
).trim()
// 添加對 "partitioned roll out complete" 狀態(tài)的檢查
if (updateStatus.contains("successfully rolled out") || updateStatus.contains("partitioned roll out complete")) {
echo "更新狀態(tài)：${updateStatus}"
return
} else {
echo "等待 ${resourceType} '${resourceName}' 成功發(fā)布。嘗試次數(shù)：${i}/${attempts}。"
sleep(sleepTime)
}
} catch (Exception e) {
echo "獲取更新狀態(tài)時發(fā)生錯誤：${e.getMessage()}。嘗試次數(shù)：${i}/${attempts}。"
sleep(sleepTime)
}
}
error "${resourceType} '${resourceName}' 在命名空間 '${namespace}' 內(nèi)未能在 ${attempts} 次嘗試之后成功發(fā)布"
}
def checkKubernetesResourceStatus(String resourceName, String namespace, String resourceType) {
int attempts = 30 // Set the number of retry attempts
int sleepTime = 10 // Set the sleep time between attempts in seconds
String readyReplicasJsonPath = resourceType == "deployment" ? ".status.readyReplicas" : ".status.readyReplicas"
for (int i = 1; i <= attempts; i++) {
// Check the resource status
String statusCheck = sh (
script: "kubectl get ${resourceType} ${resourceName} --namespace=${namespace} -o jsonpath=\"{${readyReplicasJsonPath}}\"",
returnStdout: true
).trim()
// If the number of ready replicas is not empty and equal to the desired number
if (statusCheck && statusCheck.isInteger() && statusCheck.toInteger() > 0) {
echo "${resourceType} ${resourceName} is ready."
return
} else {
echo "Waiting for ${resourceType} ${resourceName} to be ready. Attempt ${i}/${attempts}"
sleep(sleepTime)
}
}
error "${resourceType} ${resourceName} did not become ready after ${attempts} attempts"
}// 更新后的Jenkins Pipeline代碼詳細定義參照本文開頭給出的代碼
總結(jié)
本篇博客通過對Jenkins Pipeline的進化過程展開講解，展現(xiàn)了如何從簡單的部署任務轉(zhuǎn)變?yōu)橐粋€健壯且兼顧各類工作負載狀態(tài)監(jiān)測的CI/CD流程。我們強化了狀態(tài)檢測的邏輯，引入了更新策略的檢測，并保持了對不同Kubernetes資源類型的兼容性。這些改進確保了自動化流程能夠與現(xiàn)代部署實踐保持同步，給運維團隊帶來極大便利，并最大化地保障了部署的可靠性。
后記
由于篇幅限制，本篇文章未作其他更詳細演示。然而，在實際應用中，運維團隊可以根據(jù)自己的具體需求和環(huán)境進一步豐富和細化每個步驟的實現(xiàn)，確保Pipeline的健壯性和高可用性，以適應不斷變化的技術(shù)挑戰(zhàn)。以上實驗中使用了騰訊云的鏡像倉庫服務，kubernetes集群環(huán)境使用了Tke服務！
以上就是Jenkins Pipeline為Kubernetes應用部署增加狀態(tài)檢測腳本優(yōu)化的詳細內(nèi)容，更多關(guān)于Jenkins Pipeline檢測Kubernetes的資料請關(guān)注腳本之家其它相關(guān)文章！
相關(guān)文章
詳解SpringCloud Gateway之過濾器GatewayFilter
這篇文章主要介紹了詳解SpringCloud Gateway之過濾器GatewayFilter，小編覺得挺不錯的，現(xiàn)在分享給大家，也給大家做個參考。一起跟隨小編過來看看吧2018-10-10
Spring?Security權(quán)限控制的實現(xiàn)接口
這篇文章主要介紹了Spring?Security的很多功能，在這些眾多功能中，我們知道其核心功能其實就是認證+授權(quán)。Spring教程之Spring?Security的四種權(quán)限控制方式2023-03-03
這篇文章主要介紹了java中常用的阻塞隊列與非阻塞隊列用法,具有很好的參考價值,希望對大家有所幫助,如有錯誤或未考慮完全的地方,望不吝賜教2025-04-04
這篇文章主要為大家解析了JDK1.6集合框架bug：c.toArray might (incorrectly) not return Object[] (see 6260652)，具有一定的參考價值，感興趣的小伙伴們可以參考一下2017-10-10
這篇文章主要介紹了springBoot 創(chuàng)建定時任務過程詳解,文中通過示例代碼介紹的非常詳細，對大家的學習或者工作具有一定的參考學習價值,需要的朋友可以參考下2019-10-10
MybatisX-Generator不生成domain文件夾的問題及解決
在使用MybatisX-Generator生成數(shù)據(jù)庫表實體時,如果發(fā)現(xiàn)沒有生成domain文件夾以及User.java文件,是因為MybatisX版本更新,最新版需要在options里額外勾選model才能生成domain,勾選model并點擊finish后,成功生成domain文件夾及User.java文件2025-01-01
這篇文章主要為大家詳細介紹了Java簡單實現(xiàn)聊天機器人，文中示例代碼介紹的非常詳細，具有一定的參考價值，感興趣的小伙伴們可以參考一下2021-07-07
java使用Hashtable過濾數(shù)組中重復值的方法
這篇文章主要介紹了java使用Hashtable過濾數(shù)組中重復值的方法,涉及java數(shù)組遍歷及過濾的相關(guān)技巧,需要的朋友可以參考下2016-08-08
Java攔截器Interceptor實現(xiàn)原理及代碼示例
本文詳細講解了Java攔截器Interceptor實現(xiàn)原理及代碼示例,對大家的學習或工作具有一定的參考借鑒價值，需要的朋友可以參考下2021-12-12
spring簡單MVC實現(xiàn)方法(URL映射及其參數(shù)使用、查詢（id、其他參數(shù)）、增加)
這篇文章主要介紹了spring簡單MVC實現(xiàn)方法(URL映射及其參數(shù)使用、查詢（id、其他參數(shù)）、增加),方法參數(shù)使用包括在無注解下獲取參數(shù),使用@RequestParam 獲取參數(shù)的方法,每種方法講解的非常詳細,需要的朋友可以參考下2024-01-01