境部署DolphinScheduler與SeaTunnel實(shí)戰(zhàn))
1. 項(xiàng)目背景與核心挑戰(zhàn)在傳統(tǒng)企業(yè)IT架構(gòu)向云原生轉(zhuǎn)型的過程中Kubernetesk8s已成為事實(shí)上的容器編排標(biāo)準(zhǔn)。然而在實(shí)際生產(chǎn)環(huán)境中由于安全合規(guī)要求大量企業(yè)需要在內(nèi)網(wǎng)隔離環(huán)境下部署云原生應(yīng)用套件。本次要解決的正是這樣一個(gè)典型場(chǎng)景在完全離線的k8s集群中部署DolphinScheduler工作流調(diào)度系統(tǒng)和SeaTunnel數(shù)據(jù)集成工具。這個(gè)組合方案特別適合需要處理以下需求的企業(yè)存在嚴(yán)格網(wǎng)絡(luò)安全隔離要求的金融、政務(wù)等行業(yè)需要自動(dòng)化調(diào)度異構(gòu)數(shù)據(jù)源ETL流程的場(chǎng)景希望利用聲明式配置管理整個(gè)數(shù)據(jù)流水線的團(tuán)隊(duì)關(guān)鍵難點(diǎn)提示離線環(huán)境意味著所有依賴鏡像、charts包、系統(tǒng)組件都需要預(yù)先下載并完成內(nèi)部倉庫的搭建任何遺漏都可能導(dǎo)致部署過程中斷。2. 離線環(huán)境準(zhǔn)備工作2.1 基礎(chǔ)設(shè)施規(guī)劃建議建議采用以下節(jié)點(diǎn)規(guī)格作為基準(zhǔn)可根據(jù)實(shí)際負(fù)載調(diào)整節(jié)點(diǎn)類型CPU內(nèi)存磁盤數(shù)量Master4核16G100G3Worker8核32G200G至少2網(wǎng)絡(luò)方面需要確保所有節(jié)點(diǎn)間網(wǎng)絡(luò)延遲2ms節(jié)點(diǎn)間帶寬≥1Gbps提前規(guī)劃好Pod CIDR和Service CIDR2.2 離線資源打包清單需要預(yù)先準(zhǔn)備的離線資源包括基礎(chǔ)組件包k8s各節(jié)點(diǎn)所需系統(tǒng)依賴conntrack、socat等容器運(yùn)行時(shí)推薦containerd 1.6Helm 3.10二進(jìn)制文件鏡像倉庫方案# 搭建Harbor私有倉庫示例 helm repo add harbor https://helm.goharbor.io helm fetch harbor/harbor --version 1.10.0應(yīng)用鏡像清單DolphinScheduler 3.2.1全套鏡像含web、api、worker等SeaTunnel 2.3.2引擎鏡像相關(guān)中間件鏡像ZooKeeper、MySQL等Helm charts包helm repo add dolphinscheduler https://dolphinscheduler.apache.org helm pull dolphinscheduler/dolphinscheduler --version 3.2.13. 核心組件部署實(shí)戰(zhàn)3.1 DolphinScheduler部署要點(diǎn)3.1.1 數(shù)據(jù)庫初始化建議使用獨(dú)立的MySQL實(shí)例非k8s內(nèi)部署CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8mb4; CREATE USER ds_user% IDENTIFIED BY StrongPass123; GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ds_user%;3.1.2 Helm定制化配置關(guān)鍵values.yaml配置項(xiàng)postgresql: enabled: false # 禁用內(nèi)置PG externalDatabase: type: mysql host: mysql.internal port: 3306 username: ds_user password: StrongPass123 registry: type: harbor url: harbor.internal repository: library/dolphinscheduler部署命令helm install dolphinscheduler ./dolphinscheduler-3.2.1.tgz \ -n ds --create-namespace \ -f values.yaml3.2 SeaTunnel集成方案3.2.1 引擎部署模式選擇推薦使用Spark on k8s模式spark: master: k8s://https://kubernetes.default.svc deployMode: cluster image: harbor.internal/library/seatunnel-spark:2.3.23.2.2 任務(wù)配置示例典型JDBC源到HDFS的作業(yè)配置env { execution.parallelism 3 } source { JdbcSource { url jdbc:mysql://mysql.internal:3306/source_db username etl_user password EtlPass456 query SELECT * FROM orders WHERE update_time ${last_update} } } transform { Sql { query SELECT user_id, SUM(amount) AS total FROM orders GROUP BY user_id } } sink { Hdfs { path hdfs://namenode:8020/data/orders_agg file_format parquet } }4. 生產(chǎn)環(huán)境調(diào)優(yōu)指南4.1 性能關(guān)鍵參數(shù)DolphinScheduler工作節(jié)點(diǎn)配置建議worker: resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi env: - name: TASK_EXECUTE_THREADS value: 10 # 根據(jù)CPU核數(shù)調(diào)整4.2 高可用保障措施為關(guān)鍵組件配置Pod反親和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/component operator: In values: [master-server] topologyKey: kubernetes.io/hostname配置合理的健康檢查livenessProbe: httpGet: path: /actuator/health port: 12345 initialDelaySeconds: 30 periodSeconds: 105. 典型問題排查手冊(cè)5.1 鏡像拉取失敗現(xiàn)象Pod狀態(tài)為ImagePullBackOff 解決方案確認(rèn)harbor證書已添加到各個(gè)節(jié)點(diǎn)mkdir -p /etc/docker/certs.d/harbor.internal cp harbor-ca.crt /etc/docker/certs.d/harbor.internal/ca.crt檢查secret配置是否正確kubectl create secret docker-registry harbor-secret \ --docker-serverharbor.internal \ --docker-usernameadmin \ --docker-passwordHarbor12345 \ -n ds5.2 權(quán)限相關(guān)問題現(xiàn)象SeaTunnel任務(wù)報(bào)Permission denied 處理方法為Spark Driver配置合適的安全上下文securityContext: runAsUser: 1000 fsGroup: 1000在HDFS端配置ACLhdfs dfs -setfacl -R -m user:spark:rwx /data6. 監(jiān)控與運(yùn)維建議6.1 指標(biāo)采集方案推薦使用Prometheus Operator采集指標(biāo)DolphinScheduler暴露的指標(biāo)端點(diǎn)metrics: enabled: true port: 12345對(duì)應(yīng)的ServiceMonitor配置apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: release: prometheus name: dolphinscheduler-monitor spec: endpoints: - port: metrics selector: matchLabels: app.kubernetes.io/instance: dolphinscheduler6.2 日志收集策略建議采用Filebeat ELK方案filebeatConfig: filebeat.yml: | filebeat.inputs: - type: container paths: - /var/log/containers/*.log output.elasticsearch: hosts: [elasticsearch.internal:9200] 離線K8s環(huán)境部署DolphinScheduler與SeaTunnel實(shí)戰(zhàn)
1. 項(xiàng)目背景與核心挑戰(zhàn)在傳統(tǒng)企業(yè)IT架構(gòu)向云原生轉(zhuǎn)型的過程中Kubernetesk8s已成為事實(shí)上的容器編排標(biāo)準(zhǔn)。然而在實(shí)際生產(chǎn)環(huán)境中由于安全合規(guī)要求大量企業(yè)需要在內(nèi)網(wǎng)隔離環(huán)境下部署云原生應(yīng)用套件。本次要解決的正是這樣一個(gè)典型場(chǎng)景在完全離線的k8s集群中部署DolphinScheduler工作流調(diào)度系統(tǒng)和SeaTunnel數(shù)據(jù)集成工具。這個(gè)組合方案特別適合需要處理以下需求的企業(yè)存在嚴(yán)格網(wǎng)絡(luò)安全隔離要求的金融、政務(wù)等行業(yè)需要自動(dòng)化調(diào)度異構(gòu)數(shù)據(jù)源ETL流程的場(chǎng)景希望利用聲明式配置管理整個(gè)數(shù)據(jù)流水線的團(tuán)隊(duì)關(guān)鍵難點(diǎn)提示離線環(huán)境意味著所有依賴鏡像、charts包、系統(tǒng)組件都需要預(yù)先下載并完成內(nèi)部倉庫的搭建任何遺漏都可能導(dǎo)致部署過程中斷。2. 離線環(huán)境準(zhǔn)備工作2.1 基礎(chǔ)設(shè)施規(guī)劃建議建議采用以下節(jié)點(diǎn)規(guī)格作為基準(zhǔn)可根據(jù)實(shí)際負(fù)載調(diào)整節(jié)點(diǎn)類型CPU內(nèi)存磁盤數(shù)量Master4核16G100G3Worker8核32G200G至少2網(wǎng)絡(luò)方面需要確保所有節(jié)點(diǎn)間網(wǎng)絡(luò)延遲2ms節(jié)點(diǎn)間帶寬≥1Gbps提前規(guī)劃好Pod CIDR和Service CIDR2.2 離線資源打包清單需要預(yù)先準(zhǔn)備的離線資源包括基礎(chǔ)組件包k8s各節(jié)點(diǎn)所需系統(tǒng)依賴conntrack、socat等容器運(yùn)行時(shí)推薦containerd 1.6Helm 3.10二進(jìn)制文件鏡像倉庫方案# 搭建Harbor私有倉庫示例 helm repo add harbor https://helm.goharbor.io helm fetch harbor/harbor --version 1.10.0應(yīng)用鏡像清單DolphinScheduler 3.2.1全套鏡像含web、api、worker等SeaTunnel 2.3.2引擎鏡像相關(guān)中間件鏡像ZooKeeper、MySQL等Helm charts包helm repo add dolphinscheduler https://dolphinscheduler.apache.org helm pull dolphinscheduler/dolphinscheduler --version 3.2.13. 核心組件部署實(shí)戰(zhàn)3.1 DolphinScheduler部署要點(diǎn)3.1.1 數(shù)據(jù)庫初始化建議使用獨(dú)立的MySQL實(shí)例非k8s內(nèi)部署CREATE DATABASE dolphinscheduler DEFAULT CHARACTER SET utf8mb4; CREATE USER ds_user% IDENTIFIED BY StrongPass123; GRANT ALL PRIVILEGES ON dolphinscheduler.* TO ds_user%;3.1.2 Helm定制化配置關(guān)鍵values.yaml配置項(xiàng)postgresql: enabled: false # 禁用內(nèi)置PG externalDatabase: type: mysql host: mysql.internal port: 3306 username: ds_user password: StrongPass123 registry: type: harbor url: harbor.internal repository: library/dolphinscheduler部署命令helm install dolphinscheduler ./dolphinscheduler-3.2.1.tgz \ -n ds --create-namespace \ -f values.yaml3.2 SeaTunnel集成方案3.2.1 引擎部署模式選擇推薦使用Spark on k8s模式spark: master: k8s://https://kubernetes.default.svc deployMode: cluster image: harbor.internal/library/seatunnel-spark:2.3.23.2.2 任務(wù)配置示例典型JDBC源到HDFS的作業(yè)配置env { execution.parallelism 3 } source { JdbcSource { url jdbc:mysql://mysql.internal:3306/source_db username etl_user password EtlPass456 query SELECT * FROM orders WHERE update_time ${last_update} } } transform { Sql { query SELECT user_id, SUM(amount) AS total FROM orders GROUP BY user_id } } sink { Hdfs { path hdfs://namenode:8020/data/orders_agg file_format parquet } }4. 生產(chǎn)環(huán)境調(diào)優(yōu)指南4.1 性能關(guān)鍵參數(shù)DolphinScheduler工作節(jié)點(diǎn)配置建議worker: resources: limits: cpu: 2 memory: 4Gi requests: cpu: 1 memory: 2Gi env: - name: TASK_EXECUTE_THREADS value: 10 # 根據(jù)CPU核數(shù)調(diào)整4.2 高可用保障措施為關(guān)鍵組件配置Pod反親和性affinity: podAntiAffinity: requiredDuringSchedulingIgnoredDuringExecution: - labelSelector: matchExpressions: - key: app.kubernetes.io/component operator: In values: [master-server] topologyKey: kubernetes.io/hostname配置合理的健康檢查livenessProbe: httpGet: path: /actuator/health port: 12345 initialDelaySeconds: 30 periodSeconds: 105. 典型問題排查手冊(cè)5.1 鏡像拉取失敗現(xiàn)象Pod狀態(tài)為ImagePullBackOff 解決方案確認(rèn)harbor證書已添加到各個(gè)節(jié)點(diǎn)mkdir -p /etc/docker/certs.d/harbor.internal cp harbor-ca.crt /etc/docker/certs.d/harbor.internal/ca.crt檢查secret配置是否正確kubectl create secret docker-registry harbor-secret \ --docker-serverharbor.internal \ --docker-usernameadmin \ --docker-passwordHarbor12345 \ -n ds5.2 權(quán)限相關(guān)問題現(xiàn)象SeaTunnel任務(wù)報(bào)Permission denied 處理方法為Spark Driver配置合適的安全上下文securityContext: runAsUser: 1000 fsGroup: 1000在HDFS端配置ACLhdfs dfs -setfacl -R -m user:spark:rwx /data6. 監(jiān)控與運(yùn)維建議6.1 指標(biāo)采集方案推薦使用Prometheus Operator采集指標(biāo)DolphinScheduler暴露的指標(biāo)端點(diǎn)metrics: enabled: true port: 12345對(duì)應(yīng)的ServiceMonitor配置apiVersion: monitoring.coreos.com/v1 kind: ServiceMonitor metadata: labels: release: prometheus name: dolphinscheduler-monitor spec: endpoints: - port: metrics selector: matchLabels: app.kubernetes.io/instance: dolphinscheduler6.2 日志收集策略建議采用Filebeat ELK方案filebeatConfig: filebeat.yml: | filebeat.inputs: - type: container paths: - /var/log/containers/*.log output.elasticsearch: hosts: [elasticsearch.internal:9200]