Spark Command: /opt/java/openjdk/bin/java -cp /opt/spark/conf:/opt/spark/jars/slf4j-api-2.0.16.jar:/opt/spark/jars/* -Dspark.master.ui.title=Prod Spark Cluster -Dspark.master.rest.enabled=true -Dspark.master.rest.host=0.0.0.0 -Dspark.kubernetes.namespace=default -Dspark.ui.reverseProxy=true -Dspark.kubernetes.container.image=apache/spark:4.0.0 -Xmx1g org.apache.spark.deploy.master.Master --host prod-master-0 --port 7077 --webui-port 8080
========================================
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/07/27 16:25:48 INFO Master: Started daemon with process name: 18@prod-master-0
25/07/27 16:25:48 INFO SignalUtils: Registering signal handler for TERM
25/07/27 16:25:48 INFO SignalUtils: Registering signal handler for HUP
25/07/27 16:25:48 INFO SignalUtils: Registering signal handler for INT
25/07/27 16:25:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j2-defaults.properties
25/07/27 16:25:48 INFO SecurityManager: Changing view acls to: spark
25/07/27 16:25:48 INFO SecurityManager: Changing modify acls to: spark
25/07/27 16:25:48 INFO SecurityManager: Changing view acls groups to: spark
25/07/27 16:25:48 INFO SecurityManager: Changing modify acls groups to: spark
25/07/27 16:25:48 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: spark groups with view permissions: EMPTY; users with modify permissions: spark; groups with modify permissions: EMPTY; RPC SSL disabled
25/07/27 16:25:49 INFO Utils: Successfully started service 'sparkMaster' on port 7077.
25/07/27 16:25:49 INFO Master: Starting Spark master at spark://prod-master-0:7077
25/07/27 16:25:49 INFO Master: Running Spark version 4.0.0
25/07/27 16:25:49 INFO JettyUtils: Start Jetty 0.0.0.0:8080 for MasterUI
25/07/27 16:25:49 INFO Utils: Successfully started service 'MasterUI' on port 8080.
25/07/27 16:25:49 INFO MasterWebUI: Bound MasterWebUI to 0.0.0.0, and started at http://prod-master-0:8080
25/07/27 16:25:50 INFO Master: Spark Master is acting as a reverse proxy. Master, Workers and Applications UIs are available at http://prod-master-0:8080
25/07/27 16:25:50 INFO Utils: Successfully started service on port 6066.
25/07/27 16:25:50 INFO StandaloneRestServer: Started REST server for submitting applications on 0.0.0.0 with port 6066
25/07/27 16:25:50 INFO Master: Registering worker 10.161.40.7:37575 with 8 cores, 6.8 GiB RAM
25/07/27 16:25:50 INFO Master: I have been elected leader! New state: ALIVE
25/07/27 16:25:54 INFO Master: Registering worker 10.161.40.7:37575 with 8 cores, 6.8 GiB RAM
25/07/27 16:27:13 INFO Master: Registering worker 10.161.34.9:45377 with 2 cores, 6.8 GiB RAM
25/07/27 16:27:42 INFO Master: Registering worker 10.161.28.15:36615 with 2 cores, 6.8 GiB RAM
25/11/12 07:05:25 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
25/11/12 07:05:25 INFO Master: Launching driver driver-20251112070525-0000 on worker worker-20250727162712-10.161.34.9-45377
25/11/12 07:05:36 INFO Master: Removing driver: driver-20251112070525-0000 (FINISHED)
25/11/12 07:06:06 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
25/11/12 07:06:06 INFO Master: Launching driver driver-20251112070606-0001 on worker worker-20250727162712-10.161.34.9-45377
25/11/12 07:06:07 INFO Master: Driver submitted org.apache.spark.deploy.worker.DriverWrapper
25/11/12 07:06:07 INFO Master: Launching driver driver-20251112070607-0002 on worker worker-20250727162549-10.161.40.7-37575
25/11/12 07:06:13 INFO Master: Removing driver: driver-20251112070606-0001 (FINISHED)
25/11/12 07:06:13 INFO Master: Removing driver: driver-20251112070607-0002 (FINISHED)