[Bug] Frequent org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.io.IOException: Stream closed and java.net.SocketException: Socket closed

[A-little-bit-of-data]

Code of Conduct

• I agree to follow this project's Code of Conduct

Search before asking

• I have searched in the issues and found no similar issues.

Describe the bug

I configured data sources for Trino and Spark SQL separately using the Kyuubi data source version of Dolphin 3.1.5, so that I can connect to Trino and Spark SQL independently. The following is the connection printed in the cluster: Full baseConnectionParam parameters: KyuubiConnectionParam{user='a', password='********************', address='jdbc:hive2://X.X.X.X:2181,Y.Y.Y.Y:2181', database='default', jdbcUrl='jdbc:hive2://X.X.X.X:2181,Y.Y.Y.Y:2181/default;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=kyuubi;connectTimeout=100000;socketTimeout=1400000;#kyuubi.engine.type=TRINO', driverLocation='null', driverClassName='org.apache.kyuubi.jdbc.KyuubiHiveDriver', validationQuery='show functions', other='serviceDiscoveryMode=zooKeeper&zooKeeperNamespace=kyuubi&connectTimeout=100000&socketTimeout=1400000&#kyuubi.engine.type=TRINO', principal='null', javaSecurityKrb5Conf='null', loginUserKeytabUsername='null', loginUserKeytabPath='null'}

[ERROR] 2026-05-28 00:09:09.168 +0800 - execute sql error: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.net.SocketException: Socket closed
[ERROR] 2026-05-28 00:09:09.169 +0800 - sql task error
org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.net.SocketException: Socket closed
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:428)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeWithConfOverlay(KyuubiStatement.java:197)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:191)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeUpdate(KyuubiStatement.java:515)
	at org.apache.kyuubi.jdbc.hive.KyuubiPreparedStatement.executeUpdate(KyuubiPreparedStatement.java:75)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:321)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:219)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:170)
	at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49)
	at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
	Suppressed: org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.shaded.thrift.transport.TTransportException: SASL authentication not complete
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeStatementIfNeeded(KyuubiStatement.java:167)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeClientOperation(KyuubiStatement.java:172)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.close(KyuubiStatement.java:183)
		at com.zaxxer.hikari.pool.ProxyStatement.close(ProxyStatement.java:75)
		at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:323)
		... 11 common frames omitted
	Caused by: org.apache.kyuubi.shaded.thrift.transport.TTransportException: SASL authentication not complete
		at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.write(TSaslTransport.java:442)
		at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:39)
		at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:204)
		at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:119)
		at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:70)
		at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:62)
		at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.send_CloseOperation(TCLIService.java:496)
		at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:488)
		at sun.reflect.GeneratedMethodAccessor57.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.kyuubi.jdbc.hive.KyuubiConnection$SynchronizedHandler.invoke(KyuubiConnection.java:1441)
		at com.sun.proxy.$Proxy156.CloseOperation(Unknown Source)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeStatementIfNeeded(KyuubiStatement.java:160)
		... 15 common frames omitted
Caused by: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.net.SocketException: Socket closed
	at org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:216)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.flush(TSaslTransport.java:469)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.flush(TSaslClientTransport.java:39)
	at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:73)
	at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:62)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.send_GetOperationStatus(TCLIService.java:450)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:442)
	at sun.reflect.GeneratedMethodAccessor47.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.kyuubi.jdbc.hive.KyuubiConnection$SynchronizedHandler.invoke(KyuubiConnection.java:1441)
	at com.sun.proxy.$Proxy156.GetOperationStatus(Unknown Source)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:389)
	... 18 common frames omitted
Caused by: java.net.SocketException: Socket closed
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport.flush(TIOStreamTransport.java:211)
	... 30 common frames omitted

[ERROR] 2026-05-28 09:15:08.659 +0800 - execute sql error: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.io.IOException: Stream closed
[ERROR] 2026-05-28 09:15:08.662 +0800 - sql task error
org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.io.IOException: Stream closed
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:428)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeWithConfOverlay(KyuubiStatement.java:197)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.execute(KyuubiStatement.java:191)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.executeUpdate(KyuubiStatement.java:515)
	at org.apache.kyuubi.jdbc.hive.KyuubiPreparedStatement.executeUpdate(KyuubiPreparedStatement.java:75)
	at com.zaxxer.hikari.pool.ProxyPreparedStatement.executeUpdate(ProxyPreparedStatement.java:61)
	at com.zaxxer.hikari.pool.HikariProxyPreparedStatement.executeUpdate(HikariProxyPreparedStatement.java)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:321)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeFuncAndSql(SqlTask.java:219)
	at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.handle(SqlTask.java:170)
	at org.apache.dolphinscheduler.server.worker.runner.DefaultWorkerDelayTaskExecuteRunnable.executeTask(DefaultWorkerDelayTaskExecuteRunnable.java:49)
	at org.apache.dolphinscheduler.server.worker.runner.WorkerTaskExecuteRunnable.run(WorkerTaskExecuteRunnable.java:174)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
	at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
	at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
	Suppressed: org.apache.kyuubi.jdbc.hive.KyuubiSQLException: org.apache.kyuubi.shaded.thrift.transport.TTransportException: SASL authentication not complete
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeStatementIfNeeded(KyuubiStatement.java:167)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeClientOperation(KyuubiStatement.java:172)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.close(KyuubiStatement.java:183)
		at com.zaxxer.hikari.pool.ProxyStatement.close(ProxyStatement.java:75)
		at org.apache.dolphinscheduler.plugin.task.sql.SqlTask.executeUpdate(SqlTask.java:323)
		... 11 common frames omitted
	Caused by: org.apache.kyuubi.shaded.thrift.transport.TTransportException: SASL authentication not complete
		at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.write(TSaslTransport.java:442)
		at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.write(TSaslClientTransport.java:39)
		at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.writeI32(TBinaryProtocol.java:204)
		at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.writeMessageBegin(TBinaryProtocol.java:119)
		at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:70)
		at org.apache.kyuubi.shaded.thrift.TServiceClient.sendBase(TServiceClient.java:62)
		at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.send_CloseOperation(TCLIService.java:496)
		at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.CloseOperation(TCLIService.java:488)
		at sun.reflect.GeneratedMethodAccessor160.invoke(Unknown Source)
		at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.lang.reflect.Method.invoke(Method.java:498)
		at org.apache.kyuubi.jdbc.hive.KyuubiConnection$SynchronizedHandler.invoke(KyuubiConnection.java:1441)
		at com.sun.proxy.$Proxy156.CloseOperation(Unknown Source)
		at org.apache.kyuubi.jdbc.hive.KyuubiStatement.closeStatementIfNeeded(KyuubiStatement.java:160)
		... 15 common frames omitted
Caused by: org.apache.kyuubi.shaded.thrift.transport.TTransportException: java.io.IOException: Stream closed
	at org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:181)
	at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.readLength(TSaslTransport.java:338)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.readFrame(TSaslTransport.java:421)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.read(TSaslTransport.java:397)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.read(TSaslClientTransport.java:39)
	at org.apache.kyuubi.shaded.thrift.transport.TTransport.readAll(TTransport.java:109)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:464)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:362)
	at org.apache.kyuubi.shaded.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:245)
	at org.apache.kyuubi.shaded.thrift.TServiceClient.receiveBase(TServiceClient.java:77)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.recv_GetOperationStatus(TCLIService.java:456)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Client.GetOperationStatus(TCLIService.java:443)
	at sun.reflect.GeneratedMethodAccessor41.invoke(Unknown Source)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.apache.kyuubi.jdbc.hive.KyuubiConnection$SynchronizedHandler.invoke(KyuubiConnection.java:1441)
	at com.sun.proxy.$Proxy156.GetOperationStatus(Unknown Source)
	at org.apache.kyuubi.jdbc.hive.KyuubiStatement.waitForOperationToComplete(KyuubiStatement.java:389)
	... 18 common frames omitted
Caused by: java.io.IOException: Stream closed
	at java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:170)
	at java.io.BufferedInputStream.read(BufferedInputStream.java:336)
	at org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:177)
	... 36 common frames omitted

Affects Version(s)

1.10.1

Kyuubi Server Log Output

2026-05-28 00:03:01.034 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=e1b513d0-4ee6-43e0-ba73-5f6f3fa54654	opType=ExecuteStatement	state=INITIALIZED	user=a	session=3ea5ce8c-821b-4b94-a2d3-1d4f822b3734
2026-05-28 00:03:01.034 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/3ea5ce8c-821b-4b94-a2d3-1d4f822b3734/e1b513d0-4ee6-43e0-ba73-5f6f3fa54654
2026-05-28 00:03:01.034 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [a:172.31.229.166] SessionHandle [3ea5ce8c-821b-4b94-a2d3-1d4f822b3734] - Starting to wait the launch engine operation finished
2026-05-28 00:03:01.034 INFO org.apache.kyuubi.session.KyuubiSessionImpl: [a:172.31.229.166] SessionHandle [3ea5ce8c-821b-4b94-a2d3-1d4f822b3734] - Engine has been launched, elapsed time: 0 s

2026-05-28 00:09:08.234 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[e08e136b-a44d-4bcb-b006-a102928b4e46] in RUNNING_STATE
2026-05-28 00:09:08.443 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[da80cbb8-afa2-4d2b-9ded-a837f629dfa3] in RUNNING_STATE
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [3ea5ce8c-821b-4b94-a2d3-1d4f822b3734]
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.session.KyuubiSessionManager: aKyuubiSessionImpl with SessionHandle [3ea5ce8c-821b-4b94-a2d3-1d4f822b3734] is closed, current opening sessions 92
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.operation.LaunchEngine: Processing a's query[40aeb74d-839e-4101-b95f-c3e5c18516dc]: FINISHED_STATE -> CLOSED_STATE, time taken: 1382.857 seconds
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=40aeb74d-839e-4101-b95f-c3e5c18516dc	opType=LaunchEngine	state=CLOSED	user=a session=3ea5ce8c-821b-4b94-a2d3-1d4f822b3734
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing a's query[eb337100-770a-493f-84e7-cd926b5c262e]: RUNNING_STATE -> CLOSED_STATE, time taken: 5.001 seconds
2026-05-28 00:09:09.099 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=eb337100-770a-493f-84e7-cd926b5c262e	opType=ExecuteStatement	state=CLOSED	user=a	session=3ea5ce8c-821b-4b94-a2d3-1d4f822b3734
2026-05-28 00:09:09.100 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[eb337100-770a-493f-84e7-cd926b5c262e] in RUNNING_STATE
2026-05-28 00:09:09.159 INFO org.apache.kyuubi.client.KyuubiSyncThriftClient: TCloseOperationReq(operationHandle:TOperationHandle(operationId:THandleIdentifier(guid:A0 BF 76 55 2A 06 48 9A A1 DA 60 9A F4 A2 87 98, secret:C2 EE 5B 97 3E A0 41 FC AC 16 9B D7 08 ED 8F 38), operationType:EXECUTE_STATEMENT, hasResultSet:true)) succeed on engine side
2026-05-28 00:09:09.161 WARN org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport: Error closing output stream.
java.net.SocketException: Socket closed
	at java.net.SocketOutputStream.socketWrite(SocketOutputStream.java:118)
	at java.net.SocketOutputStream.write(SocketOutputStream.java:155)
	at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
	at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
	at java.io.FilterOutputStream.close(FilterOutputStream.java:158)
	at org.apache.kyuubi.shaded.thrift.transport.TIOStreamTransport.close(TIOStreamTransport.java:157)
	at org.apache.kyuubi.shaded.thrift.transport.TSocket.close(TSocket.java:264)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslTransport.close(TSaslTransport.java:364)
	at org.apache.kyuubi.shaded.thrift.transport.TSaslClientTransport.close(TSaslClientTransport.java:39)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$5(KyuubiSyncThriftClient.scala:253)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$closeSession$5$adapted(KyuubiSyncThriftClient.scala:252)
	at scala.collection.immutable.List.foreach(List.scala:431)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.closeSession(KyuubiSyncThriftClient.scala:252)
	at org.apache.kyuubi.session.KyuubiSessionImpl.close(KyuubiSessionImpl.scala:284)
	at org.apache.kyuubi.session.SessionManager.closeSession(SessionManager.scala:134)
	at org.apache.kyuubi.session.KyuubiSessionManager.closeSession(KyuubiSessionManager.scala:133)
	at org.apache.kyuubi.service.AbstractBackendService.closeSession(AbstractBackendService.scala:49)
	at org.apache.kyuubi.server.KyuubiServer$$anon$1.org$apache$kyuubi$server$BackendServiceMetric$$super$closeSession(KyuubiServer.scala:186)
	at org.apache.kyuubi.server.BackendServiceMetric.$anonfun$closeSession$1(BackendServiceMetric.scala:42)
	at scala.runtime.java8.JFunction0$mcV$sp.apply(JFunction0$mcV$sp.java:23)
	at org.apache.kyuubi.metrics.MetricsSystem$.timerTracing(MetricsSystem.scala:112)
	at org.apache.kyuubi.server.BackendServiceMetric.closeSession(BackendServiceMetric.scala:42)
	at org.apache.kyuubi.server.BackendServiceMetric.closeSession$(BackendServiceMetric.scala:40)
	at org.apache.kyuubi.server.KyuubiServer$$anon$1.closeSession(KyuubiServer.scala:186)
	at org.apache.kyuubi.service.TFrontendService.CloseSession(TFrontendService.scala:210)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1620)
	at org.apache.kyuubi.shaded.hive.service.rpc.thrift.TCLIService$Processor$CloseSession.getResult(TCLIService.java:1600)
	at org.apache.kyuubi.shaded.thrift.ProcessFunction.process(ProcessFunction.java:38)
	at org.apache.kyuubi.shaded.thrift.TBaseProcessor.process(TBaseProcessor.java:38)
	at org.apache.kyuubi.service.authentication.TSetIpAddressProcessor.process(TSetIpAddressProcessor.scala:35)
	at org.apache.kyuubi.shaded.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:250)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)
2026-05-28 00:09:09.162 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [3ea5ce8c-821b-4b94-a2d3-1d4f822b3734]


2026-05-28 08:43:41.369 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Client protocol version: HIVE_CLI_SERVICE_PROTOCOL_V10
2026-05-28 08:43:41.369 INFO org.apache.kyuubi.session.KyuubiSessionManager: Opening session for b@172.31.215.60
2026-05-28 08:43:41.372 INFO org.apache.kyuubi.session.LDAPGroupProvider: User [b] belongs to groups: etl
2026-05-28 08:43:41.372 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=44473f42-ec71-489a-9836-80dbf162f3d8	opType=LaunchEngine	state=INITIALIZED	user=b session=60234e5f-9125-43c5-b403-d816357e89c6
2026-05-28 08:43:41.372 INFO org.apache.kyuubi.operation.log.OperationLog: Creating operation log file /opt/kyuubi/work/server_operation_logs/60234e5f-9125-43c5-b403-d816357e89c6/44473f42-ec71-489a-9836-80dbf162f3d8

2026-05-28 09:13:12.898 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=cb66f47d-29d9-478b-88fa-c6b36be80a3c	opType=ExecuteStatement	state=RUNNING	user=ep_rh	session=c9c07615-b1df-4381-8c1b-386318cecf25
2026-05-28 09:13:13.560 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[6e386429-5c9d-4c9b-a7e3-31c6241a0d5b] in RUNNING_STATE
2026-05-28 09:13:13.973 INFO org.apache.kyuubi.operation.ExecuteStatement: Query[d8223e61-f953-4661-a614-e12c0d32709c] in RUNNING_STATE
2026-05-28 09:13:14.440 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Received request of closing SessionHandle [60234e5f-9125-43c5-b403-d816357e89c6]
2026-05-28 09:13:14.440 INFO org.apache.kyuubi.session.KyuubiSessionManager: b's KyuubiSessionImpl with SessionHandle [60234e5f-9125-43c5-b403-d816357e89c6] is closed, current opening sessions 103
2026-05-28 09:13:14.440 INFO org.apache.kyuubi.operation.LaunchEngine: Processing b's query[44473f42-ec71-489a-9836-80dbf162f3d8]: FINISHED_STATE -> CLOSED_STATE, time taken: 1773.067 seconds
2026-05-28 09:13:14.440 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=44473f42-ec71-489a-9836-80dbf162f3d8	opType=LaunchEngine	state=CLOSED	user=b	session=60234e5f-9125-43c5-b403-d816357e89c6
2026-05-28 09:13:14.442 INFO org.apache.kyuubi.server.KyuubiTBinaryFrontendService: Finished closing SessionHandle [60234e5f-9125-43c5-b403-d816357e89c6]
2026-05-28 09:13:14.774 INFO org.apache.kyuubi.operation.ExecuteStatement: Processing ep_rh's query[be22275c-3b1d-45d4-b0a0-12b353d3dd84]: RUNNING_STATE -> TIMEDOUT_STATE, time taken: 4.928 seconds
2026-05-28 09:13:14.774 INFO org.apache.kyuubi.operation.OperationAuditLogger: operation=be22275c-3b1d-45d4-b0a0-12b353d3dd84	opType=ExecuteStatement	state=TIMEOUT	user=ep_rh	session=1e370455-85f9-4a94-b051-cf0e43be4c75
2026-05-28 09:13:14.774 WARN org.apache.kyuubi.operation.ExecuteStatement: Ignore exception in terminal state with be22275c-3b1d-45d4-b0a0-12b353d3dd84
java.lang.InterruptedException
	at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
	at java.util.concurrent.FutureTask.get(FutureTask.java:191)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquiredAsyncRequest$1(KyuubiSyncThriftClient.scala:174)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.$anonfun$withLockAcquired$1(KyuubiSyncThriftClient.scala:159)
	at org.apache.kyuubi.Utils$.withLockRequired(Utils.scala:392)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquired(KyuubiSyncThriftClient.scala:155)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.withLockAcquiredAsyncRequest(KyuubiSyncThriftClient.scala:162)
	at org.apache.kyuubi.client.KyuubiSyncThriftClient.getOperationStatus(KyuubiSyncThriftClient.scala:398)
	at org.apache.kyuubi.operation.ExecuteStatement.waitStatementComplete(ExecuteStatement.scala:84)
	at org.apache.kyuubi.operation.ExecuteStatement.$anonfun$runInternal$1(ExecuteStatement.scala:173)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Kyuubi Engine Log Output

Kyuubi Server Configurations

kyuubi.kubernetes.namespace=test
kyuubi.frontend.connection.url.use.hostname=false
kyuubi.frontend.thrift.binary.bind.port=10009
kyuubi.frontend.thrift.http.bind.port=10010
kyuubi.frontend.rest.bind.port=10099
kyuubi.frontend.mysql.bind.port=3309
kyuubi.frontend.protocols=REST,THRIFT_BINARY

#HA configuration
kyuubi.ha.addresses=X.X.X.X:2181,Y.Y.Y.Y:2181

## session configuration
kyuubi.session.idle.timeout=PT6H
kyuubi.session.check.interval=PT5M

## engine common configuration
kyuubi.engine.share.level=GROUP
kyuubi.session.engine.check.interval=PT1M
kyuubi.session.engine.initialize.timeout=PT5M
kyuubi.session.engine.login.timeout=PT30S
kyuubi.session.engine.check.interval=PT1M

# 引擎空闲超时（没有活跃 session 时回收引擎释放内存）
kyuubi.session.engine.idle.timeout=PT30M



#trino开启了https需要以下配置
kyuubi.engine.trino.connection.insecure.enabled=true
kyuubi.engine.trino.connection.user=admin
kyuubi.engine.trino.connection.password=admin
# Trino 引擎 JVM 堆内存 默认 1g 
kyuubi.engine.trino.memory=2g
# Trino 客户端 HTTP 请求超时（默认 2 分钟，复杂查询建议加大）
kyuubi.trino.client.request.timeout=PT10M

#spark engine
kyuubi.kubernetes.spark.cleanupTerminatedDriverPod.kind=COMPLETED
kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M
kyuubi.engine.spark.group.admin.idle.timeout = PT480M
kyuubi.engine.spark.group.etl.idle.timeout = PT480M
kyuubi.kubernetes.terminatedApplicationRetainPeriod=PT1M

Kyuubi Engine Configurations

spark.master=k8s://https://kubernetes.default.svc:443
spark.submit.deployMode=cluster
spark.kubernetes.namespace=test
spark.kubernetes.container.image=spark:3.5.4-arm
spark.kubernetes.authenticate.driver.serviceAccountName=bdp-dolphin
spark.kubernetes.file.upload.path=s3a://test/kyuubi/spark
# S3A configuration
spark.hadoop.fs.s3a.impl=org.apache.hadoop.fs.s3a.S3AFileSystem
spark.hadoop.fs.s3a.endpoint=http://test:8060
spark.hadoop.fs.s3a.access.key=xxxxx
spark.hadoop.fs.s3a.secret.key=yyyyyyy
spark.hadoop.fs.s3a.path.style.access=true
spark.hadoop.fs.s3a.fast.upload=true
spark.sql.extensions=org.apache.kyuubi.plugin.spark.authz.ranger.RangerSparkExtension,org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
spark.sql.catalog.paimon=org.apache.paimon.spark.SparkCatalog
spark.sql.catalog.paimon.warehouse=s3a://test/paimon
spark.hive.metastore.uris=thrift://bdp-hivemetastore:9083
spark.hadoop.mapreduce.fileoutputcommitter.marksuccessfuljobs=false
spark.sql.adaptive.enabled=true

Additional context

No response

Are you willing to submit PR?

• Yes. I would be willing to submit a PR with guidance from the Kyuubi community to fix.
• No. I cannot submit a PR at this time.

[pan3793]

cc @ruanwenjun, do you have any ideas on this? I have seen a few similar reports from the Dolphin Scheduler users

[A-little-bit-of-data]

cc @ruanwenjun, do you have any ideas on this? I have seen a few similar reports from the Dolphin Scheduler users

I'd like to know how to prevent this session from stopping. I understand that if a task is running, the session shouldn't be closed.

[ruanwenjun]

This appears to be a bug in DolphinScheduler's datasource connection pool management and is unrelated to Kyuubi.

Specifically, the issue is here:

https://github.com/apache/dolphinscheduler/blob/7fdc30981224a8c7cc932817ec0b0793191fb635/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/plugin/DataSourceClientProvider.java#L45-L53

The connection pool is closed after it has existed for two hours. If a task is still using that connection pool when it gets closed, an error can occur. My guess is that this is what is happening in your case.

This issue needs to be fixed in DolphinScheduler itself. In the 3.3.x series, we removed the connection pool entirely, so this particular problem should no longer occur there. cc @pan3793 @A-little-bit-of-data

[A-little-bit-of-data]

This appears to be a bug in DolphinScheduler's datasource connection pool management and is unrelated to Kyuubi.

Specifically, the issue is here:

https://github.com/apache/dolphinscheduler/blob/7fdc30981224a8c7cc932817ec0b0793191fb635/dolphinscheduler-datasource-plugin/dolphinscheduler-datasource-api/src/main/java/org/apache/dolphinscheduler/plugin/datasource/api/plugin/DataSourceClientProvider.java#L45-L53

The connection pool is closed after it has existed for two hours. If a task is still using that connection pool when it gets closed, an error can occur. My guess is that this is what is happening in your case.

This issue needs to be fixed in DolphinScheduler itself. In the 3.3.x series, we removed the connection pool entirely, so this particular problem should no longer occur there. cc @pan3793 @A-little-bit-of-data

Thank you so much @ruanwenjun @pan3793

The connection pool is closed after two hours. Is this mandatory? Does this mean that data sources like kyuubi are unsuitable for connection pooling?

[ruanwenjun]

@A-little-bit-of-data Yes, this is mandatory. This is a design bug at DS.

Using kyuubi connection pool is OK, but need to be careful: if connections in the pool are currently in use, shouldn't close the connection pool.

We removed the connection pool in DolphinScheduler, mainly to avoid session parameter contamination issues. When a connection pool is used, if a task sets session-level parameters, the connection is not closed after the task finishes. As a result, those session parameters may persist and be shared across multiple tasks that reuse the same connection, leading to unexpected behavior.

And in DolphinScheduler, about 99% of the workloads are ETL jobs rather than interactive queries, so the benefits of using a connection pool are quite limited. In this scenario, maintaining a connection pool provides little value compared to the potential issues it can introduce.

[Slashoper]

Threads that are actively in use in the thread pool will not be terminated.

MAX_LIFETIME 30 minutes: Maximum lifetime of a connection in the pool. Active connections will not be terminated.

[pan3793]

@Slashoper, paste the plain text or GitHub link instead of the picture when you want to share the code; pictures cause trouble for searching.

[ruanwenjun]

Threads that are actively in use in the thread pool will not be terminated.

@Slashoper These parameters may not have much effect when directly shutting down the Hikari connection pool. Shutting down Hikari forcibly interrupts and closes connections that are currently in use.

https://github.com/brettwooldridge/HikariCP/blob/bba167f0a28905e8e63083cd7b5cbf479263271a/src/main/java/com/zaxxer/hikari/pool/HikariPool.java#L201-L255

The main logic is

this.abortActiveConnections(assassinExecutor);