Spring ShutDown Hook死鎖現(xiàn)象

偶然出現(xiàn)一次項(xiàng)目異常spring卻沒有正常停止的情況，最終發(fā)現(xiàn)是Spring Shutdown導(dǎo)致的死鎖現(xiàn)象。

某個(gè)框架里嵌入了類似這樣的一段代碼

@Component
public class ShutDownHookTest implements ApplicationListener<ContextRefreshedEvent> {
    @Override
    public void onApplicationEvent(ContextRefreshedEvent event) {
        if (onException) {
		    System.out.println("test shutdown hook deadlock");
			System.exit(0);
		}
 
    }
}

它的邏輯就是想要在出現(xiàn)異常后，通過System.exit來確保應(yīng)用程序退出。

而且沒有使用異步事件，是在主線程下跑了System.exit，然后就發(fā)現(xiàn)springboot server還是正常運(yùn)行著的。

而且程序看著好像也沒問題，由于我們是dubbo服務(wù)化系統(tǒng)，在測(cè)試環(huán)境上服務(wù)還是正常的。

這很明顯不符常理，正常來說，System.exit這樣的指令是spring能夠感知到的，并且會(huì)執(zhí)行shutDown處理的，先來看看Spring 注冊(cè)ShutdownHook

public abstract class AbstractApplicationContext extends DefaultResourceLoader
		implements ConfigurableApplicationContext {
	@Override
	public void registerShutdownHook() {
		if (this.shutdownHook == null) {
			// No shutdown hook registered yet.
			this.shutdownHook = new Thread(SHUTDOWN_HOOK_THREAD_NAME) {
				@Override
				public void run() {
				    //重點(diǎn)在這里獲取startupShutdownMonitor的監(jiān)視器鎖
					synchronized (startupShutdownMonitor) {
						doClose();
					}
				}
			};
			Runtime.getRuntime().addShutdownHook(this.shutdownHook);
		}
	}
 
	protected void doClose() {
		// Check whether an actual close attempt is necessary...
		if (this.active.get() && this.closed.compareAndSet(false, true)) {
			if (logger.isDebugEnabled()) {
				logger.debug("Closing " + this);
			}
 
			if (!NativeDetector.inNativeImage()) {
				LiveBeansView.unregisterApplicationContext(this);
			}
 
			try {
				// Publish shutdown event.
				publishEvent(new ContextClosedEvent(this));
			}
			catch (Throwable ex) {
				logger.warn("Exception thrown from ApplicationListener handling ContextClosedEvent", ex);
			}
 
			// Stop all Lifecycle beans, to avoid delays during individual destruction.
			if (this.lifecycleProcessor != null) {
				try {
					this.lifecycleProcessor.onClose();
				}
				catch (Throwable ex) {
					logger.warn("Exception thrown from LifecycleProcessor on context close", ex);
				}
			}
 
			// Destroy all cached singletons in the context's BeanFactory.
			destroyBeans();
 
			// Close the state of this context itself.
			closeBeanFactory();
 
			// Let subclasses do some final clean-up if they wish...
			onClose();
 
			// Reset local application listeners to pre-refresh state.
			if (this.earlyApplicationListeners != null) {
				this.applicationListeners.clear();
				this.applicationListeners.addAll(this.earlyApplicationListeners);
			}
 
			// Switch to inactive.
			this.active.set(false);
		}
	}
}	

也就是說spring新起了一個(gè)線程，加入了JVM Shutdown鉤子函數(shù)。

重點(diǎn)是close前要獲取startupShutdownMonitor的對(duì)象監(jiān)視器鎖，這個(gè)鎖看著就很眼熟，Spring在refresh時(shí)也會(huì)獲取這把鎖。

public abstract class AbstractApplicationContext extends DefaultResourceLoader
		implements ConfigurableApplicationContext {
 
	@Override
	public void refresh() throws BeansException, IllegalStateException {
		synchronized (this.startupShutdownMonitor) {
			StartupStep contextRefresh = this.applicationStartup.start("spring.context.refresh");
 
			// Prepare this context for refreshing.
			prepareRefresh();
 
			// Tell the subclass to refresh the internal bean factory.
			ConfigurableListableBeanFactory beanFactory = obtainFreshBeanFactory();
 
			// Prepare the bean factory for use in this context.
			prepareBeanFactory(beanFactory);
 
			......
		}
	}
}	

這個(gè)時(shí)候我們猜想，是獲取startupShutdownMonitor死鎖了。

jstack打下線程?？纯?/h2>

 "SpringContextShutdownHook" #18 prio=5 os_prio=0 tid=0x0000000024e00800 nid=0x407c waiting for monitor entry [0x000000002921f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
        at org.springframework.context.support.AbstractApplicationContext$1.run(AbstractApplicationContext.java:991)
        - waiting to lock <0x00000006c494f430> (a java.lang.Object)

"main" #1 prio=5 os_prio=0 tid=0x0000000002de4000 nid=0x1ff4 in Object.wait() [0x0000000002dde000]
   java.lang.Thread.State: WAITING (on object monitor)
        at java.lang.Object.wait(Native Method)
        - waiting on <0x00000006c4a43118> (a org.springframework.context.support.AbstractApplicationContext$1)
        at java.lang.Thread.join(Thread.java:1252)
        - locked <0x00000006c4a43118> (a org.springframework.context.support.AbstractApplicationContext$1)
        at java.lang.Thread.join(Thread.java:1326)
        at java.lang.ApplicationShutdownHooks.runHooks(ApplicationShutdownHooks.java:107)
        at java.lang.ApplicationShutdownHooks$1.run(ApplicationShutdownHooks.java:46)
        at java.lang.Shutdown.runHooks(Shutdown.java:123)
        at java.lang.Shutdown.sequence(Shutdown.java:167)
        at java.lang.Shutdown.exit(Shutdown.java:212)
        - locked <0x00000006c4845128> (a java.lang.Class for java.lang.Shutdown)
        at java.lang.Runtime.exit(Runtime.java:109)
        at java.lang.System.exit(System.java:971)
        at io.seata.server.ShutDownHookTest.onApplicationEvent(ShutDownHookTest.java:12)
        at io.seata.server.ShutDownHookTest.onApplicationEvent(ShutDownHookTest.java:7)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.doInvokeListener(SimpleApplicationEventMulticaster.java:176)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.invokeListener(SimpleApplicationEventMulticaster.java:169)
        at org.springframework.context.event.SimpleApplicationEventMulticaster.multicastEvent(SimpleApplicationEventMulticaster.java:143)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:421)
        at org.springframework.context.support.AbstractApplicationContext.publishEvent(AbstractApplicationContext.java:378)
        at org.springframework.context.support.AbstractApplicationContext.finishRefresh(AbstractApplicationContext.java:938)
        at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:586)
        - locked <0x00000006c494f430> (a java.lang.Object)
        at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:144)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:771)
        at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:763)

乍一看jstack并沒有提示線程死鎖（jvisualvm、jconsle之類的工具也不行），但是從線程棧來看：

• main線程先獲取到了startupShutdownMonitor鎖 <0x00000006c494f430>
• SpringContextShutdownHook線程在等待startupShutdownMonitor鎖
• main線程掉了Thread.join阻塞在獲取<0x00000006c4a43118>這把鎖

根本原因是main線程調(diào)System.exit阻塞住了，一直往下追蹤，會(huì)發(fā)現(xiàn)阻塞在ApplicationShutdownHooks這里

class ApplicationShutdownHooks {
    /* Iterates over all application hooks creating a new thread for each
     * to run in. Hooks are run concurrently and this method waits for
     * them to finish.
     */
    static void runHooks() {
        Collection<Thread> threads;
        synchronized(ApplicationShutdownHooks.class) {
            threads = hooks.keySet();
            hooks = null;
        }
 
        for (Thread hook : threads) {
            hook.start();
        }
        for (Thread hook : threads) {
            while (true) {
                try {
					// 等待shutdow線程結(jié)束
                    hook.join();
                    break;
                } catch (InterruptedException ignored) {
                }
            }
        }
    }
}

總結(jié)

整個(gè)死鎖的流程：

• main線程-spring refresh開始時(shí)會(huì)獲取startupShutdownMonitor對(duì)象監(jiān)視器鎖
• main線程-在spring refresh還未完成的時(shí)候，觸發(fā)了System.exit指令
• SpringContextShutdownHook線程-SpringContextShutdownHook線程開始工作，等待獲取startupShutdownMonitor對(duì)象監(jiān)視器鎖
• main線程調(diào)用Thread.join等待SpringContextShutdownHook線程結(jié)束

所以在Spring未完成refresh時(shí)，是不能夠觸發(fā)System.exit指令的

以上為個(gè)人經(jīng)驗(yàn)，希望能給大家一個(gè)參考，也希望大家多多支持腳本之家。

最新評(píng)論