java客戶端Etcd官方倉(cāng)庫(kù)jetcd中KeepAlive接口實(shí)現(xiàn)
前言
Etcd的Java客戶端有很多開(kāi)源實(shí)現(xiàn),Jetcd是Etcd官方倉(cāng)庫(kù)的Java客戶端,整體api接口設(shè)計(jì)實(shí)現(xiàn)和官方go客戶端類(lèi)似,簡(jiǎn)潔易用。其中,租期續(xù)約的接口提供了兩個(gè)分別是keepAliveOnce和keepAlive。功能如其名,keepAliveOnce是單次續(xù)約的接口,如果要保持租約,需要手動(dòng)觸發(fā)這個(gè)接口,所以這個(gè)接口基本不用。而keepAlive是自動(dòng)續(xù)約保活的接口。大多數(shù)場(chǎng)景下,使用keepAlive即可,但是針對(duì)不同的場(chǎng)景,我們還需要考慮幾個(gè)問(wèn)題,如租約ttl的設(shè)置,以及keepAlive異常時(shí)的處理。
Jetcd項(xiàng)目地址:https://github.com/etcd-io/jetcd
背景問(wèn)題
我們有一個(gè)基于mysql的binlog訂閱數(shù)據(jù)變更的應(yīng)用,線上有非常重要的應(yīng)用基于這個(gè)服務(wù),因?yàn)榇嬖趩吸c(diǎn)故障,后面使用了jetcd的lock + keepAlive的機(jī)制實(shí)現(xiàn)了主備服務(wù)秒級(jí)切換的功能,具體參見(jiàn)《高可用架構(gòu)etcd選主故障主備秒級(jí)切換實(shí)現(xiàn)》,系統(tǒng)上線運(yùn)行后發(fā)現(xiàn),binlog的服務(wù)經(jīng)常切換發(fā)生主備切換,而實(shí)際情況是,binlog的服務(wù)非常穩(wěn)定,在沒(méi)有上線主備切換服務(wù)前,從來(lái)沒(méi)有發(fā)生過(guò)線上binlog服務(wù)宕掉的情況。最后查明問(wèn)題出在了租約TTL的設(shè)置上面。這里先拋出問(wèn)題和定位,下面先看下Jetcd的keepAlive具體實(shí)現(xiàn),然后在分析為什么導(dǎo)致這個(gè)問(wèn)題。
KeepAlive實(shí)現(xiàn)
先看下keepAlive的用法
private long acquireActiveLease() throws InterruptedException, ExecutionException {
long leaseId = leaseClient.grant(leaseTTL).get().getID();
logger.debug("LeaderSelector get leaseId:[{}] and ttl:[{}]", leaseId, leaseTTL);
this.leaseCloser = leaseClient.keepAlive(leaseId, new StreamObserver() {
@Override
public void onNext(LeaseKeepAliveResponse value) {
logger.debug("LeaderSelector lease keeps alive for [{}]s:", value.getTTL());
}
@Override
public void onError(Throwable t) {
logger.debug("LeaderSelector lease renewal Exception!", t.fillInStackTrace());
cancelTask();
}
@Override
public void onCompleted() {
logger.info("LeaderSelector lease renewal completed! start canceling task.");
cancelTask();
}
});
return leaseId;
}租約實(shí)現(xiàn)都在LeaseImpl類(lèi)里,通過(guò)EtcdClient拿到LeaseImpl實(shí)例后,首先通過(guò)grant方法設(shè)置ttl拿到租約的id,然后將租約作為入?yún)⒄{(diào)用keepAlive方法,第二個(gè)入?yún)⑹且粋€(gè)觀察者對(duì)象,內(nèi)置了三個(gè)接口,分別是onNext:確定下一次租約續(xù)約時(shí)間后觸發(fā),onError:續(xù)約異常時(shí)觸發(fā),onCompleted:租約過(guò)期后觸發(fā)。
keepAlive方法代碼:
public synchronized CloseableClient keepAlive(long leaseId, StreamObserverobserver) {
if (this.closed) {
throw newClosedLeaseClientException();
}
KeepAlive keepAlive = this.keepAlives.computeIfAbsent(leaseId, (key) -> new KeepAlive(leaseId));
keepAlive.addObserver(observer);
if (!this.hasKeepAliveServiceStarted) {
this.hasKeepAliveServiceStarted = true;
this.start();
}
return new CloseableClient() {
@Override
public void close() {
keepAlive.removeObserver(observer);
}
};
}LeaseImpl內(nèi)部維護(hù)了一個(gè)以LeaseId為key,KeepAlive對(duì)象為value的map,KeepAlive的類(lèi)中維護(hù)了一個(gè)StreamObserver集合,到期時(shí)間deadLine,下次續(xù)約時(shí)間nextKeepAlive和續(xù)約leaseId。第一次調(diào)用keepAlive方法時(shí)會(huì)觸發(fā)start,啟動(dòng)續(xù)約的線程(sendKeepAliveExecutor())和檢查是否過(guò)期的線程(deadLineExecutor())。
private void sendKeepAliveExecutor() {
this.keepAliveResponseObserver = Observers.observer(
response -> processKeepAliveResponse(response),
error -> processOnError()
);
this.keepAliveRequestObserver = this.leaseStub.leaseKeepAlive(this.keepAliveResponseObserver);
this.keepAliveFuture = scheduledExecutorService.scheduleAtFixedRate(
() -> {
// send keep alive req to the leases whose next keep alive is before now.
this.keepAlives.entrySet().stream()
.filter(entry -> entry.getValue().getNextKeepAlive() < System.currentTimeMillis())
.map(Entry::getKey)
.map(leaseId -> LeaseKeepAliveRequest.newBuilder().setID(leaseId).build())
.forEach(keepAliveRequestObserver::onNext);
},
0,
500,
TimeUnit.MILLISECONDS
);
}sendKeepAliveExecutor方法是整個(gè)keepAlive功能實(shí)現(xiàn)的核心,這個(gè)方法在LeaseImpl實(shí)例里只會(huì)被觸發(fā)一次,開(kāi)啟了一個(gè)時(shí)間間隔為500毫秒的的定時(shí)任務(wù)調(diào)度。每次從keepAlives中篩選出nextkeepAlive時(shí)間小于當(dāng)前時(shí)間的KeepAlive對(duì)象,觸發(fā)續(xù)約。
nextkeepAlive初始化值就是創(chuàng)建KeepAlive實(shí)例時(shí)的當(dāng)前時(shí)間,然后在續(xù)約的響應(yīng)流觀察者實(shí)例中,執(zhí)行了processKeepAliveResponse方法,在這個(gè)里面維護(hù)了KeepAlive對(duì)象的nextkeepAlive。
private synchronized void processKeepAliveResponse(io.etcd.jetcd.api.LeaseKeepAliveResponse leaseKeepAliveResponse) {
if (this.closed) {
return;
}
final long leaseID = leaseKeepAliveResponse.getID();
final long ttl = leaseKeepAliveResponse.getTTL();
final KeepAlive ka = this.keepAlives.get(leaseID);
if (ka == null) {
// return if the corresponding keep alive has closed.
return;
}
if (ttl > 0) {
long nextKeepAlive = System.currentTimeMillis() + ttl * 1000 / 3;
ka.setNextKeepAlive(nextKeepAlive);
ka.setDeadLine(System.currentTimeMillis() + ttl * 1000);
ka.onNext(leaseKeepAliveResponse);
} else {
// lease expired; close all keep alive
this.removeKeepAlive(leaseID);
ka.onError(
newEtcdException(
ErrorCode.NOT_FOUND,
"etcdserver: requested lease not found"
)
);
}
}可以看到,在首次續(xù)約后的響應(yīng)處理中,nextKeepAlive被設(shè)置為當(dāng)前時(shí)間加上ttl的1/3時(shí)間后,也就是說(shuō)如果我們?cè)O(shè)置一個(gè)key的過(guò)期時(shí)間為6s,那么在使用keepAlive時(shí)續(xù)期的時(shí)間間隔為,每2s執(zhí)行續(xù)約一次。如果ttl小于零,說(shuō)明key已經(jīng)過(guò)期被刪除了,就直接觸發(fā)onError,傳遞了一個(gè)requested lease not found的異常對(duì)象。
文末小結(jié)
1、回到最上面binlog的主備頻繁切換的問(wèn)題,由于我們將ttl的時(shí)間設(shè)置的過(guò)小5s。只要client和etcd 服務(wù)失聯(lián)5s以上,期間可能由于各種原因?qū)е耴eepAlive沒(méi)有正常續(xù)約上,就會(huì)觸發(fā)主備切換。這個(gè)時(shí)候binlog服務(wù)本身是沒(méi)有任何問(wèn)題的,卻要因?yàn)槭ヮI(lǐng)導(dǎo)權(quán),而選擇自殺。后面將ttl調(diào)整到了20s后,主備切換就沒(méi)有那么敏感了。
2、還有一個(gè)場(chǎng)景,在將etcd作為服務(wù)注冊(cè)中心時(shí),也會(huì)使用到keepAlive,即使設(shè)置了ttl為20s,還是有可能沒(méi)有續(xù)約上,導(dǎo)致注冊(cè)的服務(wù)過(guò)期被刪了,這個(gè)時(shí)候,我們的服務(wù)進(jìn)程還是健康的。這個(gè)場(chǎng)景下,需要在onError、onCompleted事件中重新獲取租約以及添加新的KeepAlive。
以上就是java客戶端Etcd官方倉(cāng)庫(kù)jetcd中KeepAlive接口實(shí)現(xiàn)的詳細(xì)內(nèi)容,更多關(guān)于jetcd中的KeepAlive實(shí)現(xiàn)的資料請(qǐng)關(guān)注腳本之家其它相關(guān)文章!
相關(guān)文章
SpringBoot環(huán)境Druid數(shù)據(jù)源使用及特點(diǎn)
Druid 是目前比較流行的高性能的,分布式列存儲(chǔ)的OLAP框架(具體來(lái)說(shuō)是MOLAP)。本文給大家分享SpringBoot環(huán)境Druid數(shù)據(jù)源使用及特點(diǎn)介紹,感興趣的朋友跟隨小編一起看看吧2021-07-07
SpringBoot之Json的序列化和反序列化問(wèn)題
這篇文章主要介紹了SpringBoot之Json的序列化和反序列化問(wèn)題,具有很好的參考價(jià)值,希望對(duì)大家有所幫助。如有錯(cuò)誤或未考慮完全的地方,望不吝賜教2022-06-06
mybatis-plus用insertBatchSomeColumn方法批量新增指定字段
mybatisPlus底層的新增方法是一條一條的新增的,下面這篇文章主要給大家介紹了關(guān)于mybatis-plus用insertBatchSomeColumn方法批量新增指定字段的相關(guān)資料,文中通過(guò)實(shí)例代碼介紹的非常詳細(xì),需要的朋友可以參考下2023-05-05
Java中Https發(fā)送POST請(qǐng)求[親測(cè)可用]
這篇文章主要介紹了Java中Https發(fā)送POST請(qǐng)求[親測(cè)可用],本文通過(guò)實(shí)例代碼給大家介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或工作具有一定的參考借鑒價(jià)值,需要的朋友可以參考下2021-05-05
手把手帶你實(shí)現(xiàn)一個(gè)萌芽版的Spring容器
大家好,我是老三,Spring是我們最常用的開(kāi)源框架,經(jīng)過(guò)多年發(fā)展,Spring已經(jīng)發(fā)展成枝繁葉茂的大樹(shù),讓我們難以窺其全貌,這節(jié),我們回歸Spring的本質(zhì),五分鐘手?jǐn)]一個(gè)Spring容器,揭開(kāi)Spring神秘的面紗2022-03-03
SpringBoot集成消息隊(duì)列的項(xiàng)目實(shí)踐
本文主要介紹了SpringBoot集成消息隊(duì)列的項(xiàng)目實(shí)踐,文中通過(guò)示例代碼介紹的非常詳細(xì),對(duì)大家的學(xué)習(xí)或者工作具有一定的參考學(xué)習(xí)價(jià)值,需要的朋友們下面隨著小編來(lái)一起學(xué)習(xí)學(xué)習(xí)吧2024-02-02

