Skip to content

Pipe/IoTV2: Try to persist progressIndex in local for resend event more efficently #15599

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Caideyipi merged 24 commits into from
Jun 7, 2025
Merged

Pipe/IoTV2: Try to persist progressIndex in local for resend event more efficently #15599

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
24 commits
Select commit Hold shift + click to select a range
6f9579b
feat: progressIndex persistent in local process
Pengzna May 28, 2025
08d0da4
fix ci and support cancel persist task
Pengzna May 29, 2025
0a60f71
pipeMeta in cn doesn't need to persist progressIndex
Pengzna May 29, 2025
23a9f88
remove duplicate
Pengzna May 29, 2025
f75c005
clean progressIndex background service
Pengzna May 29, 2025
ff31757
fix and add conf
Pengzna May 29, 2025
1c7c373
Update PipeConfig.java
Caideyipi Jun 7, 2025
8ae61a4
Merge branch 'master' of https://github.com/apache/iotdb into pr/15599
Caideyipi Jun 7, 2025
7282ff6
Update PipeMetaDeSerTest.java
Caideyipi Jun 7, 2025
046d00d
cp-gap
Caideyipi Jun 7, 2025
3b7940e
Update PipeTaskMeta.java
Caideyipi Jun 7, 2025
2611712
partial
Caideyipi Jun 7, 2025
e4dd18c
Update PipeRuntimeMeta.java
Caideyipi Jun 7, 2025
b392a9b
Update PipeRuntimeMeta.java
Caideyipi Jun 7, 2025
d287956
Partial
Caideyipi Jun 7, 2025
e2300b9
Update PipeTaskMeta.java
Caideyipi Jun 7, 2025
cd322f5
Added config
Caideyipi Jun 7, 2025
4793d0f
clean
Caideyipi Jun 7, 2025
ee84003
Merge branch 'master' of https://github.com/apache/iotdb into pr/15599
Caideyipi Jun 7, 2025
4e82d19
Change default
Caideyipi Jun 7, 2025
0b71686
Update PipeConnectorConstant.java
Caideyipi Jun 7, 2025
985005f
Partial
Caideyipi Jun 7, 2025
cec8f6d
Update PipeTaskMeta.java
Caideyipi Jun 7, 2025
ce3f699
Update PipeTaskMeta.java
Caideyipi Jun 7, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
cp-gap
  • Loading branch information
@Caideyipi
Caideyipi committed Jun 7, 2025
commit 046d00db21dc6d28679dbc425677cafb732bc939
15 changes: 8 additions & 7 deletions iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/conf/CommonConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -263,7 +263,7 @@ public class CommonConfig {
private long pipeMetaSyncerSyncIntervalMinutes = 3;
private long pipeMetaSyncerAutoRestartPipeCheckIntervalRound = 1;
private boolean pipeAutoRestartEnabled = true;
private long pipeProgressIndexPersistCheckPoint = 20;
private long pipeProgressIndexPersistCheckPointGap = 20;

private boolean pipeAirGapReceiverEnabled = false;
private int pipeAirGapReceiverPort = 9780;
Expand Down Expand Up @@ -1213,17 +1213,18 @@ public void setPipeAutoRestartEnabled(boolean pipeAutoRestartEnabled) {
logger.info("pipeAutoRestartEnabled is set to {}.", pipeAutoRestartEnabled);
}

public long getPipeProgressIndexPersistCheckPoint() {
return pipeProgressIndexPersistCheckPoint;
public long getPipeProgressIndexPersistCheckPointGap() {
return pipeProgressIndexPersistCheckPointGap;
}

public void setPipeProgressIndexPersistCheckPoint(long pipeProgressIndexPersistCheckPoint) {
if (this.pipeProgressIndexPersistCheckPoint == pipeProgressIndexPersistCheckPoint) {
public void setPipeProgressIndexPersistCheckPointGap(long pipeProgressIndexPersistCheckPointGap) {
if (this.pipeProgressIndexPersistCheckPointGap == pipeProgressIndexPersistCheckPointGap) {
return;
}
this.pipeProgressIndexPersistCheckPoint = pipeProgressIndexPersistCheckPoint;
this.pipeProgressIndexPersistCheckPointGap = pipeProgressIndexPersistCheckPointGap;
logger.info(
"pipeProgressIndexPersistCheckPoint is set to {}.", pipeProgressIndexPersistCheckPoint);
"pipeProgressIndexPersistCheckPointGap is set to {}.",
pipeProgressIndexPersistCheckPointGap);
}

public long getPipeConnectorRetryIntervalMs() {
Expand Down
3 changes: 2 additions & 1 deletion .../src/main/java/org/apache/iotdb/commons/pipe/agent/runtime/PipePeriodicalJobExecutor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ public static void shutdownBackgroundService() {
LOGGER.warn("Pipe progressIndex background service did not terminate within {}s", 30);
}
} catch (InterruptedException e) {
LOGGER.warn("Pipe progressIndex background service still doesn't exit after 30s");
LOGGER.warn(
"Pipe progressIndex background service is interrupted while waiting for termination");
Thread.currentThread().interrupt();
}
}
Expand Down
2 changes: 1 addition & 1 deletion ...ode-commons/src/main/java/org/apache/iotdb/commons/pipe/agent/task/meta/PipeTaskMeta.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ public class PipeTaskMeta {
private final AtomicLong updateCount = new AtomicLong(0);
private final AtomicLong lastPersistCount = new AtomicLong(0);
private final long checkPointGap =
PipeConfig.getInstance().getPipeProgressIndexPersistCheckPoint();
PipeConfig.getInstance().getPipeProgressIndexPersistCheckPointGap();
private final int taskIndex;
private final File progressIndexPersistFile;
private final AtomicBoolean isRegisterPersistTask = new AtomicBoolean(false);
Expand Down
7 changes: 4 additions & 3 deletions iotdb-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/config/PipeConfig.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,8 +50,8 @@ public String getPipeProgressIndexPersistDirName() {
return COMMON_CONFIG.getPipeProgressIndexPersistDirName();
}

public long getPipeProgressIndexPersistCheckPoint() {
return COMMON_CONFIG.getPipeProgressIndexPersistCheckPoint();
public long getPipeProgressIndexPersistCheckPointGap() {
return COMMON_CONFIG.getPipeProgressIndexPersistCheckPointGap();
}

public String getPipeHardlinkWALDirName() {
Expand Down Expand Up @@ -459,7 +459,8 @@ public void printAllConfigs() {
LOGGER.info("PipeHardlinkBaseDirName: {}", getPipeHardlinkBaseDirName());
LOGGER.info("PipeHardlinkTsFileDirName: {}", getPipeHardlinkTsFileDirName());
LOGGER.info("PipeProgressIndexPersistDirName: {}", getPipeProgressIndexPersistDirName());
LOGGER.info("PipeProgressIndexPersistCheckPoint: {}", getPipeProgressIndexPersistCheckPoint());
LOGGER.info(
"PipeProgressIndexPersistCheckPointGap: {}", getPipeProgressIndexPersistCheckPointGap());
LOGGER.info("PipeHardlinkWALDirName: {}", getPipeHardlinkWALDirName());
LOGGER.info("PipeHardLinkWALEnabled: {}", getPipeHardLinkWALEnabled());
LOGGER.info("PipeFileReceiverFsyncEnabled: {}", getPipeFileReceiverFsyncEnabled());
Expand Down
6 changes: 3 additions & 3 deletions ...-core/node-commons/src/main/java/org/apache/iotdb/commons/pipe/config/PipeDescriptor.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -100,11 +100,11 @@ public static void loadPipeStaticConfig(CommonConfig config, TrimProperties prop
Boolean.parseBoolean(
properties.getProperty(
"pipe_auto_restart_enabled", String.valueOf(config.getPipeAutoRestartEnabled()))));
config.setPipeProgressIndexPersistCheckPoint(
config.setPipeProgressIndexPersistCheckPointGap(
Long.parseLong(
properties.getProperty(
"pipe_progress_index_persist_check_point",
String.valueOf(config.getPipeProgressIndexPersistCheckPoint()))));
"pipe_progress_index_persist_check_point_gap",
String.valueOf(config.getPipeProgressIndexPersistCheckPointGap()))));

config.setPipeAirGapReceiverEnabled(
Boolean.parseBoolean(
Expand Down
Loading