Skip to content

Pipe: Optimize Drop Pipe high priority tasks cannot obtain SubTask object lock #15404

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
SteveYurongSu merged 11 commits into from
May 29, 2025
Merged

Pipe: Optimize Drop Pipe high priority tasks cannot obtain SubTask object lock #15404

Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
82fd002
Pipe: Optimize Drop Pipe high priority tasks cannot obtain SubTask ob…
luoluoyuyu Apr 24, 2025
96ce5df
update PipeAbstractConnectorSubtask
luoluoyuyu Apr 24, 2025
1ffbbe4
update PipeAbstractConnectorSubtask
luoluoyuyu Apr 30, 2025
695ca9d
update PipeAbstractConnectorSubtask
luoluoyuyu Apr 30, 2025
233d9a6
update PipeAbstractConnectorSubtask
luoluoyuyu Apr 30, 2025
5bfce65
Merge remote-tracking branch 'iotdb/master' into fix-drop-slow
luoluoyuyu May 27, 2025
79f35ee
Merge remote-tracking branch 'iotdb/master' into fix-drop-slow
luoluoyuyu May 28, 2025
64002c5
fix
luoluoyuyu May 28, 2025
e21c5ff
Merge branch 'master' of https://github.com/apache/iotdb into pr/15404
SteveYurongSu May 28, 2025
2fc8278
Update PipeConnectorSubtask.java
SteveYurongSu May 28, 2025
fbd3167
Update PipeAbstractConnectorSubtask.java
SteveYurongSu May 28, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Update PipeConnectorSubtask.java
  • Loading branch information
@SteveYurongSu
SteveYurongSu committed May 28, 2025
commit 2fc8278d6c5a4dda89bb80f091b51563301ef06f
22 changes: 9 additions & 13 deletions ...main/java/org/apache/iotdb/db/pipe/agent/task/subtask/connector/PipeConnectorSubtask.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -237,35 +237,31 @@ public void discardEventsOfPipe(final String pipeNameToDrop, int regionId) {
// synchronized to use the lastEvent & lastExceptionEvent
synchronized (this) {
// Here we discard the last event, and re-submit the pipe task to avoid that the pipe task
Comment thread
SteveYurongSu marked this conversation as resolved.
// has
// stopped submission but will not be stopped by critical exceptions, because when it
// acquires
// lock, the pipe is already dropped, thus it will do nothing.
// Note that since we use a new thread to stop all the pipes, we will not encounter deadlock
// here. Or else we will.
// has stopped submission but will not be stopped by critical exceptions, because when it
// acquires lock, the pipe is already dropped, thus it will do nothing. Note that since we
// use a new thread to stop all the pipes, we will not encounter deadlock here. Or else we
// will.
if (lastEvent instanceof EnrichedEvent
&& pipeNameToDrop.equals(((EnrichedEvent) lastEvent).getPipeName())
&& regionId == ((EnrichedEvent) lastEvent).getRegionId()) {
// Do not clear last event's reference count because it may be on transferring
// Do not clear the last event's reference counts because it may be on transferring
lastEvent = null;
// Submit self to avoid that the lastEvent has been retried "max times" times and has
// stopped executing.
// 1. If the last event is still on execution, or submitted by the previous "onSuccess" or
// "onFailure", the "submitSelf" cause nothing.
// "onFailure", the "submitSelf" causes nothing.
// 2. If the last event is waiting the instance lock to call "onSuccess", then the
// callback
// method will skip this turn of submission.
// callback method will skip this turn of submission.
// 3. If the last event is waiting to call "onFailure", then it will be ignored because
// the
// last event has been set to null.
// the last event has been set to null.
// 4. If the last event has called "onFailure" and caused the subtask to stop submission,
// it's submitted here and the "report" will wait for the "drop pipe" lock to stop all
// the pipes with critical exceptions. As illustrated above, the "report" will do
// nothing.
submitSelf();
}

// We only clear the lastEvent's reference count when it's already on failure. Namely, we
// We only clear the lastEvent's reference counts when it's already on failure. Namely, we
// clear the lastExceptionEvent. It's safe to potentially clear it twice because we have the
// "nonnull" detection.
if (lastExceptionEvent instanceof EnrichedEvent
Expand Down