[INLONG-11401][Manager] Support Dolphinscheduler schedule engine #11468
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
emptyOVO
force-pushed
the
INLONG-11401
branch
6 times, most recently
from
bd7fb5d
to
6bf84a2
Compare
emptyOVO
force-pushed
the
INLONG-11401
branch
from
55c7d24
to
b0b99c4
Compare
|
Nice work! |
Yes, natively, only Crontab expressions are accepted for task scheduling. And i also check DS related implements, their underlying implementation were similar to inlong's built-in engine, uses quartz for scheduling，the difference is that DS only use |
aloyszhang
previously approved these changes
Nov 10, 2024
dockerzhang
previously approved these changes
Nov 12, 2024
fuweng11
reviewed
Nov 12, 2024
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...ava/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleContainerTestEnv.java
Outdated
Show resolved
Hide resolved
inlong-manager/manager-web/src/main/resources/application-dev.properties
Outdated
Show resolved
Hide resolved
inlong-manager/manager-web/src/main/resources/application-dev.properties
Show resolved
Hide resolved
inlong-manager/manager-web/src/main/resources/application-prod.properties
Outdated
Show resolved
Hide resolved
inlong-manager/manager-web/src/main/resources/application-test.properties
Outdated
Show resolved
Hide resolved
fuweng11
reviewed
Nov 12, 2024
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
...src/main/java/org/apache/inlong/manager/schedule/dolphinscheduler/DolphinScheduleEngine.java
Outdated
Show resolved
Hide resolved
emptyOVO
dismissed stale reviews from dockerzhang and aloyszhang
via
be574d1
emptyOVO
force-pushed
the
INLONG-11401
branch
from
be574d1
to
44da2e2
Compare
fuweng11
reviewed
Nov 14, 2024
| * @param searchVal The name of the project to search for. | ||
| * @return The unique project ID if found, or 0 if not found or an error occurs. | ||
| */ | ||
| public long checkAndGetUniqueId(String url, String token, String searchVal) { |
There was a problem hiding this comment.
As a Utils class, it should not be used through the new method. It is recommended to refer to HttpUtils.
fuweng11
reviewed
Nov 14, 2024
| /** | ||
| * Exceptions occur in the schedule procedure. | ||
| * */ | ||
| public class DolphinScheduleException extends RuntimeException { |
There was a problem hiding this comment.
This exception class has no meaning except for exceptions marked as DolphinSchedule.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes #11401
Motivation
dolphinschedulerpackage inorg.apache.inlong.manager.scheduleModifications
Because DS officially does not provide SDK to call, only provides openAPI call mode, so all scheduling behavior is based on open-api requests
Provides some disaster recovery logic
ConcurrentHashMapto store schedule data ensure thread safety during start, running or stopHow to use
inlong managerperforms dependency injection at startup via the @value annotation.start()or.stop()to manage DS engineVerifying this change
How is the token generated?
DS url
for example：
http://{ip}:{port}/dolphinschedulertestRegisterScheduleInfo
the callback request was successfully sent to inlong, and the related tasks are processed in flink
log info:
log info:
(Please pick either of the following options)
This change is a trivial rework/code cleanup without any test coverage.
This change is already covered by existing tests, such as:
(please describe tests)
This change added tests and can be verified as follows:
(example:)
Documentation