-
Notifications
You must be signed in to change notification settings - Fork 1.8k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[hotfix] [DOC] Fix missing areas in the development documentation #7959 #7967
Open
YOMO-Lee
wants to merge
6
commits into
apache:dev
Choose a base branch
from
YOMO-Lee:common-doc-7959
base: dev
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Changes from 1 commit
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
333c6ab
[hotfix] [DOC] Fix missing areas in the development documentation #7959
YOMO-Lee 45fb3e6
[BUG] [DOC] Fix missing areas in the development documentation #7959
YOMO-Lee 4b6d801
Merge remote-tracking branch 'origin/common-doc-7959' into common-doc…
YOMO-Lee ef3b51b
[BUG] [DOC] Fix missing areas in the development documentation
YOMO-Lee c40a545
Merge branch 'refs/heads/dev' into common-doc-7959
YOMO-Lee 0bd203d
Merge branch 'refs/heads/dev' into common-doc-7959
YOMO-Lee File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -10,7 +10,7 @@ SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过 | |
### **工程结构** | ||
|
||
- ../`seatunnel-connectors-v2` connector-v2代码实现 | ||
- ../`seatunnel-translation` connector-v2的翻译层 | ||
- ../`seatunnel-translation` connector-v2的翻译层 | ||
- ../`seatunnel-transform-v2` transform-v2代码实现 | ||
- ../seatunnel-e2e/`seatunnel-connector-v2-e2e` connector-v2端到端测试 | ||
- ../seatunnel-examples/`seatunnel-flink-connector-v2-example` seatunnel connector-v2的flink local运行的实例 | ||
|
@@ -39,14 +39,19 @@ SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过 | |
|
||
3.新建两个package分别对应source和sink | ||
|
||
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.source | ||
|
||
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.sink | ||
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.source | ||
package org.apache.seatunnel.connectors.seatunnel.{连接器名}.sink | ||
|
||
4.将连接器信息添加到在项目根目录的plugin-mapping.properties文件中. | ||
|
||
5.将连接器添加到seatunnel-dist/pom.xml,这样连接器jar就可以在二进制包中找到. | ||
|
||
6.source端有几个必须实现的类,分别是{连接器名}Source、{连接器名}SourceFactor、{连接器名}SourceReader,具体可以参考其他连接器 | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. missing character 'y', {连接器名}SourceFactor -> {连接器名}SourceFactory |
||
|
||
7.{连接器名}SourceFactory 里面需要在类名上标注 **@AutoService(Factory.class)** 注解,并且除了必须实现的方法外,需要额外再重写一个 createSource 方法 | ||
|
||
8.{连接器名}Source 需要重写 getProducedCatalogTables 方法 | ||
|
||
### 启动类 | ||
|
||
和老的启动类分开,我们创建了两个新的启动类工程,分别是`seatunnel-core/seatunnel-flink-starter`和`seatunnel-core/seatunnel-spark-starter`. | ||
|
@@ -103,7 +108,7 @@ SeaTunnel为与计算引擎进行解耦,设计了新的连接器API,通过 | |
中调用`SourceReader.Context.signalNoMoreElement` | ||
通知SeaTunnel没有数据读取了,那么就可以利用这100条数据进行批处理。流处理没有这个要求,那么大多数流批一体的SourceReader都会出现如下代码: | ||
|
||
```java | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Why delete code type? |
||
``` | ||
if (Boundedness.BOUNDED.equals(context.getBoundedness())) { | ||
// signal to the source that we have reached the end of the data. | ||
context.signalNoMoreElement(); | ||
|
@@ -154,13 +159,12 @@ Sink可以根据组件属性进行选择,到底是只实现`SinkCommitter`或` | |
为了实现自动化的创建Source或者Sink,我们需要连接器能够声明并返回创建他们所需要的参数列表和每个参数的校验规则。为了实现这个目标,我们定义了TableSourceFactory和TableSinkFactory, | ||
建议将其放在和SeaTunnelSource或SeaTunnelSink实现类同一目录下,方便寻找。 | ||
|
||
- `factoryIdentifier` 用于表明当前Factory的名称,这个值应该和`getPluginName`返回的值一致,这样后续如果使用Factory来创建Source/Sink, | ||
就能实现无缝切换。 | ||
- `createSink` 和 `createSource` 分别是创建Source和Sink的方法,目前不用实现。 | ||
- `factoryIdentifier` 用于表明当前Factory的名称,这个值应该和`getPluginName`返回的值一致,这样后续如果使用Factory来创建Source/Sink,就能实现无缝切换。 | ||
- `createSink` 和 `createSource` 分别是创建Source和Sink的方法。 | ||
- `optionRule` 返回的是参数逻辑,用于表示我们的连接器参数哪些支持,哪些参数是必须(required)的,哪些参数是可选(optional)的,哪些参数是互斥(exclusive)的,哪些参数是绑定(bundledRequired)的。 | ||
这个方法会在我们可视化创建连接器逻辑的时候用到,同时也会用于根据用户配置的参数生成完整的参数对象,然后连接器开发者就不用在Config里面一个个判断参数是否存在,直接使用即可。 | ||
可以参考现有的实现,比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema,所以采用了通用的Option, | ||
需要Schema则可以引用`org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`。 | ||
这个方法会在我们可视化创建连接器逻辑的时候用到,同时也会用于根据用户配置的参数生成完整的参数对象,然后连接器开发者就不用在Config里面一个个判断参数是否存在,直接使用即可。 | ||
可以参考现有的实现,比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema,所以采用了通用的Option, | ||
需要Schema则可以引用`org.apache.seatunnel.api.table.catalog.CatalogTableUtil.SCHEMA`。 | ||
|
||
别忘记添加`@AutoService(Factory.class)` 到类上面。这个Factory即TableSourceFactory 和 TableSinkFactory的父类。 | ||
|
||
|
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why delete code type?