Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Bug][flink-runtime][connectors-v2] Flink register table Environment The running mode is set tojob.mode #4826

Merged
merged 13 commits into from
Aug 14, 2023

Conversation

zhilinli123
Copy link
Contributor

@zhilinli123 zhilinli123 commented May 25, 2023

issues: #4809

Purpose of this pull request

Check list

@zhilinli123
Copy link
Contributor Author

zhilinli123 commented May 25, 2023

hive Source:

create table  test3 (
  name string,
  age int,
  emails ARRAY<STRING>,
  phones map<STRING, string>
);
INSERT INTO TABLE test3 
VALUES ('list',12,array("[email protected]", "[email protected]", "[email protected]"), map('home', "13987167117", 'office', "15178968888"));

hive Sink:

create table  test3_sink (
  name string,
  age int,
  emails ARRAY<STRING>,
  phones map<STRING, string>
);

Conf :
env {
  # You can set spark configuration here
  execution.parallelism = 1
  job.mode = "BATCH"
}



source {
  Hive {
    table_name = "default.test3"
    metastore_uri = "thrift://localhost:9083"
    result_table_name = "user"
  }
}


transform {
  # If you would like to get more information about how to configure seatunnel and see full list of transform plugins,
  # please go to https://seatunnel.apache.org/docs/category/transform
  Replace {
    source_table_name = "user"
    result_table_name = "user_1"
    replace_field = "name"
    pattern = "s"
    replacement = "w"
    is_regex = true
    replace_first=true
  }
}

sink {
# Console{ source_table_name = "user" }
Hive {
    source_table_name = "user_1"
    table_name = "default.test3_sink"
    metastore_uri = "thrift://localhost:9083"
 }
}


table result test3_sink :
liwt    12      ["[email protected]","[email protected]","[email protected]"]   {"office":"15178968888","home":"13987167117"}

Class:org.apache.seatunnel.example.flink.v2.SeaTunnelApiExample

image

@zhilinli123
Copy link
Contributor Author

image Rerun CI?

@@ -123,8 +123,10 @@ public int currentUnassignedSplitSize() {

@Override
public void registerReader(int subtaskId) {
Set<Integer> readers = context.registeredReaders();
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change it?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why change it?

I just reconfirmed that there is no such problem, I have restored it

Comment on lines 74 to 78
boolean bounded =
internalSource.getBoundedness()
== org.apache.seatunnel.api.source.Boundedness.BOUNDED
? true
: false;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
boolean bounded =
internalSource.getBoundedness()
== org.apache.seatunnel.api.source.Boundedness.BOUNDED
? true
: false;
boolean bounded = internalSource.getBoundedness() == org.apache.seatunnel.api.source.Boundedness.BOUNDED

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done

Copy link
Contributor

@TaoZex TaoZex left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@zhilinli123
Copy link
Contributor Author

image Ci rerun

@zhilinli123 zhilinli123 reopened this Jun 16, 2023
@zhilinli123
Copy link
Contributor Author

PTAL: @TyrantLucifer @TaoZex

@ic4y
Copy link
Contributor

ic4y commented Aug 5, 2023

Waiting for the e2e issue to be fixed.

@zhilinli123
Copy link
Contributor Author

Waiting for the e2e issue to be fixed.

Done

@ic4y ic4y added this to the 2.3.3 milestone Aug 6, 2023
@zhilinli123
Copy link
Contributor Author

ci run @ic4y @liugddx

Copy link
Member

@EricJoy2048 EricJoy2048 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@ic4y
Copy link
Contributor

ic4y commented Aug 10, 2023

@TyrantLucifer PTAL

@ic4y ic4y merged commit 7825830 into apache:dev Aug 14, 2023
55 checks passed
ruanwenjun added a commit to ruanwenjun/incubator-seatunnel that referenced this pull request Nov 6, 2023
* [Feature][Connector V2] expose configurable options in Cassandra (apache#3681)

* [Connector-V2][Paimon] Introduce paimon connector (apache#4178)

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE (apache#4787)

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE

* [Improve][Zeta] Improve Zeta operation max count and ignore NPE

* [Improve][Zeta] Cancel pipeline add retry to avoid cancel failed. (apache#4792)

* [Hotfix][CDC] Fix chunk start/end parameter type error (apache#4777)

Incorrect wrapping as Array<Array> types, but only Array type required

* [Feature][Zeta] Add OSS support for Imap storage to cluster-mode type (apache#4683)

* Add OSS/S3 to cluster-mode type apache#4621

* fixed bug & add e2e test

* Wait for the node to start before scheduling  & Move jar to parent pom & optimize writer

* update LICENSE

* [Hotfix][CI] Fix error repository name in ci config files (apache#4795)

* [Feature][Json-format] support read format for pulsar (apache#4111)

* [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql par… (apache#4797)

* [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql parameter for the jdbc sinj document

* [Docs][Connector-V2][Mysql] fix Mysql sink format doc (apache#4800)

* [Hotfix][Connector][Jdbc] Fix sqlserver system table case sensitivity (apache#4806)

* [Hotfix][Connector][Jdbc] Fix reconnect throw close statement exception (apache#4801)

* [Hotfix][Connector-V2][Jdbc] Fix the error of extracting primary key column in sink (apache#4815)

* [Feature][Connector-v2] Add Snowflake Source&Sink connector (apache#4470)


---------

Co-authored-by: Eric <[email protected]>
Co-authored-by: hailin0 <[email protected]>

* [Hotfix][CI] Fix redundant modules run e2e tests when change jdbc module (apache#4824)

* fix pom.xml code style (apache#4836)

* [Chore] Format the.conf file using the same style (apache#4830)

* [Hotfix][Zeta] Fix cpu load problem (apache#4828)

* [Improve][Zeta] Reduce the number of IMAPs used by checkpointIdCounter (apache#4832)

* [Bugfix][connector-v2][rabbitmq] Fix reduplicate ack msg bug and code style (apache#4842)

---------

Co-authored-by: 毕博 <[email protected]>

* [Improve][Zeta] async execute checkpoint trigger and other block method (apache#4846)

* [Improve][Zeta] async execute checkpoint trigger

* [Bug][Zeta] Fix zeta cannot normally recycle thread belong to abnormal tasks

* [Improve][Zeta] Move `restoreState` add `addSplitsBack` execute by TaskExecuteService

* [Improve][Zeta] Move `receivedReader` execute by TaskExecuteService

* [Bug][Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or failed before run (apache#4847)

* [Bug][Zeta] Fix task repeat notify failed when job not running

* [Bug][Zeta] Fix notifyTaskStatusToMaster not release lock and NPE

* [Improve][Zeta] Reduce the frequency of fetching data from imap (apache#4851)

* [Improve][Zeta] Add Metaspace size default value to config file (apache#4848)

* [Improve][Zeta] Speed up listAllJob function (apache#4852)

* [Bug][Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (apache#4849)

* [Improve][Zeta] Fix engine runtime error (apache#4850)

* [Hotfix][Zeta] Fix completePendingCheckpoint concurrent action (apache#4854)

This operation does not allow concurrent execution

* [Hotfix][Zeta] Fix master active bug (apache#4855)

* [Bugfix][DAG] Fix the incorrect setting of transform parallelism (apache#4814)

* [Hotfix][Zeta] fix pipeline state not right bug (apache#4823)

* [BUG][Doris] Add a jobId to the doris label to distinguish between tasks (apache#4853)

Co-authored-by: zhouyao <[email protected]>

* [Improve] Add a jobId to the doris label to distinguish between tasks (apache#4839)

Co-authored-by: zhouyao <[email protected]>

* [Hotfix][Zeta] Fix IMap operation timeout bug (apache#4859)

* [Bug][Zeta] Fix restoreComplete Future can't be completed when cancel task (apache#4863)

* [Feature][SQL Transform]Add catalog support for SQL Transform plugin (apache#4819)

* [improve][SelectDB] Add a jobId to the selectDB label to distinguish between tasks (apache#4864)

Co-authored-by: zhouyao <[email protected]>

* [Hotfix][Connector-v2][kafka] Fix the short interval of pull data settings and revise the format (apache#4875)

* [Bug][Connector-V2][Doris] update last checkpoint id when doing snapshot (apache#4881)

* [Hotfix][Zeta] Fix deploy operation timeout but task already finished bug (apache#4867)

* [Core][Docs]Remove incubator in README file (apache#4882)

* [Bugfix][CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (apache#4877)

* [improve][CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (apache#4856)

* [Improve][Zeta] Reduce the operation count of imap_running_job_metrics (apache#4861)

* [Bug][Zeta] Fix TaskExecutionService will return not active ExecutionContext (apache#4869)

* [Hotfix][Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (apache#4866)

* [Bugfix] [Connector-V2] [File] Fix read temp file (apache#4876)

Co-authored-by: wantao <[email protected]>

* [Bug][Zeta] Fix TaskExecutionService synchronized lock will not release (apache#4886)

* [Improve][Zeta] Move driver into lib directory and change operation count (apache#4845)

* [hotfix][kafka] Fix the problem that the partition information cannot be obtained when kafka is restored (apache#4764)

* [Bugfix][zeta] Fix the deadlock issue with JDBC driver loading (apache#4878)

* [Chore] update 2.3.2 release-note.md (apache#4892)

* [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys  (apache#4634)

* [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column apache#4634

Co-authored-by: zhilinli <[email protected]>

* [Bug][connector-v2][doris] add streamload Content-type for doris URLdecode error (apache#4880)

* [Chore] Change repository name from incubator-seatunnel to seatunnel  (apache#4868)


---------

Co-authored-by: Jia Fan <[email protected]>

* [Improve][connector-V2-Neo4j]Supports neo4j sink batch write and update docs (apache#4841)

* [Hotfix][connector-v2][e2e] Fix maven scope (apache#4901)

* quick-start-seatunnel-engine.md (apache#4943)

* fix error (apache#4888)

* [Hotfix][Connector-V2][ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (apache#4937)

* Update ClickhouseFileSinkWriter.java

Bug fix: When ClikchouseFileSinkerWriter writes to a temporary file, it does not check whether the field value is empty, so an exception will be thrown.

Modified to write an empty string when a null value is encountered

* Update ClickhouseFileSinkWriter.java

repair code style

* Update ClickhouseFileSinkWriter.java

code style

* [Improve][Zeta] Add an interface for batch retrieval of JobMetrics (apache#4576)

* [Improve] Documentation and partial word optimization. (apache#4936)

* code format

* add cdc feature

* fix cdc can not get driver error

---------

Co-authored-by: gdliu3 <[email protected]>

* [Doc][Connector-V2] StarRocks `nodeUrls` property name fix (apache#4951)

node_urls -> nodeUrls
node_urls doesn't work

* [Feature][E2E][FtpFile] add ftp file e2e test case (apache#4647)

* [WIP][Feature][Connector-e2e] add ftp e2e test

* Let e2e barely execute by excluding the commons-net jar package.

* Resolve the maven conflict

---------

Co-authored-by: hailin0 <[email protected]>

* [Hotfix][Connector-V2][StarRocks] Fix code style (apache#4966)

* [Hotfix][Connector-v2][HbaseSink]Fix default timestamp (apache#4958)

* [Doc]Change the transform website url (apache#4954)

* [Docs][Connector-V2][Http]Reconstruct the Http connector document (apache#4962)

Co-authored-by: chenzy15 <[email protected]>

* [Feature][connector-v2][mongodb] mongodb support cdc sink (apache#4833)

* [Bug] [zeta][starter]fix bug (apache#4983) (apache#4984)

Co-authored-by: wsstony <[email protected]>

* fix redis nodes format error. (apache#4981)

Co-authored-by: lightzhao <[email protected]>

* [Improve][CDC]Remove  driver for cdc connector (apache#4952)

* [Hotfix][Connector-V2][Mongodb] Fix document error content and remove redundant code (apache#4982)

Co-authored-by: chenzy15 <[email protected]>

* [Improve][Connector-V2][OSS-Jindo] Optimize jindo oss connector (apache#4964)

* [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector

* [Improve][Connector-V2][Jindo-Oss] Update module name

* [Hotfix][Connector-V2][StarRocks] Fix code style

* [bugfix] Upgrade the key log output level(apache#4993)

* [Feature][Zeta] Configuration files support user variable replacement (apache#4969)

* [Feature][Transform-V2][SQL] Support 'select *' and 'like' clause for SQL Transform plugin (apache#4991)

Co-authored-by: mcy <[email protected]>

* [Improve][CDC]change driver scope to provider (apache#5002)

* [Hotfix][Connector-V2][Hive] Support user-defined hive-site.xml (apache#4965)

* [Improve][Connector-v2][Mongodb]Optimize reading logic (apache#5001)

Co-authored-by: chenqqq11 <[email protected]>

* [Feature][Connector-V2][Clickhouse] clickhouse writes with checkpoints (apache#4999)

* [Hotfix][Connector-V2][Mongodb] Compatible with historical parameters (apache#4997)

* Split updated modules integration test for part 4 (apache#5028)

* [Hotfix] Fix the CI Job name error (apache#5032)

* [Feature][CDC] Support disable/enable exactly once for INITIAL (apache#4921)

* [bugfix][zeta] Fixed multi-table job data loss and latency issues (apache#149) (apache#5031)

* [Hotfix][CDC] Fix jdbc connection leak for mysql (apache#5037)

* [Bugfix][zeta] Fix cdc connection does not close (apache#4922)

* Fix XA Transaction bug (apache#5020)

* Set Up with Kubernetes, dockerfile document error in constructing docker image (apache#5022)

Co-authored-by: yctan <[email protected]>

* [Improve][Connector-v2][Mongodb]sink support transaction update/writing (apache#5034)

* fix:the HdfsStorage can not delete checkpoint file apache#5046 (apache#5054)

* [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (apache#5033) (apache#5035)

* [bugfix] change MySQL CDC serverId from int to long (apache#5033)

* style: 🎨 optimize code style

* [Feature][Connector-V2][cdc] Change the time zone to the default time zone (apache#5030)

* [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse (apache#5011)

* [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (apache#4989)


---------

Co-authored-by: silenceland <[email protected]>
Co-authored-by: changhuyan <[email protected]>

* [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated apache#4985 (apache#5051)

* fix after the savepoint job is restored, the checkpoint file cannot be generated

* fix class not found exception (apache#5063)

* [Feature] update action config to support run CI on fork repo (apache#5065)

* [Bugfix]fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (apache#5080)

* [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone (apache#5078)

* Add clickhouse connector time zone key,default system time zone

* Modify the document and add clickhouse server_time_zone configuration

* [Chore] Modify repeat des (apache#5088)

Co-authored-by: 80597928 <[email protected]>

* [Docs] Add Value types in Java to Schema feature (apache#5087)

* [Feature][Connector-V2] JDBC source support string type as partition key (apache#4947)

* [HotFix] Fix code style (apache#5092)

* [Docs][Zeta] Add savepoint doc (apache#5081)

* [Feature][connector-v2][mongodbcdc]Support source mongodb cdc (apache#4923)

* [Improve] Improve savemode api (apache#4767)

* [Doc] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (apache#5102)

* [Improve][Docs][Clickhouse] Reconstruct the clickhouse connector doc (apache#5085)



---------

Co-authored-by: chenzy15 <[email protected]>

* [Pom]update version to 2.3.3-SNAPSHOT (apache#5043)

* update version to 2.3.3-SNAPSHOT

* update dependency version in know dependencies file

* Add logs to find job restore from master active switch error

* [Feature][Connector-V2][mysql cdc] Conversion of tinyint(1) to bool is supported (apache#5105)

Co-authored-by: zhouyao <[email protected]>

* [Improve][Zeta] Add sleep for Task to reduce CPU cost (apache#5117)

* [Feature][JDBC Sink] Add DM upsert support (apache#5073)



---------

Co-authored-by: David Zollo <[email protected]>

* [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak (apache#4802)

[Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak

* [Hotfix]Fix mongodb cdc e2e instability (apache#5128)

Co-authored-by: chenzy15 <[email protected]>

* [Hotfix][Zeta] Fix task state memory leak (apache#5139)

* [Hotfix][Zeta] Fix checkpoint error report without msg (apache#5137)

* [Improve][Zeta] Improve CheckpointCoordinator notify complete when restore (apache#5136)

* [Improve] Improve CheckpointCoordinator notify complete when restore

* update

* [Improve][Zeta] Improve CheckpointCoordinator log error when report error from task (apache#178) (apache#5134)

* [Hotfix][Zeta] Fix MultipleTableJobConfigParser ignore env option (apache#5067)

* [Fix][Zeta] Fix MultipleTableJobConfigParser ignore env option

* update

* [Improve][Connector[File] Optimize files commit order (apache#5045)

Before using `HashMap` store files path, so every checkpoint file commit is out of order.

Now switch to using `LinkedHashMap` to ensure that files are commit in the generated order

* [Hotfix][Mongodb cdc] Solve startup resume token is negative (apache#5143)


---------

Co-authored-by: chenzy15 <[email protected]>

* [Feature][connector][kafka] Support read debezium format message from kafka (apache#5066)

* [Feature][CDC] Support tables without primary keys (with unique keys) (apache#163) (apache#5150)

* [Feature][Connector-V2][CDC] Support string type shard fields. (apache#5147)

* [feature][CDC base] Supports string type shard fields

* Delete invalid code

* [Feature][Connector-V2][File] Add cos source&sink (apache#4979)

* [Feature][Connector-V2][File] Add cos sink

* update doc&e2e and add pom file header

* add e2e file header and config

* add file-cos module into dist pom.xml

* [Feature][Connector-V2][File] Add cos source

---------

Co-authored-by: dengd1937 <[email protected]>

* [Fix][Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (apache#5152)

* [Hotfix][MongodbCDC]Refine data format to adapt to universal logic (apache#5162)

Co-authored-by: chenzy15 <[email protected]>

* [Chore] Update bug-report.yml (apache#5160)

* [Improve][CDC] support exactly-once of cdc and fix the BinlogOffset comparing bug (apache#5057)

* [Improve][CDC] support exactly-once of cdc, fix the BinlogOffset comparing bug

* [Improve][CDC] adjust code style

* [Improve][CDC] fix ci error

---------

Co-authored-by: happyboy1024 <[email protected]>

* [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document (apache#4905)

* [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document


---------

Co-authored-by: zhouyao <[email protected]>

* [Docs][Connector-V2][Doris] Reconstruct the Doris connector document (apache#4903)

* [Docs][Connector-V2][Doris] Reconstruct the Doris connector document

---------

Co-authored-by: zhouyao <[email protected]>

* [improve] [CDC Base] Add some split parameters to the optionRule (apache#5161)

* [bugfix] [File Base] Fix Hadoop Kerberos authentication related issues. (apache#5171)

* [CI] add code style check when docs changed (apache#5183)

* [Bug][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type. (apache#5170)

* [Improve][Zeta] Move checkpoint notify complete in checkpoint stage (apache#5185)

* [Feature][Catalog] Add JDBC Catalog auto create table (apache#4917)

* [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. (apache#5153)

* [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files.

* [Improve][Connector-v2][Jdbc]  check url not null throw friendly message (apache#5097)

* check url not null throw friendly message

* check jdbc source config

* modify jdbc validate method

---------

Co-authored-by: 80597928 <[email protected]>
Co-authored-by: 80597928 <[email protected]>

* [bugfix][zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (apache#5191)

* [Improve][Docs][Kafka]Reconstruct the kafka connector document (apache#4778)

* [Docs][Connector-V2][Kafka]Reconstruct the kafka connector document

---------

Co-authored-by: chenzy15 <[email protected]>

* [Bug][Improve][LocalFileSink]Fix LocalFile Sink file_format_type. (apache#5118)

* [Bug] [connector-v2] PostgreSQL versions below 9.5 are compatible use cdc sync problem (apache#5120)

* [e2e] kafka e2e error (apache#5200)

* [Hotfix][Connector-V2][JindoOssFile] Fix plugin-mapping.properties (apache#5215)

Co-authored-by: tyrantlucifer <[email protected]>

* [Improve][Zeta] Don't trigger handleSaveMode when restore (apache#5192)

* move imap storage file dependency packages to submodules (apache#5218)

* [Hotfix][CI]Declare files that will always have UNIX line endings on checkout. (apache#5221)

* [Hotfix][Connector-V2][Paimon] Bump paimon-bundle version to 0.4.0-incubating (apache#5219)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL apache#4590 (apache#4757)

* [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL

* [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (apache#4793)

* [Docs] update seatunnel-flink.yaml and Dockerfile to help the demo work

* [Docs] update release-note apache#4788

---------

Co-authored-by: flynnxue <[email protected]>
Co-authored-by: ic4y <[email protected]>

* [feature][doris] Doris factory type (apache#5061)

* [feature][doris] Web need factory and data type convertor

* [Fix] Update the Readme (apache#4968)

Use the better description for the SeaTunnel Project

* [CI] Split updated modules integration test for part 5 (apache#5208)

* [CI] Split updated modules integration test for part 5

* [CI] Split updated modules integration test for part 5

* Split e2e

* update json-smart

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* fix dm error

* revert code

* revert code

* revert code

---------

Co-authored-by: gdliu3 <[email protected]>

* [Feature][CDC][Zeta] Support schema evolution framework(DDL) (apache#5125)

* Fixed IMap file storage e2e bug (apache#5237)

* [Improve] [Connector-V2] Remove scheduler in JDBC sink apache#4736 (apache#5168)


---------

Co-authored-by: gdliu3 <[email protected]>

* [Doc] [JDBC Oracle] Add JDBC Oracle Documentation (apache#5239)

* [Feature][Zeta][REST-API]Add REST API To Submit Job (apache#5107)

* [Fix] Update the project description (apache#4967)

* Update the project description

* [Feature][Zeta] Support history service record job execute error (apache#5114)

* fix:hdfs Checkpoint Storage management fails to delete historical files

* fix:hdfs Checkpoint Storage management fails to delete historical files

* fix after the savepoint job is restored, the checkpoint file cannot be generated

* [Feature][Zeta] Support history service record job execute error

* Improve Jobstate-related class additions add serialVersionUID

* add e2e test

* [hotfix]Update .asf.yaml (apache#5242)

* Update .asf.yaml

* [Hotfix]Fix array index anomalies caused by apache#5057 (apache#5195)

* [bugfix] [savepoint test] Turn on the testSavepoint test. (apache#5199)

* [BUG][Connector-V2][Jdbc] support postgresql json type  (apache#5194)

* add Postgresql json type
Co-authored-by: 80597928 <[email protected]>

* [Bugfix][cdc] Fix mysql bit column to java byte (apache#4817)

* [Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (apache#5146)

* [Docs][Connector][Source][jdbc]Change the line boundary store value type to BigDecimal (apache#4900)

* [bug][jdbc][oracle]Fix the Oracle number type mapping problem (apache#5209)

* [Bugfix][zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation. (apache#5206)

* [Hotfix][Zeta] Avoid Redundant Job Submissions by Checking Job Status (apache#5229)

* [Bugfix][zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore. (apache#5214)

* [Imporve] [CDC Base] Add a fast sampling method that supports character types (apache#5179)

* fixed zeta ci error (apache#5254)

* [Doc][README] Remove useless github workflow, and adjust description of 'engineering structure'. (apache#4305)

* [Feature][Zeta]The expiration time of a historical Job can be config (apache#5180)

* fix:hdfs Checkpoint Storage management fails to delete historical files
Co-authored-by: hailin0 <[email protected]>

* [bugfix] [e2e] Fixed a minor bug (apache#5274)

* [Improve][SQL] Support use catalogTableName as SQL expression (apache#5273)

* [Doc] Improve S3File Source & S3File Sink document (apache#5101)

* Improve S3File Source & S3File Sink document

* Fix style error (apache#5280)

* Fix StarRocksJsonSerializer will transform array/map/row to string (apache#5281)

* [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document (apache#5129)

* [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document

* fix format

* fix format

* [Improve][API & Zeta] Using connector custom serializer encode/decode states (apache#5238)

* API: Using DefaultSerializer as connector sink default serializer
* Zeta: Using connector custom serializer encode/decode states

* [Feature][Connector-V2] connector-kafka source support data conversion extracted by kafka connect source (apache#4516)

* Compatible kafka connect json apache#4137

* [Improve][CI/CD] Remove 'paths-ignore', enable the code style check for markdown files. (apache#5286)

* [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. (apache#5263)

* [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0.

* remove max-concurrent

* [Feature][Connector-v2][RedisSink]Support redis to set expiration time. (apache#4975)

* Support redis to set expiration time.

* Set redis expire default value.

* add e2e test.

* add e2e test.

* modify config file name.

---------

Co-authored-by: lightzhao <[email protected]>

* [bugfix] Fix testGetErrorInfo case error (apache#5282)

* [Feature][Zeta] Checkpoint support hdfs ha mode (apache#4942)

* fix browser long type intercept (apache#5267)

Co-authored-by: 80597928 <[email protected]>

* [Docs] remove `incubating` keyword in document (apache#5257)

* [feature][web] hive add option because web need (apache#5154)

* [feature][web] hive add option because web need

* [feature][web] hive add option read_columns

* [feature][web] required update optional

* [bugfix] mvn spotless

* fix conf

* fix conf

---------

Co-authored-by: liuli <[email protected]>

* [Bug][flink-runtime][connectors-v2] Flink register table Environment  The running mode is set to`job.mode` (apache#4826)

* [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document (apache#5132)

* [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document

* [Improve][Connector-v2][HiveSink]remove drop partition when abort. (apache#4940)

Co-authored-by: lightzhao <[email protected]>
Co-authored-by: liuli <[email protected]>
Co-authored-by: ic4y <[email protected]>

* [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document (apache#5130)

* [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document

* fix codestyle

---------

Co-authored-by: liuli <[email protected]>

* [Docs][Connector-V2][HDFS]Refactor connector-v2 docs using unified format HDFS. (apache#4871)

* Refactor connector-v2 docs using unified format HDFS.

* add data type.

* update.

* add key feature.

* add hdfs_site_path

* 1.add data type.
2.add hdfs_site_path conf.

* add data type.

* add hdfs site conf.

---------

Co-authored-by: lightzhao <[email protected]>
Co-authored-by: liuli <[email protected]>

* [Improve] [Connector-V2] Remove scheduler in Tablestore sink (apache#5272)


---------

Co-authored-by: gdliu3 <[email protected]>

* [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase (apache#5184)

* [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase

* [Hotfix] Fix com.google.common.base.Preconditions to seatunnel shade one (apache#5284)

* [Merge] Fix merge conflict and fix jdbc fieldIde with compatibleMode confusion

---------

Co-authored-by: Cason-ACE <[email protected]>
Co-authored-by: Tyrantlucifer <[email protected]>
Co-authored-by: hailin0 <[email protected]>
Co-authored-by: Xiaojian Sun <[email protected]>
Co-authored-by: Laglangyue <[email protected]>
Co-authored-by: ZhilinLi <[email protected]>
Co-authored-by: ic4y <[email protected]>
Co-authored-by: Hao Xu <[email protected]>
Co-authored-by: Eric <[email protected]>
Co-authored-by: Bibo <[email protected]>
Co-authored-by: 毕博 <[email protected]>
Co-authored-by: Carl-Zhou-CN <[email protected]>
Co-authored-by: zhouyao <[email protected]>
Co-authored-by: Marvin <[email protected]>
Co-authored-by: monster <[email protected]>
Co-authored-by: gnehil <[email protected]>
Co-authored-by: TaoZex <[email protected]>
Co-authored-by: xiaofan2012 <[email protected]>
Co-authored-by: wantao <[email protected]>
Co-authored-by: Guangdong Liu <[email protected]>
Co-authored-by: zhilinli <[email protected]>
Co-authored-by: zhaifengbing <[email protected]>
Co-authored-by: dalong <[email protected]>
Co-authored-by: FuYouJ <[email protected]>
Co-authored-by: davidfans <[email protected]>
Co-authored-by: Fan Donglai <[email protected]>
Co-authored-by: gdliu3 <[email protected]>
Co-authored-by: DismalSnail <[email protected]>
Co-authored-by: lightzhao <[email protected]>
Co-authored-by: chenzy15 <[email protected]>
Co-authored-by: wssmao <[email protected]>
Co-authored-by: wsstony <[email protected]>
Co-authored-by: lightzhao <[email protected]>
Co-authored-by: XiaoJiang521 <[email protected]>
Co-authored-by: mcy <[email protected]>
Co-authored-by: yctanGmail <[email protected]>
Co-authored-by: yctan <[email protected]>
Co-authored-by: wu-a-ge <[email protected]>
Co-authored-by: 司马琦昂 <[email protected]>
Co-authored-by: happyboy1024 <[email protected]>
Co-authored-by: He Wang <[email protected]>
Co-authored-by: silenceland <[email protected]>
Co-authored-by: changhuyan <[email protected]>
Co-authored-by: Jarvis <[email protected]>
Co-authored-by: 阿丙 <[email protected]>
Co-authored-by: jackyyyyyssss <[email protected]>
Co-authored-by: 80597928 <[email protected]>
Co-authored-by: Chengyu Yan <[email protected]>
Co-authored-by: zhangchengming601 <[email protected]>
Co-authored-by: lihjChina <[email protected]>
Co-authored-by: David Zollo <[email protected]>
Co-authored-by: EchoLee5 <[email protected]>
Co-authored-by: dengdi <[email protected]>
Co-authored-by: dengd1937 <[email protected]>
Co-authored-by: happyboy1024 <[email protected]>
Co-authored-by: FlechazoW <[email protected]>
Co-authored-by: 80597928 <[email protected]>
Co-authored-by: kun <[email protected]>
Co-authored-by: Volodymyr <[email protected]>
Co-authored-by: javalover123 <[email protected]>
Co-authored-by: Volodymyr <[email protected]>
Co-authored-by: kksxf <[email protected]>
Co-authored-by: flynnxue <[email protected]>
Co-authored-by: fang <[email protected]>
Co-authored-by: gejinxin <[email protected]>
Co-authored-by: Wenjun Ruan <[email protected]>
Co-authored-by: Koyfin <[email protected]>
Co-authored-by: liuli <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants