Skip to content

Spark Doris Connector Release Note 25.1.0 #319

@JNSimba

Description

@JNSimba

Spark Connector

Improvement

  1. support overwrite for datasource v2 ([feature](connector) support overwrite for datasource v2 #281)
  2. support read DECIMAL128 type ([fix](connector) support read DECIMAL128 type #290)
  3. compatible with read ip data type ([read](data-type) compatible with ip data type #289)
  4. improve be thrift read log and stream load log ([improve](connector) add be thrift read log #279) ([improve] improve stream load log  #313)
  5. add doris source sink itcase and spark 3.2/3.4/3.5 ci ([Improve](case) add source it case #296) ([improve] add doris sink itcase #297) ([Improve](test) add structured streaming case #302) ([improve] add case and ci for spark 3.2,3.4,3.5 #312)
  6. enhance the FE nodes port check and improve write performace (Enhancement] enhance the FE nodes port check and improve write performace  #303)
  7. add retry interval when load fail ([improvement] add retry interval when load fail #307)
  8. add code check for fe request (add code check for fe request #309)
  9. remove unused code after refractor ([refactor](connnector) remove unused code after refractor #295)

BugFix

  1. fix filter in clause compilation ([fix](connector) fix filter in clause compilation #277)
  2. fix ut for row convertor and schema convertor ([fix](connector) fix ut for row convertor and schema convertor #278)
  3. fix npe when empty partition commit txn ([fix](connector) fix npe when empty partition commit txn #286)
  4. fix prune column result isn't pushed down issue ([fix](connector) fix prune column result isn't pushed down issue #287)
  5. fix performance degradation caused by interval mistake ([fix](connector) fix performance degradation caused by interval mistake #294)
  6. fix the issue when Spark pushes down in 'case when' ([fix](connector)Fix the issue where 'case when' does not correspond to an error when the Spark predicate is pushed down(#282) #300)
  7. fixed the issue where the loading task got stuck when stream load ended unexpectedly ([fix](connector) Fixed the issue where the loading task got stuck when stream load ended unexpectedly  #305)
  8. fixed the problem that the query could not be performed when pushing utf8 encoding ([Fix] Fixed the problem that the query could not be performed when pushing utf8 encoding #311)
  9. fix the conflict for rename arrowwriter and arrowutil ([Fix] rename arrowwriter and arrowutil #315)

Spark Load

Improvement

  1. support load hadoop conf from resource ([improve](spark load) support load hadoop conf from resource #280)

Thanks

@aoyuEra
@gnehil
@JNSimba
@IsmailTosunTnyl
@qg-lin
@vinlee19
@wary

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions