Skip to content

Spark shuffle调优 map端合并输出文件

Jump to bottom
liaotian1005 edited this page Nov 21, 2018 · 2 revisions

new SparkConf.set("spark.shuffle.consolidateFiles","true")

map缓存大小 默认32k

reducer内存占比 默认0.2
spark.shuffle.file.buffer
spark.shuffle.memoryFraction 0.2

Clone this wiki locally