Spark SQL Optimization

Reference

AQE

spark.sql.adaptive.allowBroadcastExchange.enabled	true
spark.sql.adaptive.coalescePartitionsSupportUnion.enabled	true
spark.sql.adaptive.enabled	true
spark.sql.adaptive.forceOptimizeSkewedJoin	true
spark.sql.adaptive.join.enabled	true
spark.sql.adaptive.maxNumPostShufflePartitions	500
spark.sql.adaptive.minNumPostShufflePartitions	5
spark.sql.adaptive.multipleSkewedJoin.enabled	true
spark.sql.adaptive.multipleSkewedJoinWithAggOrWin.enabled	true
spark.sql.adaptive.shuffle.targetPostShuffleInputSize	67108864
spark.sql.adaptive.shuffle.targetPostShuffleRowCount	20000000
spark.sql.adaptive.shuffleHashJoin.enabled	true
spark.sql.adaptive.shuffleHashJoin.ignoreStatsAccuracy	true
spark.sql.adaptive.shuffleHashJoin.singleShuffleInput	true
spark.sql.adaptive.skewShuffleHashJoin.enabled	true
spark.sql.adaptive.skewedJoin.enabled	true
spark.sql.adaptive.skewedJoinSupportUnion.enabled	true
spark.sql.adaptive.skewedJoinWithAgg.enabled	true
spark.sql.adaptive.skewedPartitionFactor	3
spark.sql.adaptive.skewedPartitionRowCountThreshold	10000000
spark.sql.adaptive.skewedPartitionSizeThreshold	67108864
spark.sql.adaptiveBroadcastJoinThreshold	-1

qin

取消

感谢您的支持,我会继续努力的!

扫码支持
扫码支持
扫码打赏

打开支付宝扫一扫,即可进行扫码打赏哦