嗨，column based shuffle里schema是怎么获取到的能简单解释下么

[danielzed]

val schema = SparkUtils.getSchema(dep)
针对RssShuffle中这行代码的原理，能简单解释下么？dependency中应该是没有schema字段的，是对spark-core有什么修改么

[FMX]

Yes, there is a patch that changed the spark core.
https://github.com/apache/incubator-celeborn/blob/main/assets/spark-patch/RSS_Columnar_Shuffle_spark3.patch

[danielzed]

allright,thanks

[danielzed]

@FMX hi,ethan,do you have some column shuffle performance statistic to share? that would be great help.

[FMX]

Ping @kerwin-zk can you share some performance statistics?

[kerwin-zk]

@danielzed In the 2T TPC-DS scenario, you can reduce the shuffle size by about 35%, but there is about 5% overhead.

[danielzed]

@kerwin-zk
you mean even shuffle size is reduced by 35%, column-based-shuffle is worse than non-column-shuffle, about 5%?

[kerwin-zk]

Yes, mainly because the overhead of row-to-column and column-to-row is relatively large. We are also further optimizing this part of the overhead. Currently, with a larger amount of shuffle data, it can be better than non-column-shuffle.