Spark源码分析之分区器的作用

of items, we re-sample from it

// to ensure that enough items are collected from that partition.

// 如不雅有的分区包含的数量远跨越平均值，那么须要对它从新采样。每个分区的采样数/采样返回的总的记录数

val fraction = math.min(sampleSize / math.max(numItems, 1L), 1.0)

//保存有效的采样数

val candidates = ArrayBuffer.empty[(K, Float)]

//保存数据倾斜导致的采样数过多的信息

val imbalancedPartitions = mutable.Set.empty[Int]

sketched.foreach { case (idx, n, sample) =>

if (fraction * n > sampleSizePerPartition) {

imbalancedPartitions += idx

} else {

// The weight is 1 over the sampling probability.

val weight = (n.toDouble / sample.size).toFloat

for (key <- sample) {

candidates += ((key, weight))

}

if (imbalancedPartitions.nonEmpty) {

// Re-sample imbalanced partitions with the desired sampling probability.

val imbalanced = new PartitionPruningRDD(rdd.map(_._1), imbalancedPartitions.contains)

val seed = byteswap32(-rdd.id - 1)

//基于RDD获取采样数据

val reSampled = imbalanced.sample(withWordStrment = false, fraction, seed).collect()

val weight = (1.0 / fraction).toFloat

candidates ++= reSampled.map(x => (x, weight))

}

　　推荐阅读

　　SDN能解决很多问题，但不包括安全

跟着数字化企业尽力寻求最佳安然解决筹划来保护其赓续扩大的收集，很多企颐魅正在寻求供给互操作性功能的下一代对象。软件定义收集(SDN)具有很多的优势，经由过程将多个设备的┞菲握平面整>>>详细阅读

地址：http://www.17bianji.com/lsqh/34919.html