代码之家 › 专栏 › 技术社区 › Marsellus Wallace

你选错功能了吗?

chi-squared feature-selection apache-spark-mllib apache-spark scala

0

Marsellus Wallace · 技术社区 · 7 年前

我复制粘贴 this example 从我办公室的文件里 Spark 2.3.0 壳

import org.apache.spark.ml.feature.ChiSqSelector
import org.apache.spark.ml.linalg.Vectors

val data = Seq(
  (7, Vectors.dense(0.0, 0.0, 18.0, 1.0), 1.0),
  (8, Vectors.dense(0.0, 1.0, 12.0, 0.0), 0.0),
  (9, Vectors.dense(1.0, 0.0, 15.0, 0.1), 0.0)
)

val df = spark.createDataset(data).toDF("id", "features", "clicked")

val selector = new ChiSqSelector()
  .setNumTopFeatures(1)
  .setFeaturesCol("features")
  .setLabelCol("clicked")
  .setOutputCol("selectedFeatures")

val selectorModel = selector.fit(df)
val result = selectorModel.transform(df)
result.show
+---+------------------+-------+----------------+
| id|          features|clicked|selectedFeatures|
+---+------------------+-------+----------------+
|  7|[0.0,0.0,18.0,1.0]|    1.0|          [18.0]|
|  8|[0.0,1.0,12.0,0.0]|    0.0|          [12.0]|
|  9|[1.0,0.0,15.0,0.1]|    0.0|          [15.0]|
+---+------------------+-------+----------------+

selectorModel.selectedFeatures
res2: Array[Int] = Array(2)

ChiSqSelector 误采 feature 2 而不是 feature 3 (根据文档和常识,功能3应该是正确的)

0 回复 | 直到 7 年前

1

10465355 user11020637 7 年前

卡方特征选择 operates on categorical data

ChiSqSelector 代表卡方特征选择。它对带有分类特征的标记数据进行操作

因此,这两个特征都是同样好的(尽管我们应该强调,这两个特征即使用作连续变量,也可以用来派生微不足道的完美分类器)。

import org.apache.spark.mllib.linalg.{Vectors => OldVectors}
import org.apache.spark.mllib.regression.LabeledPoint
import org.apache.spark.mllib.stat.Statistics

Statistics.chiSqTest(sc.parallelize(data.map { 
  case (_, v, l) => LabeledPoint(l, OldVectors.fromML(v)) 
})).slice(2, 4)

Array[org.apache.spark.mllib.stat.test.ChiSqTestResult] =
Array(Chi squared test summary:
method: pearson
degrees of freedom = 2
statistic = 3.0
pValue = 0.22313016014843035
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent.., Chi squared test summary:
method: pearson
degrees of freedom = 2
statistic = 3.0000000000000004
pValue = 0.22313016014843035
No presumption against null hypothesis: the occurrence of the outcomes is statistically independent..)

测试结果与其他工具一致。例如在R( used as a reference for selector tests ):

y <- as.factor(c("1.0", "0.0", "0.0"))
x2 <- as.factor(c("18.0", "12.0", "15.0"))
x3 <- as.factor(c("1.0", "0.0", "0.1"))

chisq.test(table(x2, y))

    Pearson's Chi-squared test

data:  table(x2, y)
X-squared = 3, df = 2, p-value = 0.2231

Warning message:
In chisq.test(table(x2, y)) : Chi-squared approximation may be incorrect

chisq.test(table(x3, y))

    Pearson's Chi-squared test

data:  table(x3, y)
X-squared = 3, df = 2, p-value = 0.2231

Warning message:
In chisq.test(table(x3, y)) : Chi-squared approximation may be incorrect

自选择器 just sorts data by p-value 和 sortBy is stable ,先到先得。如果切换功能的顺序,将选择另一个。