代码之家 › 专栏 › 技术社区 › Nayantara Jeyaraj

相同形式的数据集有两个不同的形状

training-data classification machine-learning python

Nayantara Jeyaraj · 技术社区 · 7 年前

我对机器学习很陌生,只是掌握技术而已。因此,我尝试使用具有4个特性的数据集和目标特性/类(真值)对以下分类器上的模型进行培训一或零 )

分类器

SGD分类器
随机森林分类器
线性支持向量分类器
高斯过程分类器

我正在对以下数据集的模型进行培训[数据集的一部分如下所示]。

培训集:train_sop_truth.csv

Subject,Predicate,Object,Computed,Truth
concept:sportsteam:hawks,concept:teamplaysincity,concept:city:atlanta,0.4255912602,1
concept:stadiumoreventvenue:honda+AF8-center,concept:stadiumlocatedincity,concept:city:anaheim,0.4276425838,1
concept:sportsteam:ducks,concept:teamplaysincity,concept:city:anaheim,0.4762486517,1
concept:sportsteam:n1985+AF8-chicago+AF8-bears,concept:teamplaysincity,concept:city:chicago,0.4106097221,1
concept:stadiumoreventvenue:philips+AF8-arena,concept:stadiumlocatedincity,concept:city:atlanta,0.4190083146,1
concept:stadiumoreventvenue:united+AF8-center,concept:stadiumlocatedincity,concept:city:chicago,0.4211134315,1

测试数据集位于另一个.csv文件中,作为 test_sop_truth.csv .

测试集:test_sop_truth.csv

Subject,Predicate,Object,Computed,Truth
Nigel_Cole,isMarriedTo,Kate_Isitt,0.9350595474,1
VÃ©ra_Clouzot,isMarriedTo,Henri-Georges_Clouzot,0.4773990512,1
Norodom_Sihanouk,produced,The_Last_Days_of_Colonel_Savath,0.3942225575,1
Farouk_of_Egypt,isMarriedTo,Farida_of_Egypt,0.4276426733,1

然后我想检查每个特性的形状,并期望看到相同数量的特性,就像我对两个数据集应用相同的转换一样。但它们不同。

蟒蛇代码

import pandas as pd
import numpy as np
from termcolor import colored

features = pd.read_csv('../Data/train_sop_truth.csv')
testFeatures = pd.read_csv('../Data/test_sop_truth.csv')
print(features.head(5))

print(colored('\nThe shape of our features is:','green'), features.shape)
print(colored('\nThe shape of our Test features is:','green'), testFeatures.shape)

print()
print(colored('\n     DESCRIPTIVE STATISTICS\n','yellow'))
print(colored(features.describe(),'cyan'))
print()
print(colored(testFeatures.describe(),'cyan'))


features = pd.get_dummies(features)
testFeatures = pd.get_dummies(testFeatures)

features.iloc[:,5:].head(5)
testFeatures.iloc[:,5].head(5)

labels = np.array(features['Truth'])
testlabels = np.array(testFeatures['Truth'])


features= features.drop('Truth', axis = 1)
testFeatures = testFeatures.drop('Truth', axis = 1)

feature_list = list(features.columns)
testFeature_list = list(testFeatures.columns)

features = np.array(features)
testFeatures = np.array(testFeatures)

train_samples = 100


testX_train, textX_test, testy_train, testy_test = model_selection.train_test_split(testFeatures, testlabels, test_size=0.25, random_state = 42)

X_train, X_test, y_train, y_test = model_selection.train_test_split(features, labels, test_size = 0.25, random_state = 42)

print(colored('\n    TRAINING & TESTING SETS','yellow'))
print(colored('\nTraining Features Shape:','magenta'), X_train.shape)
print(colored('Training Labels Shape:','magenta'), X_test.shape)
print(colored('Testing Features Shape:','magenta'), y_train.shape)
print(colored('Testing Labels Shape:','magenta'), y_test.shape)

print()

print(colored('\n    TRAINING & TESTING SETS','yellow'))
print(colored('\nTraining Features Shape:','magenta'), testX_train.shape)
print(colored('Training Labels Shape:','magenta'), textX_test.shape)
print(colored('Testing Features Shape:','magenta'), testy_train.shape)
print(colored('Testing Labels Shape:','magenta'), testy_test.shape)

产量

The shape of our features is: (1860, 5)

The shape of our Test features is: (1386, 5)


     DESCRIPTIVE STATISTICS

          Computed        Truth
count  1860.000000  1860.000000
mean      0.443222     0.913441
std       0.110788     0.281264
min       0.000000     0.000000
25%       0.418164     1.000000
50%       0.427643     1.000000
75%       0.450023     1.000000
max       1.000000     1.000000

          Computed        Truth
count  1386.000000  1386.000000
mean      0.511809     0.992063
std       0.197954     0.088765
min       0.009042     0.000000
25%       0.418649     1.000000
50%       0.429140     1.000000
75%       0.515809     1.000000
max       1.702856     1.000000

    TRAINING & TESTING SETS

Training Features Shape: (1395, 1045)
Training Labels Shape: (465, 1045)
Testing Features Shape: (1395,)
Testing Labels Shape: (465,)


    TRAINING & TESTING SETS

Training Features Shape: (1039, 1790)
Training Labels Shape: (347, 1790)
Testing Features Shape: (1039,)
Testing Labels Shape: (347,)

这里我不明白的是,特征形状是如何不同的 1045 对于功能(训练集)和 1790 对于测试特性(测试集),尽管在csv文件中进行了相同的转换,并且具有相同数量的特性和特性形式。

在此方面的任何建议或澄清将非常感谢。

1 回复 | 直到 7 年前

Venkatachalam 7 年前

当您为测试数据集应用get-dummies时,可能会根据分类变量的数据值添加或删除一些列。

def add_missing_dummy_columns( d, columns ):
        missing_cols = set( columns ) - set( d.columns )
        for c in missing_cols:
            d[c] = 0

def fix_columns( d, columns ):  

    add_missing_dummy_columns( d, columns )

    # make sure we have all the columns we need
    assert( set( columns ) - set( d.columns ) == set())

    extra_cols = set( d.columns ) - set( columns )
    if extra_cols: print ("extra columns:", extra_cols)

    d = d[ columns ]
    return d

testFeatures= fix_columns( testFeatures, features.columns )