代码之家  ›  专栏  ›  技术社区  ›  LOST

如何将验证数据传递到Model.fit+数据集?

  •  0
  • LOST  · 技术社区  · 7 年前

    我正在努力训练一个简单的学生 Sequential

    为了提供输入,我使用 tf.data.Dataset 此处建议的API类似: https://stackoverflow.com/a/48134765/231238

    var train = Dataset.from_tensor_slices(ValueTuple.Create(trainInputs, trainOutputs));
    train = train
        .repeat(2000000)
        .shuffle(buffer_size: 1024 * 8 * InterpolateBy)
        .batch(1024);
    model.fit_dyn(train,
        epochs: 6*1024,
        steps_per_epoch: 4
        // line below does not work:
        , validation_data: (testInputs, testOutputs)
    );
    

    它在没有压力的情况下工作得很好 validation_data

    如果我通过 验证数据 作为张量的元组,如上例所示,例如。 (testInputs, testOutputs) ,它抛出 'TypeError : float() argument must be a string or a number, not 'NoneType' . (这也是我在切换到之前对列车数据所做的 Dataset ,并进行了验证)

    如果我包装 testInputs 和 testOutputs 数据集 与列车数据类似,例如:。 Dataset.from_tensor_slices(ValueTuple.Create(testInputs, testOutputs))

    ValueError : Error when checking input: expected sequential_input to have 2 dimensions, but got array with shape (347,)

    这里347是特征向量的大小,因此 testInputs.shape is(221347)和 testOutputs.shape

    1 回复  |  直到 7 年前
        1
  •  1
  •   kvish    7 年前

    从我们的讨论中,我们可以澄清一些事情。

    至于tf数据的输入验证,当我们使用 from_tensor_slices

    如果我们使用 from_tensors 方法,它创建一个具有单个元素的数据集,该元素由给定的张量作为输入组成。因此,它相当于直接通过dataset对象提供numpy数据。

    下面是一个简单的例子,说明了如何在更小的维度上实现这一点:

    import numpy as np
    import tensorflow as tf
    np.random.seed(42)
    example_train = np.random.randn(4, 4)
    example_test = np.random.randn(4, 1)
    
    print("Example Train:", example_train)
    print("Example Test:", example_test)
    
    dataset1 = tf.data.Dataset.from_tensor_slices((example_train, example_test))
    dataset2 = tf.data.Dataset.from_tensors((example_train, example_test))
    
    it1 = dataset1.make_one_shot_iterator().get_next()
    it2 = dataset2.make_one_shot_iterator().get_next()
    
    with tf.Session() as sess:
        for i in range(4):
            print("Element {} of dataset1: {}".format(i,sess.run([it1])))
        print ("Element 0 of dataset2: ", sess.run([it2]))
    

    Example Train: [[ 0.49671415 -0.1382643   0.64768854  1.52302986]
     [-0.23415337 -0.23413696  1.57921282  0.76743473]
     [-0.46947439  0.54256004 -0.46341769 -0.46572975]
     [ 0.24196227 -1.91328024 -1.72491783 -0.56228753]]
    Example Test: [[-1.01283112]
     [ 0.31424733]
     [-0.90802408]
     [-1.4123037 ]]
    Element 0 of dataset1: [(array([ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986]), array([-1.01283112]))]
    Element 1 of dataset1: [(array([-0.23415337, -0.23413696,  1.57921282,  0.76743473]), array([0.31424733]))]
    Element 2 of dataset1: [(array([-0.46947439,  0.54256004, -0.46341769, -0.46572975]), array([-0.90802408]))]
    Element 3 of dataset1: [(array([ 0.24196227, -1.91328024, -1.72491783, -0.56228753]), array([-1.4123037]))]
    Element 0 of dataset2:  [(array([[ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986],
           [-0.23415337, -0.23413696,  1.57921282,  0.76743473],
           [-0.46947439,  0.54256004, -0.46341769, -0.46572975],
           [ 0.24196227, -1.91328024, -1.72491783, -0.56228753]]), array([[-1.01283112],
           [ 0.31424733],
           [-0.90802408],
           [-1.4123037 ]]))]
    

    关于我对 batch 方法,通过将批处理大小设置为221,以便将内容重新组合在一起,如果我们将dataset1代码更改为类似的内容,并将打印修改为类似的内容,例如:

    dataset1 = tf.data.Dataset.from_tensor_slices((example_train, example_test)).batch(4)
    
    with tf.Session() as sess:
        print ("Element 0 of dataset1: ", sess.run([it1]))    
    

    Element 0 of dataset1:  [(array([[ 0.49671415, -0.1382643 ,  0.64768854,  1.52302986],
           [-0.23415337, -0.23413696,  1.57921282,  0.76743473],
           [-0.46947439,  0.54256004, -0.46341769, -0.46572975],
           [ 0.24196227, -1.91328024, -1.72491783, -0.56228753]]), array([[-1.01283112],
           [ 0.31424733],
           [-0.90802408],
           [-1.4123037 ]]))]