代码之家 › 专栏 › 技术社区 › fpghost

tensorflow数据集中窗口的规范化

tensorflow

0

fpghost · 技术社区 · 6 年前

我试图从一元时间序列构建一个窗口化的数据集。如果这个系列看起来像 [1, 2, 3, 4, 5, 6] 窗户的长度是2 [[1, 2, 3], [2, 3, 4], [3, 4, 5], [4, 5, 6]] 然后我将对它们进行无序处理,以避免产生偏差,并从每个窗口的目标输出中分离出输入特性: [[[1, 2], [3]], [[2, 3], [4]], [[3, 4], [5]], [[4, 5], [6]]]

def windowed_dataset(series):
    # Initially the data is (N,) expand dims to (N, 1)
    series = tf.expand_dims(series, axis=-1)

    # Tensorflow Dataset from the array
    ds = tf.data.Dataset.from_tensor_slices(series)

    # Create the windows that will serve as input features and label (hence +1)
    ds = ds.window(window_len + 1, shift=1, drop_remainder=True)
    ds = ds.flat_map(lambda w: w.batch(window_len + 1))

    # randomize order 
    ds = ds.shuffle(shuffle_buffer)
    # Separate  the inputs and the target output(label)
    ds = ds.map(lambda w: (w[:-1], w[-1]))
    return ds.batch(batch_size).prefetch(1)

不过,我想补充一些规范化。例如,如果我的窗口是 w=[1, 2, 3] [p/w[0] - 1 for p in w]

我想我可以用 ds.map 和

    def normalize_window(w):
      return [((i/w[0]) -1) for i in w]


    ds = ds.map(normalize_window)

map 对于lambda函数,但我认为它也适用于正则函数

有人知道该怎么做吗?

编辑

我得到的回溯是

<ipython-input-39-929295e1b775> in <module>()
----> 1 dataset = model_forecast_datasets(btc_model, np_data[:6])

11 frames
/usr/local/lib/python3.6/dist-packages/tensorflow/python/autograph/impl/api.py in wrapper(*args, **kwargs)
    263       except Exception as e:  # pylint:disable=broad-except
    264         if hasattr(e, 'ag_error_metadata'):
--> 265           raise e.ag_error_metadata.to_exception(e)
    266         else:
    267           raise

OperatorNotAllowedInGraphError: in user code:

    <ipython-input-38-b3d0f7e17689>:12 normalize_window  *
        return [(i/w[0] -1) for i in w]
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:561 __iter__
        self._disallow_iteration()
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:557 _disallow_iteration
        self._disallow_in_graph_mode("iterating over `tf.Tensor`")
    /usr/local/lib/python3.6/dist-packages/tensorflow/python/framework/ops.py:537 _disallow_in_graph_mode
        " this function with @tf.function.".format(task))

    OperatorNotAllowedInGraphError: iterating over `tf.Tensor` is not allowed in Graph execution. Use Eager execution or decorate this function with @tf.function.

1 回复 | 直到 6 年前

1

0

prouast 6 年前

你需要一个向量化计算的函数,比如

def normalize(data):
    mean = tf.math.reduce_mean(data)
    std = tf.math.reduce_std(data)
    data = tf.subtract(data, mean)
    data = tf.divide(data, std)
    return data

ds = ds.map(normalize)

编辑:对于您的特定规范化,这可能有用:

def normalize(data):
    data1 = tf.subtract(data, tf.constant(1))
    data1 = tf.divide(data1, data[0])
    return data1

ds = ds.flat_map(...)