并行张量流平均小批量梯度

作为一个简单的示例,以该脚本为例,该脚本在抛物线y=x^2的N个数据点上训练神经网络:

import tensorflow as tf
import numpy as np

def add_layer(inputs, in_size, out_size, activation_function=None):
    Weights = tf.Variable(tf.random_normal([in_size, out_size]))
    biases = tf.Variable(tf.random_normal([1, out_size]))
    Wx_plus_b = tf.matmul(inputs, Weights) + biases
    if activation_function is None:
        outputs = Wx_plus_b
    else:
        outputs = activation_function(Wx_plus_b)
    return outputs

# Make up some real data
N = 50
x_data = np.linspace(-2, 2, N)[:, np.newaxis]
noise = np.random.normal(0, 0.05, x_data.shape)
y_data = np.square(x_data) # - 0.5 + noise

# Define placeholder for x_data and y_data
xs = tf.placeholder(tf.float32, [None, 1])
ys = tf.placeholder(tf.float32, [None, 1])

""" Build the network"""
# Add hidden layer
l1 = add_layer(xs, 1, 5, activation_function=tf.tanh)
# Add output layer
prediction = add_layer(l1, 5, 1, activation_function=None)

# Define loss
loss = tf.reduce_mean(tf.reduce_sum(tf.square(ys-prediction), reduction_indices=[1]))

# Define optimizer
opt = tf.train.GradientDescentOptimizer(learning_rate=1e-2)
# Compute the gradients for a list of variables.
grads_and_vars = opt.compute_gradients(loss)
# Ask the optimizer to apply the gradients
train_opt = opt.apply_gradients(grads_and_vars)

# Initialize global variables
sess = tf.Session()
init = tf.global_variables_initializer()
sess.run(init)

for i in range(2000):
    # training
    sess.run(train_opt, feed_dict={xs: x_data, ys: y_data})
    if i % 50 == 0:
        prediction_value = sess.run(prediction, feed_dict={xs: x_data})
        print(sess.run(loss, feed_dict={xs: x_data, ys: y_data}))

我想并行化的部分是梯度的计算,然后我想把这些梯度带回到主过程进行平均,然后应用到训练步骤。我想将N个数据点拆分为 x_data

我认为这就是所谓的“同步训练”,我看过相关的参考资料,但没有人解释过。