Consider the following equation for gradient descent:
ww-nell. D. wl
whereg is the gradient function, I the loss function, and D, the dataset and η denotes the learning rate and the corresponding pseudo-code for the mini-batch variant:
1. for i = 1 to num_iter {
2 shuffle (data);
3. for batch in get_batches (data, batch_size) (
4. gradeval_gradient (loss_function, batch, w);
ww-learning_rate grad:
Identify the line numbers in this code that can be parallelized. Argue how the said lines can be parallelized using a distributed parameter server model. Will there be a difference between shared memory machine model and distributed memory machine model implementation for this. Justify your answer.