Consider a simple feedforward neural language model based on the following simple 2-layer network. Assume we're using a context window of 1 (unigram). That is, we're trying to predict the next word solely from the current one, x. h = Wx z = Uh y = softmax(z) Assume we have a vocabulary of size 30,000, hidden layer size 500, and word embeddings of size 300. What are the dimensions of the W and U weight matrices. W is ________ x ________ U is ________ x ________
Added by Cindy W.
Close
Step 1
The input word x is represented by a word embedding vector of size 300. Show more…
Show all steps
Your feedback will help us improve your experience
Labs
Want to see this concept in action?
Explore this concept interactively to see how it behaves as you change inputs.
Key Concepts
Recommended Videos
Recurrent neural networks (RNN) can be used as classification models for time series data. Here we have a simple RNN as shown in the figure above, where st = f1(Ws, sst−1 + Ws, xx t), t = 1, 2,..., T and y = f2(Ws, ysT + W0). We assume all offsets are 0 except W0 for the final output layer and we decide the two activation functions to be: f1(z) = RELU(z) = max(0, z) and f2(z) = sign(z) = {1, 0, if z ≥ 0 if z < 0. Note that the RELU(z) can be applied elementwise if z is a vector. Suppose we want to apply this model to classify sentences into different categories (e.g. positive/negative sentiment), we need to encode each word in a sentence into a vector as the input x t to the model. One way to do this is to represent the tth word as a column vector of length |V|, where V is the set of the entire vocabulary. The ith element of x t is 1 if the word is the ith word in the vocabulary and all other elements are zero. We first explore a simple scenario where our vocabulary contains only 2 words, V = {A, B}. Let s t ∈ R2 and we set the initial state s0 and the weights before the last layer as follows: s0 = [0 0], Ws,s = [−1 0; 0 1], Ws,x = [1 0; 0 1]. Now given 3 training sentences: AA, ABB, BAA. Encode each of them into a sequence of vectors. As an example, the sentence AA is encoded as x(1) = (x(1)1, x(1)2), where x(1)1 = x(1)2 = [1 0]T. (To enter the sequence above, type [[1,0],[1,0]].) Now encode the other 2 sentences into x(2) and x(3).
Madhur L.
Neural Network. Consider a three-layer neural network to learn a function f : X → Y, where X = [X1, X2] consists of two features. The weights w1,...,w6 can be arbitrary. There are two possible choices for the function implemented by each unit in this network: - S: sigmoid function, S(z) = 1/(1+exp(-z)) - L: linear function, L(z) = cz where in both cases z = Σi wi Xi. Assign proper activation functions (S or L) to each unit in the following graph so that we can generate functions of the form f(X1, X2) = 1/(1+exp(β1 X1 + β2 X2)) at the output of the neural network Y. Derive β1 and β2 as a function of w1,...,w6.
Shaiju T.
Part Two: Implement Perceptron [Graded] Implement function perceptron. This should contain a loop that calls perceptron update until it converges or the maximum iteration count, 100, has been reached. Make sure you randomize the order of the training data on each iteration (you can use np.random.permutation() to do this.) def perceptron(xs, ys): """ function w = perceptron(xs, ys); Implementation of a Perceptron classifier Input: xs - n input vectors of d dimensions (nxd) ys - n labels (-1 or +1) Output: w - weight vector (1xd) b - bias term """ n, d = xs.shape w = np.zeros(d) b = 0.0 # so we have n input vectors, of d dimensions each # YOUR CODE HERE
Akash M.
Recommended Textbooks
Computer Science and Information Technology
Introduction to Programming Using Python
Computer Science - An Overview
Transcript
Watch the video solution with this free unlock.
EMAIL
PASSWORD