CISC 7412X Midterm Exam

1. given: P(O) = 0.1, P(-O) = 0.9
          P(C|O) = 0.9, P(-C|O) = 0.1, P(C|-O) = 0.2, P(-C|-O) = 0.8
          P(I|O) = 0.8, P(-I|O) = 0.2, P(I|-O) = 0.05, P(-I|-O) = 0.95

   a) P(O|C) = P(C|O)P(O) / P(C)
             = P(C|O)P(O) / (P(C|O)P(O) + P(C|-O)P(-O))
             = (0.9 * 0.1) / (0.9 *0.1 + 0.2*0.9)
             = 0.3333

   a) P(O|I) = P(I|O)P(O) / P(I)
             = P(I|O)P(O) / (P(I|O)P(O) + P(I|-O)P(-O))
             = (0.8*0.1) / (0.8*0.1 + 0.05 * 0.9)
             = 0.6400

2. assume P(C,I|O) = P(C|O)P(I|O)
   a) P(O|C,I) = P(C,I|O)P(O) / P(C,I)
               = P(C|O)P(I|O)P(O) / (P(C|O)P(I|O)P(O) + P(C|-O)P(I|-O)P(-O))
               = (0.9*0.8*0.1) / (0.9*0.8*0.1 + 0.2*0.05*0.9)
               = 0.8889

   b) P(O|C,-I) = P(C,-I|O)P(O) / P(C,-I)
                = P(C|O)P(-I|O)P(O) / (P(C|O)P(-I|O)P(O) + P(C|-O)P(-I|-O)P(-O))
                = (0.9*0.2*0.1) / (0.9*0.2*0.1 + 0.2*0.95*0.9)
                = 0.095238

3. we are given: P(C,I|O) = 0.75, P(C,I|-O) = 0.04
    a) P(O|C,I) = P(C,I|O)P(O) / P(C,I)
                = P(C,I|O)P(O) / (P(C,I|O)P(O) + P(C,I|-O)P(-O))
                = (0.75*0.1) / (0.75*0.1 + 0.04*0.9)
                = 0.6757
    b) q2(a) = 0.89 is larger than q3(a) = 0.67
      The two sensors are not independent---meaning that when we look
      at one sensor there is positive correlation with the other 
      sensor. By treating them as independent, we are counting the same
      evidense more than once.

4. 
  a) z1 = 1*2 + (-1)*1 + 0.5 = 1.5; h1 = ReLU(1.5) = 1.5
     z2 = (-1)*2 + 1*1 + 0   = -1; h2 = ReLU(-1)  = 0

  b) y = 2*1.5 + 3*0 - 1 = 3 + 0 - 1 = 2

  c) L = 0.5*(2 - 1)^2 = 0.5

5. 
  a) z = 0.5*2 = 1  
     h = ReLU(1) = 1
     y = 3*1 = 3  
     L = 0.5*(3-2)^2 = 0.5
  b) dL/dy  = y - t = 3 - 2 = 1
     dL/dw2 = dL/dy * dy/dw2 = 1 * h = 1
     dL/dh  = dL/dy * dy/dh  = 1 * w2 = 3
     dL/dz  = dL/dh * dh/dz  = 3 * 1 = 3
     dL/dw1 = dL/dz * dz/dw1 = 3 * x = 6
  c) w1 = 0.5 - 0.1*6 = -0.1
     w2 = 3 - 0.1*1 = 2.9

  d) With w1 = -0.5: z = -1, h = ReLU(-1) = 0, y = 0, dL/dy = y - t = -2.
     dL/dw2 = dL/dy * h = -2 * 0 = 0
     dL/dh = -2 * 3 = -6, but dh/dz = ReLU'(-1) = 0, so dL/dz = 0 and dL/dw1 = 0
     
     Both gradients are zero: no learning signal (a "dead" ReLU neuron), so the
     weights are stuck even though the loss (L = 0.5*(0-2)^2 = 2) is large.

6. 
  a) xavg = 1.5, yavg = 2
     x - xavg = (-1.5, -0.5, 0.5, 1.5)  
     y - yavg = (-1, 0, 0, 1)
     Numerator = 1.5 + 0 + 0 + 1.5 = 3   
     Denominator = 2.25+0.25+0.25+2.25 = 5
     w = 3/5 = 0.6   
     b = 2 - 0.6*1.5 = 1.1
     
  b) for x=4, we have: 0.6*4 + 1.1 = 3.5

  c) Predictions: 1.1, 1.7, 2.3, 2.9  
     Residuals (y - yavg): -0.1, 0.3, -0.3, 0.1
     Squared: 0.01, 0.09, 0.09, 0.01
     sum: 0.20
     sum/4, MSE = 0.20/4 = 0.05
     
  d) true. Linear refers to linearity in the PARAMETERS (w1, w2, b), not in x:
     the model is linear in the feature vector (x, x^2). 

7. f'(w) = 2(w - 3)
  a) w1 = 0 - 0.25 * 2(0-3) = 0 - 0.25*(-6) = 1.5
     w2 = 1.5 - 0.25 * 2(1.5-3) = 1.5 - 0.25*(-3) = 2.25
     
  b) w1 = 0 - 1.5*(-6) = 9   
     It diverges: the error w - 3 gets multiplied by (1 - 2*eta) = -2 
     each step (w2 = 9 - 1.5*12 = -9, error alternates in sign and 
     doubles in size), i.e., the learning rate is too large.
     
  c) overfitting (the model is memorizing the training data / not generalizing).
     possible fixes: early stopping, weight decay, dropout, 
     more training data, a smaller model (force it not to memorize).

8. 
  a) a = 2 + 3 = 5
     f = 5 * 3 = 15
     
  b) df/df = 1
     df/da = y = 3.
     df/dx = df/da * da/dx = 3 * 1 = 3
     y has two paths: 
        directly through f = a*y gives df/dy|direct = a = 5, 
        and through a gives df/da * da/dy = 3 * 1 = 3
        contributions are aggregated: df/dy = 5 + 3 = 8

  c) A neural network has one scalar loss but millions of parameters; one reverse
     pass gives the gradient with respect to ALL parameters at a cost comparable
     to a forward pass, whereas forward-mode needs one pass per parameter/input.
     
9. 

  a) Pair counts in ababcab: (a,b) = 3, (b,a) = 1, (b,c) = 1, (c,a) = 1
     Most frequent pair = (a,b), merged into a new token "ab"
     New sequence: [ab, ab, c, ab]
     New vocabulary: {a, b, c, ab} (size 4)
     
  b) <king,queen> = 1*2 + 2*1 = 4
     |king| = |queen| = sqrt(5) 
     <king,queen> / (|king| * |queen|) = 4/5 = 0.8
     
     <king,apple> = 1*2 + 2*(-1) = 0
     |apple| = sqrt(5)
     <king,apple> / (|king| * |apple|) = 0/5 = 0

     queen is closer to king than apple. 

  ci)  Subwords keep the vocabulary a manageable size while still being able 
      to represent any word

  cii) one-hot vectors are huge, sparse, and are all equally distant
   (nearby words are not "near" any other similar).

10.    <a,b> is dot/inner product of a,b.

   a) sqrt(d_k) = 2.  
     <q,k1> = 4
     s1 = 4/2 = 2   
     <q,k2> = 0
     s2 = 0
     
   b) alpha1 = e^2 / (e^2 + e^0) = 7.39 / 8.39 =~ 0.8808
      alpha2 = 1 / 8.39 =~ 0.119   (weights sum to 1).
      
   c) output = alpha1*v1 + alpha2*v2 = 0.881*(10,0) + 0.119*(0,10)
             = (8.81, 1.19)  

   d) Dot products of d_k-dimensional vectors grow in magnitude with d_k
      Large scores push softmax into saturated regions where gradients 
      are tiny. Dividing by sqrt(d_k) keeps scores at a stable scale so 
      softmax and training remain well-behaved.