due Sunday, November 15th, 2026 at 11:59PM Ann Arbor Time (no slip days allowed!)
Except for the video link in Problem 1, write your solutions to the following problems either by writing them on a piece of paper or on a tablet and scanning your answers as a PDF. Note that you are not allowed to use LaTeX, Google Docs, or any other digital document creation software to type your answers. Homeworks are due to Pensive by 11:59PM on the due date. See the syllabus for details on the slip day policy.
Homework will be evaluated not only on the correctness of your answers, but on your ability to present your ideas clearly and logically. You should always explain and justify your conclusions, using sound reasoning. Your goal should be to convince the reader of your assertions. If a question does not require explanation, it will be explicitly stated.
Before proceeding, make sure you’re familiar with the collaboration policy.
Total Points: \(10 + 12 + 10 + 10 + 8 + 18 + 12 = 80\)
Record a 2- to 5-minute video answering one of the prompts below. These prompts are related to the ideas in Homework 7.
When finding the optimal parameter \(\vec w^{\ast}\) in multiple linear regression, depending on the dataset, there may be infinitely many optimal \(\vec w^{\ast}\)s that all equally minimize mean squared error, but there is always only one optimal prediction vector. Explain how both of these facts are true simultaneously.
Suppose you project a vector onto \(\text{colsp}(X)\), and then project the resulting vector onto \(\text{colsp}(X)\) again. What happens and why? What property of a projection matrix \(P\) does this relate to?
Suppose we add a feature that is a linear combination of the existing columns of the design matrix. What impact will this have on the (training) mean squared error of the resulting model?
The video should be just you speaking into the camera. Your face must be visible while you are talking for the entire video. Do not include slides, a screen recording, written work, or any other visuals.
Submit your video link to the separate assignment on Pensive called Homework 8, Problem 1 Video Link. You can upload your video to YouTube as unlisted, share a Zoom recording link, or share a Google Drive link; whatever works. Make sure the course staff can view the video using that link.
Your submission time for Homework 8 is the latest of your two submission times: your Problem 1 video link and your PDF.
This problem is graded on completion. Record a video of the required length and make an honest effort to explain the question, and you will receive full credit. Your explanation does not need to be polished or perfect.
Suppose we’d like to fit a multiple linear regression model with an intercept and two features. The model is fit by minimizing mean squared error on \(n > 3\) training points.
Let \(X\) be the \(n \times 3\) design matrix, whose first column is all 1s, and assume its columns are linearly independent. Let \(\vec y \in \mathbb{R}^n\) be the observation vector and \(\vec w^{\ast} \in \mathbb{R}^3\) be the optimal parameter vector. Let \(\vec 1 \in \mathbb{R}^n\) be the vector of all 1s.
For each expression below, state its type – either scalar, vector, or matrix – and shape (for non-scalars), and choose the most appropriate English description from among the options provided. No justification is required. The first item has been completed for you.
\((X^TX)^{-1}X^T\vec y\)
Solution: Optimal parameter vector; vector in \(\mathbb{R}^3\).
(2 pts) \(X(X^TX)^{-1}X^T\)
(2 pts) \(X(X^TX)^{-1}X^T\vec y\)
(2 pts) \(\vec y - X\vec w^{\ast}\)
(2 pts) \(X^TX\vec w^{\ast} - X^T\vec y\)
(2 pts) \(\vec 1^T(\vec y - X\vec w^{\ast})\)
(1 pt) \(\lVert \vec y - X\vec w^{\ast} \rVert^2\)
(1 pt) \(\displaystyle\frac{1}{n}\lVert \vec y - X\vec w^{\ast} \rVert^2\)
Possible descriptions
Training mean squared error
Optimal parameter vector
Projection matrix
Zero vector
Residual vector
Sum of squared training errors
Vector of predictions on the training points
The scalar zero
Reflection matrix
Consider a set of \(n\) points, \((\vec x_1, y_1), (\vec x_2, y_2), …, (\vec x_n, y_n)\), where each \(\vec x_i\) is a feature vector in \(\mathbb{R}^d\) and each \(y_i\) is a scalar.
(4 pts) To fit the model
we minimize mean squared error,
meaning that \(\vec w^{\ast}\) is chosen to satisfy the normal equations. Explain why the components of the error vector,
are guaranteed to sum to 0.
(6 pts) If we decide instead to fit the model
which has no intercept term, are the components of the error vector \(\vec e = \vec y - X \vec w^{\ast}\) still guaranteed to sum to 0? If they are, explain why. If they are not, explain why not, but give at least one example dataset where they still do sum to 0.
Let \(X\) be an \(n \times 4\) design matrix whose first column is all 1s, let \(\vec y\) be an observation vector, and let \(\vec w^{\ast} = (X^TX)^{-1}X^T \vec y\).
In this problem, you’ll reason about modifications to the design matrix and see how they affect the components of \(\vec w^{\ast}\).
(3 pts) Let \(X_a\) be the design matrix that results from swapping the first two columns of \(X\). Let
\(\vec v^{\ast} = (X_a^TX_a)^{-1}X_a^T \vec y\). Express the components of \(\vec v^{\ast}\) in terms of \(w_0^{\ast}, w_1^{\ast}, w_2^{\ast}, w_3^{\ast}\).
(3 pts) Let \(X_b\) be the design matrix that results from adding 3 to each entry in the first column of \(X\). Let \(\vec v^{\ast} = (X_b^TX_b)^{-1}X_b^T \vec y\). Express the components of \(\vec v^{\ast}\) in terms of \(w_0^{\ast}, w_1^{\ast}, w_2^{\ast}, w_3^{\ast}\).
(4 pts) Let \(X_c\) be the design matrix that results from adding 3 to each entry in the second column of \(X\). Let \(\vec v^{\ast} = (X_c^TX_c)^{-1}X_c^T \vec y\). Express the components of \(\vec v^{\ast}\) in terms of \(w_0^{\ast}, w_1^{\ast}, w_2^{\ast}, w_3^{\ast}\).
Let \(f(\vec x) = (x_1 - 5)^2 + (x_1^2 - x_2)^2 + 1\).
(5 pts) Find \(\nabla f(\vec x)\), the gradient of \(f(\vec x)\).
(3 pts) To minimize \(f(\vec x)\), we’ll use gradient descent. Perform one iteration of gradient descent by hand, using the initial guess \(\vec x^{(0)} = \begin{bmatrix} 0 \\ 1 \end{bmatrix}\) and learning rate \(\alpha = \frac{1}{2}\). What is \(\vec x^{(1)}\)?
Our goal in this problem is to study the behavior of the function
where \(x \in \mathbb{R}^n\) and \(A\) is a symmetric \(n \times n\) matrix (meaning \(A = A^T\)). This function, called the Rayleigh quotient, will play an important role in Chapter 10 of the course, when we eventually study the dimensionality reduction problem first introduced in Chapter 1.1.
But first, we have to get a handle on a few gradient rules.
(6 pts) As described in the Norm and Chain Rule in Chapter 8.2, the chain rule for gradients says that if
\(g: \mathbb{R}^d \to \mathbb{R}\) is a vector-to-scalar function, and
\(h: \mathbb{R} \to \mathbb{R}\) is a scalar-to-scalar function,
then the gradient of the vector-to-scalar function \(f(\vec x) = h(g(\vec x))\) is given by
or, perhaps more intuitively,
Note that we need to pay close attention to the types of functions we’re working with. \(h(g(\vec x))\) is well-defined, but \(g(h(\vec x))\) is not, since \(h\) doesn’t take in vectors (it takes in scalars).
Find the gradients of each of the following functions.
(3 pts) \(f_1(\vec x) = \log(\vec x^T A \vec x)\), where \(\vec x \in \mathbb{R}^n\) and \(A\) is a symmetric \(n \times n\) matrix
(3 pts) \(f_2(\vec x) = e^{-\sin(\vec a^T\vec x)}\), where \(\vec x, \vec a \in \mathbb{R}^n\)
Here, \(\log(x)\) denotes the base-\(e\) logarithm, i.e. \(\ln(x)\).
Hint: You can use any of the three important gradient rules from Chapter 8.2 without proof.
(6 pts) The product rule for gradients is a natural extension of the product rule for derivatives. If \(f(\vec x) = g(\vec x) h(\vec x)\), then
Find the gradients of each of the following functions.
(2 pts) \(f_3(\vec x) = (\vec a \cdot \vec x)(\vec b \cdot \vec x)\), where \(\vec x, \vec a, \vec b \in \mathbb{R}^n\)
(4 pts) \(f_4(\vec x) = \vec a^T \vec x \vec x^T A \vec x\), where \(\vec x, \vec a \in \mathbb{R}^n\) and \(A\) is a symmetric \(n \times n\) matrix
(6 pts) Putting together the chain rule and product rule, show that if
where \(\vec x \in \mathbb{R}^n\) and \(A\) is a symmetric \(n \times n\) matrix, then
In Chapter 8.5, we introduce the formal definition of convexity for vector-to-scalar functions. Formally, the function \(f: \mathbb{R}^d \to \mathbb{R}\) is convex if for all \(\vec x, \vec y \in \mathbb{R}^d\) and all \(t \in [0,1]\),
This is a formal way of saying that when you connect any two points on the graph of \(f\) with a line segment, the line segment lies on or above the graph of \(f\), never below.
For each statement below, prove that the statement is true using the formal definition above, or give a counterexample. You don’t need to prove that a counterexample does not satisfy the formal definition; visual counterexamples are sufficient.
(5 pts) The sum of two convex functions must also be convex.
(3 pts) The difference of two convex functions must also be convex.
(4 pts) Suppose \(f(x)\) and \(g(x)\) are both scalar-to-scalar convex functions and that, for some scalar \(a\), \(f(a) = g(a)\). Then, \(h(x)\) is also convex, where
Hint: The statement is false, so focus your energy on finding a counterexample.