Homework 3: Vectors and the Dot Product

due Friday, September 25th, 2026 at 11:59PM Ann Arbor Time

Write your solutions to the following problems either by writing them on a piece of paper or on a tablet and scanning your answers as a PDF. Note that you are not allowed to use LaTeX, Google Docs, or any other digital document creation software to type your answers. Homeworks are due to Pensive by 11:59PM on the due date. See the syllabus for details on the slip day policy.

Homework will be evaluated not only on the correctness of your answers, but on your ability to present your ideas clearly and logically. You should always explain and justify your conclusions, using sound reasoning. Your goal should be to convince the reader of your assertions. If a question does not require explanation, it will be explicitly stated.

Before proceeding, make sure you’re familiar with the collaboration policy.

Total Points: 10 + 6 + 6 + 7 + 13 + 12 + 10 + 8 + 9 + 13 + 10 = 104

Problem 1: Homework 2 Solutions Review (10 pts)

Review the solutions to Homework 2. Pick two problem parts (for example, Problem 2a and Problem 4b) from Homework 2 in which your solutions have the most room for improvement, i.e., where they have unsound reasoning, could be significantly more efficient or clearer, etc. Include a screenshot of your solution to each problem part, and in a few sentences, explain what was deficient and how it could be fixed.

Alternatively, if you think one of your solutions is significantly better than the posted one, copy it here and explain why you think it is better. If you didn’t do Homework 2, choose two problem parts from it that look challenging to you, and in a few sentences, explain the key ideas behind their solutions in your own words.

Solution

Problem 2: Feedback (6 pts)

We’d like to get your feedback on how the course has been going so far, now that we’re a few weeks in.

You can find the survey at this link. It is not anonymous, but it links to an anonymous feedback form if you’d like to provide some feedback anonymously.

When submitting to Pensive, it does not matter which page of your submission you assign to Problem 2; we will enter survey completion credit in manually.

Thank you for your feedback — it’s helping shape our relatively new course.

Problem 3: Drawing Linear Combinations (6 pts)

In this problem, draw all vectors on the same set of axes, like the one below. Your final graph should contain five vectors. Draw each vector starting at the origin, and label it.

Draw $\vec u=\begin{bmatrix}3\\4\end{bmatrix}$ .
Draw $\vec v=\begin{bmatrix}-1\\-4\end{bmatrix}$ .
(2 pts) Draw $\vec a=-\vec u-\vec v$ . Verify that your drawing is correct by finding $\vec a$ ’s components.
(2 pts) Draw $\vec b=2\vec u+\vec v$ . Verify that your drawing is correct by finding $\vec b$ ’s components.
(2 pts) Draw $\vec c=\frac12\vec u-\frac12\vec v$ . Verify that your drawing is correct by finding $\vec c$ ’s components.

Figure from the assignment PDF

Solution

The five vectors have the following components:

$\displaystyle \vec u=\begin{bmatrix}3\\4\end{bmatrix}$ .
$\displaystyle \vec v=\begin{bmatrix}-1\\-4\end{bmatrix}$ .
$\displaystyle \vec a=-\vec u-\vec v =-\begin{bmatrix}3\\4\end{bmatrix}-\begin{bmatrix}-1\\-4\end{bmatrix} =\begin{bmatrix}-2\\0\end{bmatrix}$ .
$\displaystyle \vec b=2\vec u+\vec v =2\begin{bmatrix}3\\4\end{bmatrix}+\begin{bmatrix}-1\\-4\end{bmatrix} =\begin{bmatrix}5\\4\end{bmatrix}$ .
$\displaystyle \vec c=\frac12\vec u-\frac12\vec v =\frac12\begin{bmatrix}3\\4\end{bmatrix}-\frac12\begin{bmatrix}-1\\-4\end{bmatrix} =\begin{bmatrix}2\\4\end{bmatrix}$ .

Each arrow below starts at the origin and ends at the corresponding point.

Figure from the assignment PDF

Problem 4: Displacement and Distance (7 pts)

A drone begins at the point

$$ P=(2,-1,4) $$

and flies in a straight line to the point

$$ Q=(6,7,-4). $$

The coordinates here are in meters.

(1 pt) Let $\vec{d}$ be the vector that describes the displacement from $$P$$ to $$Q$$ . Find $\vec{d}$ .

Solution

Subtract the initial coordinates from the final coordinates:

\vec{d}=\begin{bmatrix}6-2\\\\7-(-1)\\\\-4-4\end{bmatrix} =\begin{bmatrix}4\\\\8\\\\-8\end{bmatrix}.

(2 pts) Find the exact distance traveled by the drone.

Solution

The distance is the length of the displacement:

\|\vec{d}\|=\sqrt{4^2+8^2+(-8)^2}=\sqrt{144}=12\text{ meters}.

(4 pts) Suppose the drone continues to travel another 10 meters in the same direction. What are its coordinates?

Solution

Normalize the displacement to obtain the direction:

\frac{\vec{d}}{\|\vec{d}\|}=\frac1{12}\begin{bmatrix}4\\\\8\\\\-8\end{bmatrix} =\begin{bmatrix}\frac13\\\\[2pt]\frac23\\\\[2pt]-\frac23\end{bmatrix}.

Scale this unit vector by $$10$$ and add the additional displacement to $$Q$$ , where the drone starts this part of its flight:

\begin{bmatrix}6\\\\7\\\\-4\end{bmatrix} +10\begin{bmatrix}\frac13\\\\[2pt]\frac23\\\\[2pt]-\frac23\end{bmatrix} =\begin{bmatrix}\frac{28}{3}\\\\[2pt]\frac{41}{3}\\\\[2pt]-\frac{32}{3}\end{bmatrix}.

The new coordinates are $\left(\frac{28}{3},\frac{41}{3},-\frac{32}{3}\right)$ .

Problem 5: Distributing the Dots (13 pts)

Let $\vec{u},\vec{v}\in\mathbb{R}^n$ satisfy

\lVert\vec{u}\rVert=3,\qquad \lVert\vec{v}\rVert=2,\qquad (3\vec{u}-4\vec{v})\cdot(\vec{u}+9\vec{v})=-71.

(3 pts) Find $\vec{u}\cdot\vec{v}$ . Show your work.

Solution

Distributing the dot product gives

$$ $$

\begin{aligned} (3\vec{u}-4\vec{v})\cdot(\vec{u}+9\vec{v}) &=3\lVert\vec{u}\rVert^2+27\vec{u}\cdot\vec{v} -4\vec{v}\cdot\vec{u}-36\lVert\vec{v}\rVert^2

&=27+23\vec{u}\cdot\vec{v}-144. \end{aligned}

$$ $$

Thus $23\vec{u}\cdot\vec{v}=46$ , so $\boxed{\vec{u}\cdot\vec{v}=2}$ .

(2 pts) Find $\lVert-2\vec{u}\rVert$ and $\lVert3\vec{v}\rVert$ . Explain why neither length is negative.

Solution

The scaling property gives

\lVert-2\vec{u}\rVert=|-2|\lVert\vec{u}\rVert=2(3)=6, \qquad \lVert3\vec{v}\rVert=|3|\lVert\vec{v}\rVert=3(2)=6.

Lengths are nonnegative. A negative scalar reverses direction, but scales length by its absolute value.

(3 pts) Recall, the cosine similarity of two nonzero vectors $\vec u,\vec v$ is defined as

\cos\theta=\frac{\vec u\cdot\vec v}{\lVert\vec u\rVert\lVert\vec v\rVert},

where $\theta$ is the angle between the vectors.

Find the cosine similarity of $\vec{u}$ and $\vec{v}$ . Then find the cosine similarity of $-2\vec{u}$ and $3\vec{v}$ . If the two cosine similarities are different, why are they different?

Solution

The original cosine similarity is

\frac{\vec{u}\cdot\vec{v}}{\lVert\vec{u}\rVert\lVert\vec{v}\rVert} =\frac{2}{3(2)}=\frac13.

For the scaled vectors, it is

\frac{(-2\vec{u})\cdot(3\vec{v})}{\lVert-2\vec{u}\rVert\lVert3\vec{v}\rVert} =\frac{-6(\vec{u}\cdot\vec{v})}{6(6)} =\frac{-12}{36}=-\frac13.

The cosine similarities differ because multiplying $\vec{u}$ by $$-2$$ reverses its direction, while multiplying $\vec{v}$ by $$3$$ preserves its direction. Reversing exactly one vector changes the sign of the cosine similarity; the positive scaling factors cancel between the numerator and denominator.

(1 pt) Find $\vec{u}\cdot\vec{u}$ and $\vec{v}\cdot\vec{v}$ .

Solution

A vector’s dot product with itself is its squared length, so

\vec{u}\cdot\vec{u}=\lVert\vec{u}\rVert^2=3^2=9, \qquad \vec{v}\cdot\vec{v}=\lVert\vec{v}\rVert^2=2^2=4.

(4 pts) Find $\lVert\vec{u}+\vec{v}\rVert$ . Is it equal to $\sqrt{\lVert\vec{u}\rVert^2+\lVert\vec{v}\rVert^2}$ ? Explain what condition would make these quantities equal.

Hint: Start by writing $\lVert\vec{u}+\vec{v}\rVert^2$ as $(\vec{u}+\vec{v})\cdot(\vec{u}+\vec{v})$ .

Solution

Expanding gives

\lVert\vec{u}+\vec{v}\rVert^2 =\lVert\vec{u}\rVert^2+2\vec{u}\cdot\vec{v}+\lVert\vec{v}\rVert^2 =9+4+4=17.

Thus $\lVert\vec{u}+\vec{v}\rVert=\sqrt{17}$ , whereas $\sqrt{\lVert\vec{u}\rVert^2+\lVert\vec{v}\rVert^2}=\sqrt{13}$ . The quantities are equal exactly when $\vec{u}\cdot\vec{v}=0$ , meaning the vectors are orthogonal.

Problem 6: Parallelogram Law (12 pts)

(2 pts) Let $\vec{u} = \begin{bmatrix} 3 \\ -6 \\ 0 \\ 2 \end{bmatrix}$ and $\vec{v} = \begin{bmatrix} 2 \\ 1 \\ 4 \\ -2 \end{bmatrix}$ . Compute the following quantities:

$\lVert \vec{u} \rVert$
$\lVert \vec{v} \rVert$
$\lVert \vec{u} + \vec{v} \rVert$
$\lVert \vec{u} - \vec{v} \rVert$

Solution

$\lVert \vec{u} \rVert = \sqrt{3^2 + (-6)^2 + 0^2 + 2^2} = \sqrt{9 + 36 + 0 + 4} = \sqrt{49} = 7$
$\lVert \vec{v} \rVert = \sqrt{2^2 + 1^2 + 4^2 + (-2)^2} = \sqrt{4 + 1 + 16 + 4} = \sqrt{25} = 5$
$\begin{align*} \lVert \vec{u} + \vec{v} \rVert &= \sqrt{(3+2)^2 + (-6+1)^2 + (0+4)^2 + (2-2)^2} \\\\ &= \sqrt{25 + 25 + 16 + 0} \\\\ &= \sqrt{66} \end{align*}$
$\begin{align*} \lVert \vec{u} - \vec{v} \rVert &= \sqrt{(3-2)^2 + (-6-1)^2 + (0-4)^2 + (2 - (-2))^2} \\\\ &= \sqrt{1 + 49 + 16 + 16} \\\\ &= \sqrt{82} \end{align*}$

(2 pts) Using the same vectors as in part a), compute the angle between $\vec{u}$ and $\vec{v}$ . Leave your answer in terms of $\cos^{-1}$ . We asked you to do something very similar in Problem 5c, but here we are asking for the angle itself, which is why your answer involves $\cos^{-1}$ .

Solution

Since $\vec{u} \cdot \vec{v} = \lVert \vec{u} \rVert \lVert \vec{v} \rVert \cos \theta$ , we have:

\begin{align*} \vec u \cdot \vec v &= 3\cdot 2 + (-6)\cdot 1 + 0\cdot 4 + 2\cdot (-2) = 6 - 6 + 0 - 4 = -4,\\\\ \|\vec u\| &= \sqrt{3^2 + (-6)^2 + 0^2 + 2^2} = \sqrt{49} = 7,\\\\ \|\vec v\| &= \sqrt{2^2 + 1^2 + 4^2 + (-2)^2} = \sqrt{25} = 5,\\\\ \cos\theta &= \frac{\vec u \cdot \vec v}{\|\vec u\|\;\|\vec v\|} = \frac{-4}{7\cdot 5} = -\frac{4}{35}.\\\\ \theta &= \cos^{-1}\!\left(-\dfrac{4}{35}\right) \end{align*}

(4 pts) Now, suppose $\vec{u} = \begin{bmatrix} u_1 \\ u_2 \\ \vdots \\ u_n \end{bmatrix}$ and $\vec{v} = \begin{bmatrix} v_1 \\ v_2 \\ \vdots \\ v_n \end{bmatrix}$ are any two vectors in $\mathbb{R}^n$ . Prove that:

\lVert \vec{u} + \vec{v} \rVert^2 + \lVert \vec{u} - \vec{v} \rVert^2 = 2\lVert \vec{u} \rVert^2 + 2\lVert \vec{v} \rVert^2

The statement above is called the parallelogram law of vectors.

Hint: The point of part a) was to give you a feel for which quantities are involved in this statement. Your proof should not use these values in particular. Refer to the hint from Problem 5e.

Solution

Since $|\vec u|^2 = \vec u \cdot \vec u$ , we can expand both $|\vec u+\vec v|^2$ and $|\vec u-\vec v|^2$ using the distributive property of the dot product, as mentioned in the hint. Below, we color $\vec u\cdot\vec u$ in orange, $\vec v\cdot\vec v$ in blue, and $\vec u\cdot\vec v$ in magenta.

\begin{align*} \|\vec u+\vec v\|^2 &= (\vec u+\vec v)\cdot(\vec u+\vec v) \\\\ &= \vec u\cdot\vec u + \vec u\cdot\vec v + \vec v\cdot\vec u + \vec v\cdot\vec v \\\\ &= \vec u\cdot\vec u + \vec v\cdot\vec v + 2\,\vec u\cdot\vec v. \end{align*}

\begin{align*} \|\vec u-\vec v\|^2 &= (\vec u-\vec v)\cdot(\vec u-\vec v) \\\\ &= \vec u\cdot\vec u - \vec u\cdot\vec v - \vec v\cdot\vec u + \vec v\cdot\vec v \\\\ &= \vec u\cdot\vec u + \vec v\cdot\vec v - 2\,\vec u\cdot\vec v. \end{align*}

Note that the $+2\vec u\cdot\vec v$ from the first expansion and the $-2\vec u\cdot\vec v$ from the second cancel when we add them. Thus,

\begin{align*} \|\vec u+\vec v\|^2 + \|\vec u-\vec v\|^2 &= \left(\vec u\cdot\vec u + \vec v\cdot\vec v + 2\,\vec u\cdot\vec v\right) \\\\ &\quad + \left(\vec u\cdot\vec u + \vec v\cdot\vec v - 2\,\vec u\cdot\vec v\right) \\\\ &= 2\,\vec u\cdot\vec u + 2\,\vec v\cdot\vec v \\\\ &= 2\|\vec u\|^2 + 2\|\vec v\|^2. \end{align*}

(4 pts) Why is the equality from part c) called the parallelogram law? Let’s explore.

Suppose points $$A$$ , $$B$$ , $$C$$ , and $$D$$ in $\mathbb{R}^n$ form a parallelogram: a polygon with four sides where opposite sides are parallel and equal in length.

Using the results of the previous part of this problem, prove that the sum of the squares of the side lengths of the parallelogram is equal to the sum of the squares of the diagonals. In other words, prove that:

$$ (AB)^2 + (BC)^2 + (CD)^2 + (DA)^2 = (AC)^2 + (BD)^2 $$

where $$AB$$ represents the length of the segment from point $$A$$ to point $$B$$ , etc.

Hint: Define two vectors, $\vec u$ and $\vec v$ , and explain why the result from the previous part of this problem implies the desired equality. This is mostly an English problem.

Solution

Let $\vec u$ be the vector along $\overrightarrow{AB}$ (purple arrow in the figure) and $\vec v$ be the vector along $\overrightarrow{BC}$ (green arrow).

Opposite sides of a parallelogram are equal and parallel, so the remaining sides point as

\overrightarrow{CD}=-\,{\vec u},\qquad \overrightarrow{DA}=-\,{\vec v}.

Next, read each diagonal as a head–to–tail sum of side vectors.

\overrightarrow{AC}=\overrightarrow{AB}+\overrightarrow{BC} = {\vec u}+{\vec v},\qquad \overrightarrow{BD}=\overrightarrow{BC}+\overrightarrow{CD} = {\vec v}-{\vec u}.

Now translate side lengths into norms of the corresponding vectors.

\begin{align*} (AB) &= \|{\vec u}\|, & (BC) &= \|{\vec v}\|, & (CD) &= \|{-\,{\vec u}}\|=\|{\vec u}\|, & (DA) &= \|{-\,{\vec v}}\|=\|{\vec v}\|. \end{align*}

Squaring and summing the four sides gives

\begin{align*} \big(AB\big)^2+\big(BC\big)^2 +\big(CD\big)^2+\big(DA\big)^2 &= \|{\vec u}\|^2+\|{\vec v}\|^2 +\|{\vec u}\|^2+\|{\vec v}\|^2 \\\\ &= 2\|{\vec u}\|^2 + 2\|{\vec v}\|^2. \end{align*}

Do the same for the diagonals, using the vector expressions above.

\begin{align*} (AC) &= \|\,{\vec u}+{\vec v}\,\|, & (BD) &= \|\,{\vec v}-{\vec u}\,\|. \end{align*}

Squaring and adding the diagonals yields

\begin{align*} \big(AC\big)^2+\big(BD\big)^2 &= \|\,{\vec u}+{\vec v}\,\|^2 + \|\,{\vec u}-{\vec v}\,\|^2. \end{align*}

By the parallelogram law proven in part (c),

\|\,{\vec u}+{\vec v}\,\|^2 +\|\,{\vec u}-{\vec v}\,\|^2 = 2\|{\vec u}\|^2 + 2\|{\vec v}\|^2.

Combining with the side-length sum computed above,

\boxed{\; \big(AB\big)^2+\big(BC\big)^2 +\big(CD\big)^2+\big(DA\big)^2 \;=\; \big(AC\big)^2+\big(BD\big)^2 \;}

Problem 7: Linear Combinations (10 pts)

Much of our study of linear algebra involves understanding the set of possible linear combinations of a given set of vectors. As the notes detail, our multiple linear regression problem boils down to finding the best possible linear combination of the features, so it’s important that we understand how linear combinations work.

Let

\vec v_1 = \begin{bmatrix} 2 \\\\ 3 \\\\ 0 \end{bmatrix},\quad \vec v_2 = \begin{bmatrix} -1 \\\\ 2 \\\\ -1 \end{bmatrix},\quad \vec v_3 = \begin{bmatrix} 0 \\\\ 5 \\\\ 2 \end{bmatrix},\quad \vec x = \begin{bmatrix} -6 \\\\ 1 \\\\ 4 \end{bmatrix}.

You can find an interactive, three-dimensional visualization of these four vectors at this link:

https://eecs245.org/resources/homeworks/hw03/hw03-problem-7.html

We recommend you have this visual open while you work through this problem.

(4 pts) Find constants $$a$$ , $$b$$ , and $$c$$ such that

a \vec v_1 + b \vec v_2 + c \vec v_3 = \vec x

In other words, write $\vec x$ as a linear combination of $\vec v_1$ , $\vec v_2$ , and $\vec v_3$ .

Hint: Start by writing out the equation as a system of equations. Then, use your favorite method for solving systems of equations to find $$a$$ , $$b$$ , and $$c$$ . We reviewed how to solve systems of equations in Lab 3.

Solution

$$a=-3, b=0, c=2$$

We can express the equation above as a system of three equations in the three unknowns $$a,b,c$$ .

\begin{align*} a\vec v_1 + b\vec v_2 + c\vec v_3 &= \vec x \\\\ a\begin{bmatrix}2\\\\3\\\\0\end{bmatrix} + b\begin{bmatrix}-1\\\\2\\\\-1\end{bmatrix} + c\begin{bmatrix}0\\\\5\\\\2\end{bmatrix} &= \begin{bmatrix}-6\\\\1\\\\4\end{bmatrix} \end{align*}

Equating components gives the system

\begin{align*} 2a - b &= -6 \tag{1}\\\\ 3a + 2b + 5c &= 1 \tag{2}\\\\ -b + 2c &= 4 \tag{3} \end{align*}

In $$(1)$$ , isolate $$b$$ :

\begin{align*} b &= 6 + 2a \tag{4} \end{align*}

Substitute this expression for $$b$$ into $$(3)$$ :

\begin{align*} -(6+2a) + 2c &= 4 \\\\ 2c &= 10 + 2a \\\\ c &= 5 + a \tag{5} \end{align*}

Now substitute $$(4)$$ and $$(5)$$ into $$(2)$$ :

\begin{align*} 3a + 2(6+2a) + 5(5+a) &= 1 \\\\ 3a + 12 + 4a + 25 + 5a &= 1 \\\\ 12a + 37 &= 1 \\\\ 12a &= -36 \\\\ a &= -3 \tag{6} \end{align*}

Back-substitute to get $$b$$ and $$c$$ :

\begin{align*} b &= 6 + 2(-3) = 0 \\\\ c &= 5 + (-3) = 2 \end{align*}

So: to verify that these values are correct, compute the linear combination:

\begin{align*} a\vec v_1 + b\vec v_2 + c\vec v_3 &= (-3)\begin{bmatrix}2\\\\3\\\\0\end{bmatrix} + 0\begin{bmatrix}-1\\\\2\\\\-1\end{bmatrix} + 2\begin{bmatrix}0\\\\5\\\\2\end{bmatrix} \\\\ &= \begin{bmatrix}-6\\\\-9\\\\0\end{bmatrix} + \begin{bmatrix}0\\\\0\\\\0\end{bmatrix} + \begin{bmatrix}0\\\\10\\\\4\end{bmatrix} \\\\ &= \begin{bmatrix}-6\\\\1\\\\4\end{bmatrix} = \vec x \end{align*}

\boxed{\,a=-3,\; b=0,\; c=2\,}

(3 pts) Try and find constants $$d$$ and $$e$$ such that

d \vec v_1 + e \vec v_3 = \vec x

If you are able to find constants $$d$$ and $$e$$ , explain why, even though there are two unknowns but three equations for them. If you are unable to find constants $$d$$ and $$e$$ , explain why no solution exists.

Solution

$$d=-3, e=2$$

This is a system of three equations and two unknowns, since there are only two vectors in the linear combination. We’ll start by solving it the same way we did in part (a).

\begin{align*} d\vec v_1 + e\vec v_3 &= \vec x \\\\ d\begin{bmatrix}2\\\\3\\\\0\end{bmatrix} + e\begin{bmatrix}0\\\\5\\\\2\end{bmatrix} &= \begin{bmatrix}-6\\\\1\\\\4\end{bmatrix} \end{align*}

Equating components gives

\begin{align*} 2d &= -6 \tag{1}\\\\ 3d + 5e &= 1 \tag{2}\\\\ 2e &= 4 \tag{3} \end{align*}

We can solve directly from $$(1)$$ and $$(3)$$ :

\begin{align*} d &= -3 \tag{4}\\\\ e &= 2 \tag{5} \end{align*}

Now check equation $$(2)$$ with these values:

\begin{align*} 3(-3) + 5(2) &= -9 + 10 = 1 \end{align*}

so $$(2)$$ holds as well. Notice we did not need $$(2)$$ to find $$d$$ and $$e$$ ; it served as a consistency check, which it passes, so a solution exists.

You could also have seen this without re-solving: in part (a), the coefficient on $\vec v_2$ was $$b=0$$ . Since $\vec v_2$ does not appear in the combination here, the same coefficients on $\vec v_1$ and $\vec v_3$ work, namely $$d=a=-3$$ and $$e=c=2$$ .

\boxed{\,d=-3,\; e=2\,}

Geometrically, $\vec x$ lies in the plane spanned by $\vec v_1$ and $\vec v_3$ (the set $\mathrm{span}\lbrace\vec v_1,\vec v_3\rbrace$ ), so we did not need $\vec v_2$ to reach $\vec x$ . (See next page)

Visual note: In the view above, the arrows for $\vec v_1$ , $\vec v_3$ , and $\vec x$ look almost colinear only because of the camera angle. What matters is that they are coplanar: $\vec x \in \mathrm{span}\lbrace\vec v_1,\vec v_3\rbrace$ . The arrow for $\vec v_2$ is not in that plane, which is why we did not need it in the combination. To see this clearly, open the interactive viewer and rotate the scene:

https://eecs245.org/resources/homeworks/hw03/hw03-problem-7.html

(3 pts) Try and find constants $$p$$ and $$q$$ such that

p \vec v_1 + q \vec v_2 = \vec x

If you are able to find constants $$p$$ and $$q$$ , explain why, even though there are two unknowns but three equations for them. If you are unable to find constants $$p$$ and $$q$$ , explain why no solution exists.

Solution

No solution.

This is a system of three equations in two unknowns, because the linear combination uses only two vectors. We’ll attempt to solve it the same way as in part (a).

\begin{align*} p\vec v_1 + q\vec v_2 &= \vec x \\\\ p\begin{bmatrix}2\\\\3\\\\0\end{bmatrix} + q\begin{bmatrix}-1\\\\2\\\\-1\end{bmatrix} &= \begin{bmatrix}-6\\\\1\\\\4\end{bmatrix} \end{align*}

Equating components gives the system

\begin{align*} 2p - q &= -6 \tag{1}\\\\ 3p + 2q &= 1 \tag{2}\\\\ -q &= 4 \tag{3} \end{align*}

From $$(3)$$ we get $$q=-4$$ . Use $$(1)$$ with $$q=-4$$ :

\begin{align*} 2p - (-4) &= -6 \\\\ 2p + 4 &= -6 \\\\ 2p &= -10 \\\\ p &= -5 \end{align*}

Now check $$(2)$$ with $p=-5,\ q=-4$ :

\begin{align*} 3(-5) + 2(-4) &= -15 - 8 = -23 \neq 1 \end{align*}

Since $$(2)$$ is violated, the system is inconsistent and there are no constants $$p,q$$ satisfying all three equations.

Why this also makes sense visually. From the 3D visualization, $\vec x$ lies in the plane generated by linear combinations of $\vec v_1$ and $\vec v_3$ $\big(\mathrm{span}\lbrace\vec v_1,\vec v_3\rbrace\big)$ . Trying to use only $\vec v_1$ and $\vec v_2$ keeps you in the different plane $\mathrm{span}\lbrace\vec v_1,\vec v_2\rbrace$ , which does not contain $\vec x$ . Hence no solution with $p\vec v_1+q\vec v_2=\vec x$ .

\boxed{\text{No solution}}

Problem 8: Correlation (8 pts)

In Chapter 2.4, you were told that the correlation coefficient, $$r$$ , ranges between $$-1$$ and $$1$$ , where $$-1$$ implies a perfect negative linear association and $$1$$ implies a perfect positive linear association. However, you were never given a proof of the fact that $-1 \leq r \leq 1$ .

Here, you will prove this fact, given your newfound understanding of vectors, the dot product, and angles.

(2 pts) Let $\vec x$ and $\vec y$ be two vectors in $\mathbb{R}^n$ . We define the “mean-centered” version of $\vec x$ to be:

\vec{x}_{\text{c}} = \begin{bmatrix} x_1 - \bar{x} \\\\ x_2 - \bar{x} \\\\ \vdots \\\\ x_n - \bar{x} \end{bmatrix}

where $\displaystyle \bar{x} = \frac{1}{n} \sum_{i=1}^n x_i$ is the mean of the components of $\vec{x}$ . The mean-centered version of $\vec y$ , named $\vec{y}_{\text{c}}$ , is defined similarly.

Express $\vec{x}_{\text{c}} \cdot \vec{y}_{\text{c}}$ using summation notation.

Solution

\begin{align*} \vec{x}_{\text{c}} \cdot \vec{y}_{\text{c}} &= \begin{bmatrix} x_1-\bar{x} \\\\[2pt] x_2-\bar{x} \\\\[2pt] \vdots \\\\[2pt] x_n-\bar{x} \end{bmatrix} \cdot \begin{bmatrix} y_1-\bar{y} \\\\[2pt] y_2-\bar{y} \\\\[2pt] \vdots \\\\[2pt] y_n-\bar{y} \end{bmatrix} \\\\[6pt] &= (x_1-\bar{x})(y_1-\bar{y}) + (x_2-\bar{x})(y_2-\bar{y}) + \cdots + (x_n-\bar{x})(y_n-\bar{y}) \\\\[6pt] &= \sum_{i=1}^n (x_i-\bar{x})(y_i-\bar{y}) \end{align*}

(4 pts) Prove that:

r = \frac{\vec{x}_{\text{c}} \cdot \vec{y}_{\text{c}}}{\lVert \vec{x}_{\text{c}} \rVert \lVert \vec{y}_{\text{c}} \rVert}

Do so by starting with the right-hand side of the equation, expanding it, and simplifying it until you reach the definition of $$r$$ .

Solution

From part (a), the dot product of the mean–centered vectors is

\vec{x}_{\mathrm c}\cdot \vec{y}_{\mathrm c} \;=\; \sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)

and the norms are

\|\vec{x}_{\mathrm c}\| \;=\; \sqrt{\sum_{i=1}^n (x_i-\bar x)^2}, \qquad \|\vec{y}_{\mathrm c}\| \;=\; \sqrt{\sum_{i=1}^n (y_i-\bar y)^2}

Start from the right-hand side and substitute these definitions:

\begin{align*} \frac{\vec{x}_{\mathrm c}\cdot \vec{y}_{\mathrm c}}{\|\vec{x}_{\mathrm c}\|\,\|\vec{y}_{\mathrm c}\|} &= \frac{\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)} {\sqrt{\sum_{i=1}^n (x_i-\bar x)^2}\;\sqrt{\sum_{i=1}^n (y_i-\bar y)^2}} \end{align*}

Recall the formula for the variance:

\sigma_x^2 = \frac{1}{n}\sum_{i=1}^n (x_i-\bar x)^2.

We need to isolate $\sqrt{\sum_{i=1}^n (x_i-\bar x)^2}$ . Multiplying both sides by $$n$$ and then taking the square root gives

n\sigma_x^2 = \sum_{i=1}^n (x_i-\bar x)^2 \qquad\Longrightarrow\qquad \sqrt{\sum_{i=1}^n (x_i-\bar x)^2} = \sqrt{n}\,\sigma_x.

Similarly, $\sqrt{\sum_{i=1}^n (y_i-\bar y)^2} = \sqrt{n}\sigma_y$ , so the denominator is $(\sqrt{n}\sigma_x)(\sqrt{n}\sigma_y) = n\sigma_x\sigma_y$ . This gives

\begin{align*} &= \frac{\sum_{i=1}^n (x_i-\bar x)(y_i-\bar y)}{\,n\,\sigma_x\,\sigma_y} \end{align*}

Finally, pull out the $$1/n$$ and group terms to match the correlation definition (average of standardized products):

\begin{align*} \frac{\vec{x}_{\mathrm c}\cdot \vec{y}_{\mathrm c}}{\|\vec{x}_{\mathrm c}\|\,\|\vec{y}_{\mathrm c}\|}&= \frac{1}{n}\sum_{i=1}^n \left(\frac{x_i-\bar x}{\sigma_x}\right) \left(\frac{y_i-\bar y}{\sigma_y}\right) \;=\; r \end{align*}

(2 pts) In 1-2 English sentences, explain why the result from part b) implies that $-1 \leq r \leq 1$ .

Solution

The result from part b) implies that $-1 \leq r \leq 1$ because the right-hand side of the equation is the cosine of the angle between the mean-centered vectors, and the cosine of an angle is bounded between $$-1$$ and $$1$$ .

Problem 9: Projections (9 pts)

In Chapter 3.4, we study the concept of projecting one vector onto one or more other vectors. In this problem, you’ll see how this concept can be thought of in terms of our friend from the first two weeks of the course: calculus.

Let $\vec x$ and $\vec y$ be two vectors in $\mathbb{R}^n$ . Consider the function $f: \mathbb{R} \to \mathbb{R}$ , defined as:

f(k) = \lVert \vec y - k \vec x \rVert^2

By $\mathbb{R} \to \mathbb{R}$ , we mean that $$f$$ takes in a single real number (i.e. a scalar, not a vector) and outputs a single real number. This means that we can find $\frac{\text{d} f}{\text{d} k}$ , the derivative of $$f$$ with respect to $$k$$ .

(4 pts) Rewrite $$f(k)$$ using the properties of the dot product from Chapter 3.3. Then, show that:

\frac{\text{d} f}{\text{d} k} = -2 \vec x \cdot \vec y + 2k \vec x \cdot \vec x

Solution

Start by rewriting the squared norm as a dot product and expanding. We color the dot–product pieces to track them: $\vec y\cdot\vec y$ (orange), $\vec x\cdot\vec x$ (blue), and $\vec x\cdot\vec y$ (magenta).

\begin{align*} f(k) &= \|\vec y - k\vec x\|^2 \\\\ &= (\vec y - k\vec x)\!\cdot\!(\vec y - k\vec x) \\\\ &= \vec y\!\cdot\!\vec y \;-\; k\,(\vec y\!\cdot\!\vec x) \;-\; k\,(\vec x\!\cdot\!\vec y) \;+\; k^2\,(\vec x\!\cdot\!\vec x) \\\\ &= \|\vec y\|^2 \;-\; 2k\,(\vec x\!\cdot\!\vec y) \;+\; k^2\,(\vec x\!\cdot\!\vec x) \qquad\text{(since }\vec x\!\cdot\!\vec y=\vec y\!\cdot\!\vec x\text{).} \end{align*}

Now differentiate term–by–term with respect to $$k$$ (the dot products are constants with respect to $$k$$ ):

\frac{\text{d}f}{\text{d}k} = 0 \;-\; 2\,(\vec x\!\cdot\!\vec y) \;+\; 2k\,(\vec x\!\cdot\!\vec x).

\boxed{\;\displaystyle \frac{\text{d}f}{\text{d}k} \;=\; -2\,\vec x\!\cdot\!\vec y \;+\; 2k\,(\vec x\!\cdot\!\vec x)\;}

(2 pts) Find $k^{\ast}$ , the value of $$k$$ that minimizes $$f(k)$$ . A second derivative test is not necessary.

Solution

From part a), we know that $\dfrac{\text{d}f}{\text{d}k} = -2\vec x\cdot\vec y + 2k(\vec x\cdot\vec x)$ . Let’s set this derivative to $$0$$ and solve for $$k$$ .

\begin{align*} -2\,\vec x\!\cdot\!\vec y + 2k\,(\vec x\!\cdot\!\vec x) &= 0 \\\\ 2k\,(\vec x\!\cdot\!\vec x) &= 2\,\vec x\!\cdot\!\vec y \\\\ k^* &= \frac{\vec x\!\cdot\!\vec y}{\vec x\!\cdot\!\vec x} \end{align*}

This gives the minimizer when $\vec x\cdot\vec x>0$ . If $\vec x\cdot\vec x=0$ , the only possibility is $\vec x=\vec 0=\begin{bmatrix}0\\0\\ \vdots\\ 0\end{bmatrix}$ . In that case,

f(k)=\|\vec y - k\vec 0\|^2=\|\vec y\|^2

which does not depend on $$k$$ . Thus every value of $$k$$ yields the same objective value, and any $$k$$ minimizes $$f$$ .

\boxed{\,k^*=\dfrac{\vec x\!\cdot\!\vec y}{\vec x\!\cdot\!\vec x}\ \text{ when }\ \vec x\neq \vec 0\,}

(3 pts) Show that the vectors $k^{\ast} \vec x$ and $\vec y - k^{\ast} \vec x$ are orthogonal.

Solution

From part b), $k^{\ast}=\dfrac{\vec x\cdot\vec y}{\vec x\cdot\vec x}$ when $\vec x\neq \vec 0$ . To check whether two vectors are orthogonal, we compute their dot product; they are orthogonal if and only if the dot product equals $$0$$ .

\begin{align*} (k^* \vec x)\cdot(\vec y - k^* \vec x) &= (k^*\vec x)\cdot \vec y \;-\; (k^*\vec x)\cdot(k^*\vec x) \\\\ &= k^*\,(\vec x\cdot \vec y) \;-\; (k^*)^2\,(\vec x\cdot \vec x) \\\\ &= \frac{\vec x\cdot \vec y}{\vec x\cdot \vec x}\,(\vec x\cdot \vec y) \;-\; \left(\frac{\vec x\cdot \vec y}{\vec x\cdot \vec x}\right)^2 (\vec x\cdot \vec x) \\\\ &= \frac{(\vec x\cdot \vec y)^2}{\vec x\cdot \vec x} \;-\; \frac{(\vec x\cdot \vec y)^2}{\vec x\cdot \vec x} \\\\ &= 0 \end{align*}

Since the dot product is $$0$$ , the vectors $k^{\ast}\vec x$ and $\vec y - k^{\ast}\vec x$ are orthogonal.

Problem 10: Norms (13 pts)

In the Other Norms section of Chapter 3.2, we introduced the concept of vector norms other than the “default” Euclidean norm. Each of those norms describes a different way of measuring the length of a vector — just like how different loss functions described different ways of measuring the error of a prediction.

(3 pts) In this part only, let $\vec v = \begin{bmatrix} 3 \\ -6 \\ 0 \\ 2 \end{bmatrix}$ . Compute $\lVert \vec v \rVert_2$ , $\lVert \vec v \rVert_1$ , and $\lVert \vec v \rVert_\infty$ .

Solution

\begin{align*} \|\vec v\|_2 &= \sqrt{3^2 + (-6)^2 + 0^2 + 2^2} = \sqrt{9 + 36 + 0 + 4} = \sqrt{49} = 7,\\\\ \|\vec v\|_1 &= |3| + |-6| + |0| + |2| = 3 + 6 + 0 + 2 = 11,\\\\ \|\vec v\|_\infty &= \max\{\,|3|,\,|-6|,\,|0|,\,|2|\,\} = \max\{3,6,0,2\} = 6. \end{align*}

(3 pts) In the Parallelogram Law problem, we introduced the following identity:

\lVert \vec{u} + \vec{v} \rVert^2 + \lVert \vec{u} - \vec{v} \rVert^2 = 2\lVert \vec{u} \rVert^2 + 2\lVert \vec{v} \rVert^2

In general, the parallelogram law only holds for the $$L_2$$ norm, not necessarily other norms.

Find a counterexample involving two vectors $\vec{u}$ and $\vec{v}$ such that the parallelogram law does not hold for the $$L_1$$ norm.

Solution

We want to check whether the parallelogram law holds for the $$L_1$$ norm

\|\vec u+\vec v\|_1^2 + \|\vec u-\vec v\|_1^2 \stackrel{?}{=} 2\|\vec u\|_1^2 + 2\|\vec v\|_1^2

Choose

\vec u=\begin{bmatrix}1\\\\0\end{bmatrix} \qquad \vec v=\begin{bmatrix}0\\\\1\end{bmatrix}

Left-hand side

\vec u+\vec v=\begin{bmatrix}1\\\\1\end{bmatrix},\quad \|\vec u+\vec v\|_1=|1|+|1|=2,\quad \|\vec u+\vec v\|_1^2=2^2=4

\vec u-\vec v=\begin{bmatrix}1\\\\-1\end{bmatrix},\quad \|\vec u-\vec v\|_1=|1|+|-1|=2,\quad \|\vec u-\vec v\|_1^2=2^2=4

\|\vec u+\vec v\|_1^2+\|\vec u-\vec v\|_1^2=4+4=8

Right-hand side

\|\vec u\|_1=|1|+|0|=1,\quad \|\vec v\|_1=|0|+|1|=1

2\|\vec u\|_1^2+2\|\vec v\|_1^2 = 2\cdot 1^2 + 2\cdot 1^2 = 2+2=4

Since $8\neq 4$ , the parallelogram law does not hold for the $$L_1$$ norm with this choice of $\vec u,\vec v$

(3 pts) Prove that

\lVert \vec v \rVert_2 \leq \sqrt{n}\lVert \vec v \rVert_\infty

Hint: Start by writing out the definition of the $$L_2$$ norm, and then square it to remove the square root. You will have a sum of $$n$$ terms. Explain why each of those $$n$$ terms is less than or equal to $\lVert \vec v \rVert_\infty^2$ . This is most of the way to the proof, but there’s still some work you’ll need to do after you get to that point.

Solution

Start from the definition of the $$L_2$$ norm

\begin{align*} \|\vec v\|_2^2 &= \sum_{i=1}^n v_i^2 \end{align*}

For each index $$i$$ , by the definition of the $L_\infty$ norm we have $|v_i|\le |\vec v|_\infty$

\begin{align*} v_i^2 \le \|\vec v\|_\infty^2 \quad \text{for each } i \end{align*}

Sum these inequalities over $i=1,\dots,n$

\begin{align*} \sum_{i=1}^n v_i^2 \le \sum_{i=1}^n \|\vec v\|_\infty^2 = n\,\|\vec v\|_\infty^2 \end{align*}

Thus

\begin{align*} \|\vec v\|_2^2 \le n\,\|\vec v\|_\infty^2 \end{align*}

Both sides are nonnegative, so taking square roots preserves the inequality

\|\vec v\|_2 \le \sqrt{n}\,\|\vec v\|_\infty

(4 pts) Prove that

\lVert \vec v \rVert_2 \leq \lVert \vec v \rVert_1

Hint: Start with the fact that

\lVert \vec v \rVert_1^2 = \left( |v_1| + |v_2| + \cdots + |v_n| \right)^2

Solution

As the hint implies, let’s start by expanding the definition of $|\vec v|_1^2$ . We write the $$L_1$$ norm as one square to avoid carrying a square root

\|\vec v\|_1^2 = \big(|v_1| + |v_2| + \cdots + |v_n|\big)^2

For intuition, recall the perspective from part (c): by definition, the $L_\infty$ norm is the largest absolute value among the components, so $|\vec v|_\infty \ge |v_i|$ for every $$i$$ , and this remains true after squaring because both sides are nonnegative. Here, we will instead use the hint and expand the $$L_1$$ square into “square” terms and “cross” terms

\|\vec v\|_1^2 = \underbrace{|v_1|^2 + |v_2|^2 + \cdots + |v_n|^2}_{\text{squares}} \;+\; \underbrace{2\big(|v_1||v_2| + |v_1||v_3| + \cdots + |v_{n-1}||v_n|\big)}_{\text{cross terms}}

Each cross term $$|v_i||v_j|$$ is nonnegative, so the cross–term sum shown above is $\ge 0$

\|\vec v\|_1^2 \;\ge\; |v_1|^2 + |v_2|^2 + \cdots + |v_n|^2

The right–hand side is exactly the square of the $$L_2$$ norm

|v_1|^2 + \cdots + |v_n|^2 \;=\; \|\vec v\|_2^2

Therefore

\|\vec v\|_1^2 \;\ge\; \|\vec v\|_2^2

Both sides are nonnegative, so taking square roots preserves the inequality

\|\vec v\|_2 \;\le\; \|\vec v\|_1

Problem 11: Neighbors (10 pts)

This problem involves writing code and submitting it to the Pensive autograder.

There are two ways to access the supplemental Jupyter Notebook:

Option 1 (preferred): Set up a Jupyter Notebook environment locally, use git to clone our course repository, and open homeworks/hw03/hw03.ipynb. For instructions on how to do this, see the Environment Setup page of the course website.
Option 2: Click here to open hw03.ipynb on DataHub. Before doing so, read the instructions on the Environment Setup page on how to use the DataHub.

To receive credit for the programming portion of the homework, you’ll need to submit your completed notebook to the autograder on Pensive. Your submission time for Homework 3 is the latter of your PDF and code submission times.