due by the end of your lab section on Wednesday, September 9th, 2026
Each lab worksheet will contain several activities, some of which will involve writing code and others that will involve writing math on paper. To receive credit for a lab, you must complete all required activities and show your lab TA by the end of the lab section.
While you must get checked off by your lab TA individually, we encourage you to form groups with 1-2 other students to complete the activities together.
In Chapter 1.3, we introduced the three-step modeling recipe for finding optimal model parameters, which ultimately helps us make the best possible predictions.
Choose a model.
Choose a loss function.
Minimize average loss (also called empirical risk) to find optimal model parameters.
Constant model, squared loss: \(\displaystyle R_{\text{sq}}(w) = \frac{1}{n} \sum_{i=1}^n (y_i - w)^2 \implies w^{\ast} = \bar{y}\)
Constant model, absolute loss: \(\displaystyle R_{\text{abs}}(w) = \frac{1}{n} \sum_{i=1}^n |y_i - w| \implies w^{\ast} = \text{Median}(y_1, y_2, \ldots, y_n)\)
Simple linear regression model, squared loss:
Suppose we’d like to find the optimal parameter, \(w^{\ast}\), for the constant model \(h(x_i) = w\). To do so, we use the following loss function, called the relative squared loss:
What value of \(w\) minimizes the average loss (i.e. empirical risk) when using the relative squared loss function – that is, what is \(w^{\ast}\)? Your answer should only be in terms of the variables \(n, y_1, y_2, \ldots, y_n\), and any constants.
Since \(h(x_i) = w\) for the constant model, relative squared loss for the constant model is:
and so average relative squared loss for the constant model is:
To find the value of \(w\) that minimizes \(R_{\text{rsq}}(w)\), we’ll first find its first derivative and set it to zero. The first derivative of \(R_{\text{rsq}}(w)\) is:
At this point, it’ll be useful to step aside and find the derivative of \(L_{\text{rsq}}(y_i, w)\) with respect to \(w\), as this is the expression being summed. The derivative of \(L_{\text{rsq}}(y_i, w)\) with respect to \(w\) is:
Back to \(\frac{\text{d}}{\text{d} w}R_{\text{rsq}}(w)\), we have:
Setting this equal to 0 yields:
This is known as the harmonic mean of \(y_1, y_2, …, y_n\).
Consider a dataset of \(n\) integers, \(y_1, y_2, \ldots, y_n\), whose histogram is given below:
Which of the following is closest to the constant prediction \(w^{\ast}\) that minimizes:
\(30\).
The minimizer of average 0-1 loss is the mode.
Which of the following is closest to the constant prediction \(w^{\ast}\) that minimizes:
\(7\).
The minimizer of average absolute loss is the median. The outliers near \(30\) shift it from \(6\) to \(7\).
Which of the following is closest to the constant prediction \(w^{\ast}\) that minimizes:
\(11\).
The minimizer of average squared loss is the mean, pulled upward by the heavy right tail, so it’s above the median (\(7\)) and closest to \(11\).
Which of the following is closest to the constant prediction \(w^{\ast}\) that minimizes:
Hint: Think about the effect of outliers.
\(15\).
As \(p \to \infty\), the minimizer is the midrange, halfway between min and max.
Consider a dataset of 8 points, \(y_1, y_2, \ldots, y_8\) that are in sorted order, i.e. \(y_1 < y_2 < \ldots < y_8\).
Recall that mean absolute error, \(R_{\text{abs}}(w)\), for the constant model \(h(x_i) = w\) is defined as:
This is a piecewise linear function that changes slope at each data point. The slope of \(R_{\text{abs}}(w)\) at any \(w\) that is not a data point is:
Suppose that \(y_4=10\), \(y_5=14\), \(y_6=22\), and \(R_{\text{abs}}(11)=9\). What is \(R_{\text{abs}}(22)\)?
Hint: You don’t have all 8 of the \(y\)-values, so you can’t find \(R_\text{abs}(22)\) just by plugging in numbers into the formula for \(R_\text{abs}(w)\). Instead, think about how to use the slope formula.
\(R_{\text{abs}}(22)=11\).
We can write the points given to us as:
Since there are an even number of data points (\(n=8\)), the minimizer of absolute error is not a single point but the entire interval between the two middle points. Here, the middle two are \(10\) and \(14\), so every \(w \in [10,14]\) minimizes \(R_{\text{abs}}(w)\). This explains why the error is flat inside that interval: the number of points on the left equals the number on the right, so shifting \(w\) around does not change the error. As a result, \(R_{\text{abs}}(11)=9\) and \(R_{\text{abs}}(14)=9\).
Once we move beyond \(14\), the balance breaks. There are now five points to the left and only three to the right, so the slope of \(R_{\text{abs}}(w)\) becomes positive. The slope formula tells us:
so for any \(w \in (14,22)\) we have
This means that for every one unit we move to the right of \(w=14\), the error increases by \(\tfrac{1}{4}\). Moving from \(w=14\) to \(w=22\) is a distance of \(22-14=8\) units, so the error increases by
Adding this to the baseline error of \(R_{\text{abs}}(14)=9\), we get:
Complete the tasks in the lab02.ipynb notebook.
There are two ways to access the supplemental Jupyter Notebook:
Option 1 (preferred): Set up a Jupyter Notebook environment locally, use git to clone our course repository, and open labs/lab02/lab02.ipynb. For instructions on how to do this, see the Environment Setup page of the course website.
Option 2: Click here to open lab02.ipynb on DataHub. Before doing so, read the instructions on the Environment Setup page on how to use the DataHub.
To receive credit for Activity 4, you’ll need to show your lab TA that all test cases have passed.
Suppose we want to fit a simple linear model (using squared loss) that predicts the number of ingredients in a product given its price. We’re given that:
The average cost of a product in our dataset is $40, i.e. \(\bar x=40\)
The average number of ingredients in a product in our dataset is 15, i.e. \(\bar y =15\)
The intercept and slope of the regression line are \(w_0^{\ast}=11\) and \(w_1^{\ast}=\frac{1}{10}\), respectively.
Suppose Victors’ Veil (a skincare product) costs $40 and has 11 ingredients. What is the squared loss of our model’s predicted number of ingredients for Victors’ Veil?
Using the equation of the regression model we have seen in class:
Plugging in \(w_0^{\ast}=11\), \(w_1^{\ast}=\frac{1}{10}\), and \(x=40\) gives us:
The squared loss is \(L=(y_i-h(x_i))^2\), substituting \(y=11\) (actual) and \(h(x_i)=15\) (predicted) gives us:
Is it possible to answer part a) above just by knowing \(\bar x\) and \(\bar y\), i.e. without knowing the values of \(w_0^{\ast}\) and \(w_1^{\ast}\)? Once you select an answer, explain it to your peers.
Hint: If you’re stuck, look through the headings of the sections of Chapter 2.3.
Yes, the values of \(w_0^{\ast}\) and \(w_1^{\ast}\) don’t impact the answer to part a).
The simple linear model minimizing mean squared error will always go through the point \((\bar x, \bar y)\). We’re given \(\bar x=40\) and \(\bar y=15\), meaning that for a product that costs $40 we will predict that it has 15 ingredients, no matter what the slope and intercept end up being.
Suppose we’d like to predict the number of minutes a delivery will take, \(y\), as a function of distance, \(x\). To do so, we look to our dataset of \(n\) deliveries, \((x_1, y_1), (x_2,y_2), \dots, (x_n,y_n)\), and fit two simple linear models:
\(F(x_i)=a_0+a_1x_i\), where:
Here, \(\bar{x}\) and \(\bar{y}\) are the respective means of the \(x\)-values and \(y\)-values.
\(G(x_i)=b_0+b_1x_i\), where \(b_0\) and \(b_1\) are chosen such that \(G(x_i)=b_0+b_1x_i\) minimizes mean absolute error on the dataset. Assume that no other line minimizes mean absolute error on the dataset, i.e. that the values of \(b_0\) and \(b_1\) are unique. Assume also that \(F\) and \(G\) are distinct lines.
Fill in the \(\boxed{???}\):
The quantity on the left hand side is the total squared error of model \(F\). By definition, \(F\), is the line that minimizes MSE over all possible linear models.
The quantity on the right hand side is the total squared error of model \(G\). However, \(G\) is optimized for mean absolute error, not MSE.
The question tells us that \(F\) and \(G\) are different, so \(\displaystyle \sum_{i=1}^{n}(y_i-F(x_i))^2 < \sum_{i=1}^{n}(y_i-G(x_i))^2\) must be true.
Fill in the \(\boxed{???}\):
The quantity on the left hand side is the squared total absolute error of model \(F\).
The quantity on the right hand side is the squared total absolute error of model \(G\). By definition, \(G\), is the line that minimizes MAE over all possible linear models.
The total absolute error of \(G\) must be less than the total absolute error of \(F\), and squaring them doesn’t change that relationship because both totals are guaranteed to be positive numbers (sums of absolute values).
Finally, \(F\) and \(G\) are different, so \(\displaystyle \left(\sum_{i=1}^{n}|y_i-F(x_i)|\right)^2 > \left(\sum_{i=1}^{n}|y_i-G(x_i)|\right)^2\)
Below, we’ve drawn the lines for both \(F\) and \(G\) along with a scatter plot for the original \(n\) deliveries:
Which line corresponds to \(F\)?
Line 1
The key idea is that models trained with squared loss (MSE) are more sensitive to outliers than models trained with absolute loss (MAE).
Since Line 1 appears to be “pulled up” more strongly by an outlier, it suggests that this line was influenced more heavily by extreme values. This behavior aligns with how MSE-based regression works: outliers have a greater impact on the overall loss because squaring the errors makes large deviations even more significant.
In contrast, MAE-based regression (Line 2) is less sensitive to outliers because absolute differences do not grow as quickly.
Therefore, Line 1 corresponds to \(F\), the MSE-minimizing line.
The following are extra practice. Don’t feel pressured to answer all of these problems in lab, but make sure to attempt them at some point.
Recall the formula for relative squared loss from Activity 1:
Let \(C(y_1, y_2, …, y_n)\) be your minimizer \(w^{\ast}\) from Activity 1. That is, for a particular dataset \(y_1, y_2, …, y_n\), \(C(y_1, y_2, …, y_n)\) is the value of \(w\) that minimizes empirical risk for relative squared loss on that dataset.
What is the value of \(\displaystyle\lim_{y_4 \rightarrow \infty} C(1, 3, 5, y_4)\) in terms of \(C(1, 3, 5)\)? Your answer should involve the function \(C\) and/or one or more constants.
Hint: To notice the pattern, evaluate \(C(1, 3, 5, 100)\), \(C(1, 3, 5, 10000)\), and \(C(1, 3, 5, 1000000)\).
What is the value of \(\displaystyle\lim_{y_4 \rightarrow 0} C(1, 3, 5, y_4)\)? Again, your answer should involve the function \(C\) and/or one or more constants.
Based on the results of the previous two parts, when is the prediction \(C(y_1, y_2, …, y_n)\) robust to outliers? When is it not robust to outliers?
\(C(y_1, y_2, …, y_n)\) is great at ignoring large outliers. No matter how large you make any particular value, \(C(y_1, y_2, …, y_n)\) is upper-bounded by \(\frac{n}{n-1}\) multiplied by the value of \(C\) applied to all data points excluding the large outlier. This is as opposed to the regular “arithmetic mean”, where if you make a single data point arbitrarily large, the mean also becomes arbitrarily large (i.e. if \(y_n \rightarrow \infty\), then \(\text{Mean}(y_1, y_2, …, y_n) \rightarrow \infty\) too).
However, \(C(y_1, y_2, …, y_n)\) is not robust to small outliers. As a particular data point approaches 0, the value of \(C(y_1, y_2, …, y_n)\) also approaches 0 no matter how large the other data points are.
Consider a dataset of \(y_1, y_2, \dots, y_n\), all of which are positive. We want to fit a constant model, \(h(x_i)=w\), to the data.
Let \(w_p^{\ast}\) be the optimal constant prediction that minimizes average degree-\(p\) loss, \(R_p(w)\), defined below:
For example, \(w_2^{\ast}\) is the optimal constant prediction that minimizes \(R_2(w)= \displaystyle \frac{1}{n} \sum_{i=1}^{n}|y_i-w|^2\)
In each of the parts below, determine the value of the quantity provided. By “the data”, we are referring to \(y_1, y_2, \dots, y_n\). The answer choices are as follows; select one item in each row.
A: The standard deviation of the data
B: The variance of the data
C: The mean of the data
D: The median of the data
E: The midrange of the data, \(\frac{y_\text{min} + y_\text{max}}{2}\)
F: The mode of the data
G: None of these
| A | B | C | D | E | F | G | ||
| (i) | \(w_0^{\ast}\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) |
| (ii) | \(w_1^{\ast}\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) |
| (iii) | \(R_1(w_1^{\ast})\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) |
| (iv) | \(w_2^{\ast}\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) |
| (v) | \(R_2(w_2^{\ast})\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) | \(\bigcirc\) |
(i) \(w_0^{\ast}\) is none of the these. The original intention was to have \(R_0\) be 0-1 loss, in which case \(w_0^{\ast}\) would be the mode.
(ii) \(w_1^{\ast}\) is the median of the data, since \(R_1(w)= \displaystyle \frac{1}{n} \sum_{i=1}^{n}|y_i-w|\)
(iii) \(R_1(w_1^{\ast})\) is the minimum mean absolute error, which is none of these.
(iv) \(w_2^{\ast}\) is the mean of the data, since \(R_2(w)= \displaystyle \frac{1}{n} \sum_{i=1}^{n}|y_i-w|^2\) is equivalent to mean squared error.
(v) \(R_2(w_2^{\ast})\) is the variance of the data, or the minimum mean squared error, shown below:
Now, suppose we want to find the optimal constant prediction, \(w_\text{U}^{\ast}\), using the “Ulta” loss function, defined below:
To find \(w_\text{U}^{\ast}\), we minimize \(R_\text{U}(w)\), the average Ulta loss. How does \(w_\text{U}^{\ast}\) compare to the mean of the data, \(M\)?
Minimizing the average Ulta loss means minimizing the empirical risk:
The minimizer is \(w_\text{U}^{\ast} = \frac{\sum_{i=1}^n y_i^2}{\sum_{i=1}^n y_i}\). Since \(M > 0\), we have \(w_\text{U}^{\ast} - M = \frac{\frac{1}{n}\sum_{i=1}^n (y_i-M)^2}{M} \geq 0\). Thus, \(w_\text{U}^{\ast} \geq M\), with equality exactly when all the \(y_i\) values are equal.
Finally, to find the optimal constant prediction, we will instead minimize regularized average Ulta loss, \(R_\lambda(w)\), defined below:
Here, assume \(\lambda > 0\) is some positive constant. (We will cover regularization in more detail later in the term.)
Find \(w^{\ast}\), the constant prediction that minimizes \(R_\lambda(w)\). Give your answer as an expression in terms of the \(y_i\)’s, \(n\), and/or \(\lambda\).
To minimize the regularized average Ulta loss, we solve for \(w\) by setting \(\displaystyle \frac{\partial R}{\partial w}=0\) and solving for \(w\).
Step 1: Compute the derivative and set to 0.
Step 2: Expand and simplify.
Step 3: Solve for \(w\).
Step 4: Multiply by \(\frac{n}{n}\)