Statistical Methods for Data Science
DATA7202
Semester 1, 2024
Assignment 1 (Weight: 25%)
Assignment 1 is due on Wednesday 20.03.24 16:00).
Please answer the questions below. For theoretical questions, you should present rigorous proofs
and appropriate explanations. Your report should be visually appealing and all questions should
be answered in the order of their appearance. For programming questions, you should present your
1. [15 Marks] Repeat the advertisement exercise with the following changes.
(a) The data is generated via the following data generation mechanism: Xi ∼ Gamma(1, 1)
for i ∈ {1, 2, 3}; here Gamma(1, 1) stands for the continuous Gamma distribution with
both scale and shape parameters equal to 1.
(b) In addition, the model for y is as follow:
Y = 0.5X1 + 3X2 + 5X3 + 5X2X3 + 2X1X2X3 +W, (1)
where W ∼ N(0, σ2) where σ = 2.
Similar to the original example, generate train and test sets of size N = 1000. Fit the linear re-
gression and the random forest models to the data. For the linear regression, make an inference
about the coefficients, specifically, comment about the contributions of different advertisement
types to sales. Use the linear model and the RF (with 500 trees), to make a prediction (using
the test set), and report the corresponding mean squared errors.
When constructing datasets, please use “1” and “2” seeds for the train and the test sets,
respectively.
2. [10 Marks] Consider a function
f(x) = 3 + x2 − 2sin(x) 1 ⩽ x ⩽ 8.
Write a Crude Monte Carlo algorithm for the estimation of
ℓ =
∫ 8
1
f(x) dx,
using N = 10000 sample size. Deliver the 95% confidence interval. Compare the obtained
estimation with the true value ℓ.
1
3. [10 Marks] Consider the following variant of the cross-validation procedure.
(i) Using the available data, find a subset of “good” predictors that show correlation with
the response variable.
(ii) Using these predictors, construct a model (for regression or classification).
(iii) Use cross-validation to estimate the model prediction error.
Is this a good method? Do you expect to obtain the true prediction error? Explain your answer.
Please note that no coding is required here and one paragraph general answer is sufficient.
4. [5 Marks] Suppose that we observe X1, . . . , Xn ∼ F . We model F as a Gamma distribution
with shape parameter α > 0 and rate parameter β > 0. For this problem, determine the
hypothesis class
H = {f(x, θ); θ ∈ Θ}.
and state explicitly what is θ and Θ.
5. [15 Marks] LetH be a class of binary classifiers over a set Z. Let D be an unknown distribution
over X , and let g be a target hypothesis in H. Show that the expected value of LossT (g) over
the choice of T equals LossD(g), namely,
ET LossT (g) = LossD(g).
6. [15 Marks (see details below)] Consider the following dataset.
x1 y
0 1
1 2
2 3
3 2
4 1
Now, suppose that we would like to consider two models.
Model1 : y = β0 + ε,
and
Model2 : y = β1x1 + ε,
where ε ∼ N(0, 1). That is, we consider two linear models Model1 is the constant model and
Model2 is a regular linear model without the intercept.
(a) [5 Marks)] Fit these models tot the data and write the corresponding coefficients. Namely,
fill the following table:
Model β0 β1
Model1 0
Model2
(b) [5 Marks)] Consider the squared error loss, the absolute error loss, and the L1.5 loss. Find
the average loss for each model. Namely, fill the following table:
Model squared error loss absolute error loss L1.5 loss
Model1
Model2
2
(c) [5 Marks)] Draw a conclusion from the obtained results.
7. [30 Marks (see details below)] Consider as data-set given in data.csv. Our objective is to
predict a response variable y via a linear model. The data contains 4 explanatory variables
(x1, x2, x3, x4). Here, x1, x3 and x4 are numeric variables and x2 is a categorical variable.
(a) [10 Marks)] Load the data-set and replace all categorical values (x2) with numbers.
You do not know if x2 contains an ordinary categorical variable or not, unfortunately.
Therefore, you decide to create two datasets, where in the first dataset, x2 is considered
to be ordinary, and, in the second dataset, x2 is assumed to be unordered.
(b) [20 Marks)] Fit linear regression and report 10-Fold Cross-Validation mean squared errors
for to datasets. What is your conclusion about x2, is it ordered or not?
3
