Academic project · University of Aberdeen · September 2026
Regularised Regression & Model Selection
Comparing least squares, Ridge, and Lasso through cross-validation, with fold-wise preprocessing, hyperparameter selection, held-out evaluation, and interpretation of model coefficients.
- Regression models
- 3
- Validation folds
- 10
- Held-out test records
- 30
The problem
Predict a log-transformed cancer antigen measurement from eight clinical predictors in a 97-record dataset. I investigated how L1 and L2 regularisation affected validation error, selected a model using the training partition, and interpreted the fitted coefficients within that model.
My contribution
- Preserved the supplied 67-record training and 30-record test split, excluded the record index and split indicator from predictors, and checked for missing values.
- Implemented an explicit cross-validation loop for ordinary least squares, Ridge, and Lasso. For regularised models, scaling parameters were fitted inside each training fold and reused for its validation fold.
- Compared regularisation strengths, refitted the selected Ridge model on the complete training partition, evaluated held-out mean squared error, and examined standardised coefficients.
Approach
Keep model selection within training data
Use the supplied partition: 67 training records and 30 test records. The target is levelCancerAntigen, a log-transformed measurement; the remaining eight clinical variables are predictors.
Fit preprocessing within each fold
Use the same shuffled 10-fold splits with seed 42 for model comparisons. For Ridge and Lasso, compute each predictor's mean and standard deviation using that fold's training subset, then apply those values to its validation subset.
Compare regularisation strengths
Evaluate 100 logarithmically spaced alpha values from 0.0001 to 10,000 for Ridge and Lasso. Select each alpha by its mean cross-validation MSE and compare the resulting models with unpenalised least squares.
Refit and evaluate the selected model
Refit Ridge on all 67 training records using the selected alpha and training-only scaling parameters. Apply the same transformation to the held-out records, calculate test MSE, and inspect the fitted coefficients.
Results & findings
Ridge had the lowest mean cross-validation MSE and was selected with alpha approximately 3.3516. Its held-out test MSE was 0.4994 on 30 records. Ridge and Lasso were close in validation performance, so the result supports this selection rule without establishing a reliable superiority claim.
| Model | Selected alpha | Mean CV MSE | Fold-MSE SD |
|---|---|---|---|
| Ordinary least squares | — | 0.543844 | 0.376483 |
| Ridge regression | 3.3516 | 0.535699 | 0.362882 |
| Lasso regression | 0.0126 | 0.536008 | 0.365087 |
Lower MSE is better. Errors are measured on the log-transformed target. Fold-MSE SD describes variation across the ten validation folds; it is not a confidence interval. The held-out test result is for the selected Ridge model.
- Ridge and Lasso improved mean cross-validation MSE slightly over least squares. Their mean errors differed by about 0.000308, much less than the variation across folds.
- The regularisation curves show that larger penalties eventually increased validation error. The experiment selected alpha using those curves rather than assuming a stronger penalty would always help.
- logCancerVol had the largest positive standardised coefficient in the fitted Ridge model, approximately 0.6351. This describes its conditional association with the predicted target in this model.
Experiments & analysis
Select a figure to view it at full size.
Inspect fold-wise preprocessing and validation
The helper fits each model inside a validation loop. Ridge and Lasso use scaling parameters from the training fold only. The call below selects ten folds.
Python / scikit-learn · Source excerpt
def train_cv(X, y, model_type = 'linear', alpha = 1.0, n_splits=5, shuffle=True, random_state=42):
k_fold = KFold(n_splits=n_splits, shuffle=shuffle,
random_state=random_state if shuffle else None)
MSE = []
for train_index, val_index in k_fold.split(X):
X_train, X_val = X[train_index], X[val_index]
y_train, y_val = y[train_index], y[val_index]
if model_type == 'linear':
reg_model = linear_model.LinearRegression()
elif model_type == 'ridge':
reg_model = linear_model.Ridge(alpha=alpha)
elif model_type == 'lasso':
reg_model = linear_model.Lasso(alpha=alpha, max_iter=10000)
else:
raise ValueError('Invalid model type.')
if model_type in ['ridge', 'lasso']:
mean = X_train.mean(axis=0)
std = X_train.std(axis=0)
X_train = standardise_data(X_train, mean, std)
X_val = standardise_data(X_val, mean, std)
reg_model.fit(X_train, y_train)
y_val_pred = reg_model.predict(X_val)
MSE.append(mean_squared_error(y_val, y_val_pred))
return MSE
linear_mse = train_cv(X_train, Y_train, model_type='linear', n_splits=10)Limitations & what I learned
- The dataset has 97 records, with 30 reserved for testing. The reported errors may change substantially with different samples or splits.
- The same cross-validation results are used to select alpha and compare models. Nested or repeated cross-validation would provide a stronger assessment of the complete selection procedure.
- Coefficient magnitudes depend on regularisation and relationships among predictors. They describe this fitted model and do not establish causal effects or generalisation to other populations.
Where I would take this next
- Use repeated or nested cross-validation to assess selection stability and report uncertainty across splits.
- Inspect residual patterns and coefficient stability, and evaluate the selected approach on additional held-out data.