Cross Validation Examples
Nao Mimoto – Class Web Page
1. Load Demo Data
As we did in Ch2-Lab.
2. Separate into Training vs Test
## [1] 41# randomly select rows
set.seed(887)
testSample = sample(1:41, 20)
testSample # rows selected to be the test set## [1] 28 3 4 12 17 36 7 8 22 25 32 2 6 18 27 26 37 1 41 38# separate DF into Training vs Test Set
DFtr = DF[-testSample,] # Training Set
DFte = DF[testSample,] # Test Set
# Visualize
layout(matrix(c(1,2,4,3), 2, 2))
plot(DF, main="Combined", ylim=c(-2500, 2500))
plot(DFtr, main="Training Set", ylim=c(-2500, 2500))
plot(DFte, pch=19, col="red", main="Test Set", ylim=c(-2500, 2500))
layout(1)
3. Fit Training w polynomial
# Set degree of polynomial that you are fitting
deg.poly = 10
# Fit the Training Set
Fit01 <- lm( Y ~ poly(X, deg.poly, raw=TRUE), data=DFtr)
Fit.pred<- predict(Fit01, newdata=DF)
MSE.train <- mean(Fit01$residuals^2) # MSE within training set
# Calculating where it will be on the poly line
# if you use X from the test set
Fit01.pred<- predict(Fit01, newdata=DFte)
MSE.valid <- mean((DFte$Y - Fit01.pred)^2)
# Plot Training and Test Set
layout(matrix(c(1,2), 1, 2))
plot(DFtr, main="Training Set", ylim=c(-2500, 2500))
lines(DF$X, Fit.pred, lwd=2, col="black" )
text(-0,-2000, paste("d=",deg.poly,": training MSE=",round(MSE.train,0)), cex=2)
plot(DFte, pch=19, col="red", main="Test Set", ylim=c(-2500, 2500))
lines(DF$X, Fit.pred, lwd=2, col="black" )
text(-0,-2000, paste("test MSE=",round(MSE.valid,0)), col="red", cex=2)
3b. If you change deg.poly from 1 to 7
# Loop going through deg.poly from 1 to 7
MSE.train <- MSE.valid <- array(0, 7)
for (deg.poly in 1:7) {
Fit01 <- lm( Y ~ poly(X, deg.poly, raw=TRUE), data=DFtr)
#--- CV Training MSE
MSE.train[deg.poly] <- mean(Fit01$residuals^2)
#--- CV Validation MSE
Fit01.pred<- predict(Fit01, newdata=DFte)
MSE.valid[deg.poly] <- mean((DFte$Y - Fit01.pred)^2)
}## [1] 372493.61 270640.06 78913.93 78227.78 78066.07 75849.07 69983.69## [1] 736948.76 1341212.47 65683.25 85992.30 111604.23 640310.64 1943344.56# Plot Average MSE
plot(MSE.train, type="o", ylab="MSE", ylim=c(40000, max(MSE.valid)))
lines(MSE.valid, type="o", col="red")
legend(3, 1200000, lty=1, c("Train MSE","Test MSE"), col=c("black", "red"))
4. Why do we need CV on top of Training vs Testing?
So that’s what happens if we separate the dataset into training vs test, and keep fitting the training set. Then why do we need to make k-fold Cross Validation?
# - (note) Training vs Testing doesn't have to be 50/50
# - If we keep peeking to the test set, we are eventually catoring to the given test set.
# - Some cases, CV is a luxury to have.5. Visualize 5-fold CV procedure
## 1. Load Dataset
## [-----------120------------]
## 2. Choose seed. Randomly assin the data to...
## Training Set vs Test set - Test Set will be set aside
## Training | Test
## [ 100 ][ 20 ]
## 3. Use CV on Training Set (alreday randomized)
## [ 100 ]
## Separate them into
## [20][20][20][20][20]
## f1 f2 f3 f4 f5
## 4. We need to tune a hyperparameter.
## For each value of the hyperparameter, go through CV rounds.
## FIT THE TRAINING SET, then use the model parameter to predict VALIDATION SET
## Training | Validation
## Round1 [f2][f3][f4][f5] -> [f1]
## Round2 [f3][f4][f5][f1] -> [f2]
## Round3 [f4][f5][f1][f2] -> [f3]
## Round4 [f5][f1][f2][f3] -> [f4]
## Round5 [f1][f2][f3][f4] -> [f5]
## At the end, pick the value of hyperparameter with lowest Av. Validation MSE
## (best value of the hyperparameter) = b0
## 5. The Final Fit
## Use the best hyperparameter (b0) and see how it works on the Test set.
## Using b0, FIT THE ENTIRE TRAINING SET. Then predict the Test Set.
## The Test set is used for the first time.
## Training | Test
## [ 100 ] -> [ 20 ]
## Test MSE is the final number that represents the fit.6. 5-fold CV Example
A = c("Ash", "Bri", "Caleb", "David", "Emily",
"Fiona", "Gary", "Hazel", "Ian", "Jocelyn",
"Kai", "Luna", "Milo", "Nova", "Orion",
"Penelope", "Quincy", "Rowan", "Sammy", "Theo",
"Uma", "Victor", "Willow", "Xavi")
B = sample(A, 24)
D = matrix(B, 6, 4)
D## [,1] [,2] [,3] [,4]
## [1,] "Nova" "Sammy" "David" "Uma"
## [2,] "Gary" "Quincy" "Fiona" "Orion"
## [3,] "Caleb" "Ash" "Theo" "Milo"
## [4,] "Hazel" "Penelope" "Xavi" "Victor"
## [5,] "Ian" "Jocelyn" "Emily" "Luna"
## [6,] "Rowan" "Kai" "Willow" "Bri"CODE ONLY
##--------------------------------------
## 1. Load Demo Data
# For generating data. Just for record. Not shown to students.
X = seq(-10,10,.5)
Y = 3 - 100*X + 2*X^2 + 3*X^3 + rnorm(length(X), 0, 200)
plot(Y)
DF = as_tibble(cbind(X, round(Y)))
DF
library(tidyverse)
DF = read_csv("https://nmimoto.github.io/datasets/cvDemoData.csv")
DF
plot(DF)
##--------------------------------------
## 2. Separate into Training vs Test
# number of rows
nrow(DF)
# randomly select rows
set.seed(887)
testSample = sample(1:41, 20)
testSample # rows selected to be the test set
# separate DF into Training vs Test Set
DFtr = DF[-testSample,] # Training Set
DFte = DF[testSample,] # Test Set
# Visualize
layout(matrix(c(1,2,4,3), 2, 2))
plot(DF, main="Combined", ylim=c(-2500, 2500))
plot(DFtr, main="Training Set", ylim=c(-2500, 2500))
plot(DFte, pch=19, col="red", main="Test Set", ylim=c(-2500, 2500))
layout(1)
##--------------------------------------
## 3. Fit Training w polynomial
# Set degree of polynomial that you are fitting
deg.poly = 10
# Fit the Training Set
Fit01 <- lm( Y ~ poly(X, deg.poly, raw=TRUE), data=DFtr)
Fit.pred<- predict(Fit01, newdata=DF)
MSE.train <- mean(Fit01$residuals^2) # MSE within training set
# Calculating where it will be on the poly line
# if you use X from the test set
Fit01.pred<- predict(Fit01, newdata=DFte)
MSE.valid <- mean((DFte$Y - Fit01.pred)^2)
# Plot Training and Test Set
layout(matrix(c(1,2), 1, 2))
plot(DFtr, main="Training Set", ylim=c(-2500, 2500))
lines(DF$X, Fit.pred, lwd=2, col="black" )
text(-0,-2000, paste("d=",deg.poly,": training MSE=",round(MSE.train,0)), cex=2)
plot(DFte, pch=19, col="red", main="Test Set", ylim=c(-2500, 2500))
lines(DF$X, Fit.pred, lwd=2, col="black" )
text(-0,-2000, paste("test MSE=",round(MSE.valid,0)), col="red", cex=2)
layout(1)
##----------
### 3b. If you change deg.poly from 1 to 7
# Loop going through deg.poly from 1 to 7
MSE.train <- MSE.valid <- array(0, 7)
for (deg.poly in 1:7) {
Fit01 <- lm( Y ~ poly(X, deg.poly, raw=TRUE), data=DFtr)
#--- CV Training MSE
MSE.train[deg.poly] <- mean(Fit01$residuals^2)
#--- CV Validation MSE
Fit01.pred<- predict(Fit01, newdata=DFte)
MSE.valid[deg.poly] <- mean((DFte$Y - Fit01.pred)^2)
}
# For deg.poly 1 throgh 7
MSE.train
# For deg.poly 1 throgh 7
MSE.valid
# Plot Average MSE
plot(MSE.train, type="o", ylab="MSE", ylim=c(40000, max(MSE.valid)))
lines(MSE.valid, type="o", col="red")
legend(3, 1200000, lty=1, c("Train MSE","Test MSE"), col=c("black", "red"))
##--------------------------------------
## 4. Why do we need CV on top of Training vs Testing?
# So that's what happens if we separate the dataset into
# training vs test, and keep fitting the training set.
# Then why do we need to make k-fold Cross Validation?
# - (note) Training vs Testing doesn't have to be 50/50
# - If we keep peeking to the test set, we are eventually catoring to the given test set.
# - Some cases, CV is a luxury to have.
##--------------------------------------
## 5. Visualize 5-fold CV procedure
## 1. Load Dataset
## [-----------120------------]
## 2. Choose seed. Randomly assin the data to...
## Training Set vs Test set - Test Set will be set aside
## Training | Test
## [ 100 ][ 20 ]
## 3. Use CV on Training Set (alreday randomized)
## [ 100 ]
## Separate them into
## [20][20][20][20][20]
## f1 f2 f3 f4 f5
## 4. We need to tune a hyperparameter.
## For each value of the hyperparameter, go through CV rounds.
## FIT THE TRAINING SET, then use the model parameter to predict VALIDATION SET
## Training | Validation
## Round1 [f2][f3][f4][f5] -> [f1]
## Round2 [f3][f4][f5][f1] -> [f2]
## Round3 [f4][f5][f1][f2] -> [f3]
## Round4 [f5][f1][f2][f3] -> [f4]
## Round5 [f1][f2][f3][f4] -> [f5]
## At the end, pick the value of hyperparameter with lowest Av. Validation MSE
## (best value of the hyperparameter) = b0
## 5. The Final Fit
## Use the best hyperparameter (b0) and see how it works on the Test set.
## Using b0, FIT THE ENTIRE TRAINING SET. Then predict the Test Set.
## The Test set is used for the first time.
## Training | Test
## [ 100 ] -> [ 20 ]
## Test MSE is the final number that represents the fit.
##--------------------------------------
## 6. 5-fold CV Example
A = c("Ash", "Bri", "Caleb", "David", "Emily",
"Fiona", "Gary", "Hazel", "Ian", "Jocelyn",
"Kai", "Luna", "Milo", "Nova", "Orion",
"Penelope", "Quincy", "Rowan", "Sammy", "Theo",
"Uma", "Victor", "Willow", "Xavi")
B = sample(A, 24)
D = matrix(B, 6, 4)
D