1) nondecreasing Assuming the times are non-decreasing and that each time is less than 24 hours from the prior time we can determine the day of each time by adding 1 every time we encounter an hour that is less than the prior hour. Add 24 times the day to hour giving hours2 which is the total number of hours since hour 0. Finally take the mean or median modulo 24 to ensure it is in the interval [0, 24) .
hours <- c(20, 21, 22, 23 , 0, 1, 2, 3, 4)
day <- cumsum(c(0, diff(hours) < 0))
hours2 <- hours + 24 * day
mean(hours2) %% 24
## [1] 0
median(hours2) %% 24
## [1] 0
2) circular In this alternative we map the times to a circle and use mean.circular and median.circular from the circular package. More information on that package is available in its help files as well at
Answering biological questions using circular data and analysis in R
library(circular)
hours <- c(20, 21, 22, 23 , 0, 1, 2, 3, 4)
hours.circ <- circular(hours, template = "clock24", units = "hours")
mean.circ <- mean(hours.circ)
as.numeric(mean.circ) %% 24
## [1] 0
median.circ <- median(hours.circ)
as.numeric(median.circ) %% 24
## [1] 0
plot(hours.circ)
points(mean.circ, col = "red", cex = 3)
points(median.circ, col = "blue", cex = 2)
[continued after graph]

Note
You may also find it useful to try the above with a more asymmetric input.
hours <- c(20, 21, 22, 23 , 12)