Sure, you could build function factories for the fun.ymin and fun.ymax arguments that only give the value if the input length exceeds some number. The only downside is though that it has to return some value, so you'll get a warning about that.
library(ggplot2)
nmax <- function(n) {
n <- force(n)
function(x) {
if (length(x) >= n) {
max(x)
} else {
NA_real_
}
}
}
nmin <- function(n) {
n <- force(n)
function(x) {
if (length(x) >= n) {
min(x)
} else {
NA_real_
}
}
}
d <- qplot(cyl, mpg, data=mtcars)
d + stat_summary(fun.ymin = nmin(8), fun.ymax = nmax(8), geom = "errorbar")
#> Warning: Removed 1 rows containing missing values (geom_errorbar).

Created on 2020-05-06 by the reprex package (v0.3.0)
If you find that specifying N two times is too cumbersome / error prone, you can also make regular functions that take n as an argument and pass fun.args:
nmax2 <- function(x, n) {
if (length(x) >= n) {
max(x)
} else {
NA_real_
}
}
nmin2 <- function(x, n) {
if (length(x) >= n) {
min(x)
} else {
NA_real_
}
}
d <- qplot(cyl, mpg, data=mtcars)
d + stat_summary(fun.ymin = nmin2, fun.ymax = nmax2, fun.args = list(n = 8), geom = "errorbar")