I had the same issue, field validation worked well, can display xx is required below the input window. But the validation-summary didn't show up.
At first, I have code below in my cshtml file,
<div class="row">
<div asp-validation-summary="ModelOnly"></div>
@Html.ValidationSummary(false, "", new { @class = "text-danger" })
</div>
and I had code below in my controller,
ModelState.AddModelError("key", "validation summary error");
return View(rdl);
then only @Html.ValidationSummary() worked, and asp-validation-summary didn't show up(if it displayed message, it should have black font)

Then I tried asp-validation-summary="All", it worked. The difference between ALL and ModelOnly is ModelOnly will only display the invalid message for the model.

So I use ModelState.AddModelError("", "validation summary error."); in my Controller to make the <div asp-validation-summary="ModelOnly"></div> display error message.
I only know it can make it display correctly, but I don't know why...
display: none !important;on any element that has class.validation-summary-errors. This is a valid guess for example for why validation is working at field levels but not at the summary level LOL.<span asp-validation-for=""></span>and put them on top of the form. That way it shows the errors when a field is changed. Now you might want a more solid answer than this. If you could ask a separate question, I might be able to work on it and give you another answer. I don't think your problem is the same as this original asker had.