overflow: hidden will not work because you're not placing the messages inside the container at all. Instead, you're stacking them with absolute positioning and calculated bottom-margin as its sibling. With your current markup, there's no way to cut off the messages at top, unless you also want to calculate that by hand.
Also, your question is not very clear: do you want to stop adding messages once it's outside the container or you just want to crop them off the screen (but they'd still be visible in the DOM).
Either way, your current structure can be much improved if you create it the following way:
<div id=chat>
<input type=text id=chat-message>
<div class=message>First message</div>
<div class=message>Second message</div>
<div class=message>Third message</div>
</div>
This way, all you have to do to make them appear like you want it to use Flexbox:
#chat {
display: flex;
flex-direction: column-reverse; /* from bottom to top */
}
Your JavaScript will also be much simpler, since you'll just be creating a new element as the last child of #chat, meaning no additional calculations. (You can still remove the oldest messages after a certain amount of messages have appeared on the screen).