SOMAXCONN is the maximum size of the back log. The back log is the number of requests that will be queued up and wait for processing by a TCP server application.
For example, if you code this:
It will queue up to 5 connections. The number 5 in the second parameter is referred to as the "back log". So my listen() call allows for a back log of 5 connections. However, the back log number passed to listen() can never be higher than the SOMAXCONN value. So SOMAXCONN is the "maximum" number of backlogged connections.
It's important to understand that each backlogged connection uses up memory in your system. So if you allowed a million backlogged connections, and each one uses a minimum of 1 KB of memory, that's a gigabyte of memory that could potentially be used. (1 KB is the minimum each backlogged connection would use. There may be more, for example, if a receive buffer is used on the socket.) Someone hitting you with a denial-of-service attack can therefore force your computer to use a lot of extra memory, slowing you down (or potentially even crashing the server.) Therefore, a maximum needed to be set system-wide so sysadmins can protect themselves from DOS attacks like htis.
I'm not 100% sure what happens when there are more connections than you allowed. In my example, if you had more than 5, backlogged connections, what would the 6th attempt get? I think they might get a code like "Connection Refused" or possibly "Connection Reset". I experimented with this many years ago, back then I could've told you what would happen...but as of right now, I can't remember. But, it'll be something like that.
And if you have a firewall that blocks ICMP messages (which, at least at one time, were very popular) the caller would just notice the connection attempt "hanging" until they hit their timeout value. Which may sound like bad behavior, but... if it's an attacker trying to flood you with connection attempts, that actually would be a desired response since it'd slow the attacker down to a crawl.