Adding CSS file for IE

Internet Explorer up to and including version 9 does support conditional comments like those:

<!--[if IE]><link rel="stylesheet" type="text/css" href="/Style Library/custom.css"></link><![endif]-->

<SharePoint:CssRegistration Name="/Style Library/custom.css" ConditionalExpression="IE" runat="server"></SharePoint:CssRegistration>

However, Internet Explorer 10 and above has dropped support for those conditional comments hence you are not able to target CSS in an easy way to IE 10+.

Update

If you need to target IE 10+ don't use JavaScript to sniff the Web browser user agent as the accepted answer of the link @Tiago provided does.

Instead either use pure CSS like this:

@media screen and (-ms-high-contrast: active), (-ms-high-contrast: none)
{
   /* IE10 and IE11-specific styles go here */
}
@media screen and (min-width:0\0)
{
    /* IE9 and IE10 rule sets go here */
}

or use JavaScript feature detection, like checking for the existence of document.documentMode and document.documentMode===10 or document.documentMode===11 like this:

if (document.documentMode != undefined && document.documentMode===10)
{
  // This is IE 10
}
if (document.documentMode != undefined && document.documentMode===11)
{
  // This is IE 11
}

Having said that, it all depends heavily on the SharePoint version you are using (SP2010 puts IE 9/10/11+ into 8-mode, SP2013 puts IE 11+ into 10-mode) and if you are running with the out-of-the-box masterpages (e.g. seattle.master on SP2013) or if you have customized your masterpage and modified the X-UA-Compatible meta tag (which I personally recommend against heavily as it might break standard SP functionality).