SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

Question

I tried to install Oracle Entitlements Server Client. When I call

config.cmd -smConfigId Sample-SM -prpFileName C:\oracle\product\11.1.2\as_1\oessm\SMConfigTool\smconfig.java.controlled.prp

I got this Exception:

    javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.ssl.Alerts.getSSLException(Alerts.java:192)
        at sun.security.ssl.SSLSocketImpl.fatal(SSLSocketImpl.java:1884)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:276)
        at sun.security.ssl.Handshaker.fatalSE(Handshaker.java:270)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1341)
        at sun.security.ssl.ClientHandshaker.processMessage(ClientHandshaker.java:153)
        at sun.security.ssl.Handshaker.processLoop(Handshaker.java:868)
        at sun.security.ssl.Handshaker.process_record(Handshaker.java:804)
        at sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1016)
        at sun.security.ssl.SSLSocketImpl.performInitialHandshake(SSLSocketImpl.java:1312)
        at sun.security.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:702)
        at sun.security.ssl.AppOutputStream.write(AppOutputStream.java:122)
        at java.io.OutputStream.write(OutputStream.java:75)
        at oracle.security.oes.enroll.EnrollmentClient.writeToSocket(EnrollmentClient.java:330)
        at oracle.security.oes.enroll.EnrollmentClient.enroll(EnrollmentClient.java:161)
        at oracle.security.oes.enroll.EnrollmentClient.main(EnrollmentClient.java:478)
        at oracle.security.oes.tools.EnrollmentTool.doEnroll(EnrollmentTool.java:103)
        at oracle.security.oes.tools.SMConfigTool.doEnrollment(SMConfigTool.java:1192)
        at oracle.security.oes.tools.SMConfigTool.run(SMConfigTool.java:617)
        at oracle.security.oes.tools.SMConfigTool.main(SMConfigTool.java:546)
    Caused by: sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:350)
        at sun.security.validator.PKIXValidator.engineValidate(PKIXValidator.java:260)
        at sun.security.validator.Validator.validate(Validator.java:260)
        at sun.security.ssl.X509TrustManagerImpl.validate(X509TrustManagerImpl.java:326)
        at sun.security.ssl.X509TrustManagerImpl.checkTrusted(X509TrustManagerImpl.java:231)
        at sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(X509TrustManagerImpl.java:126)
        at sun.security.ssl.ClientHandshaker.serverCertificate(ClientHandshaker.java:1323)
        ... 15 more
    Caused by: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA
        at sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(PKIXMasterCertPathValidator.java:159)
        at sun.security.provider.certpath.PKIXCertPathValidator.doValidate(PKIXCertPathValidator.java:351)
        at sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(PKIXCertPathValidator.java:191)
        at java.security.cert.CertPathValidator.validate(CertPathValidator.java:279)
        at sun.security.validator.PKIXValidator.doValidate(PKIXValidator.java:345)
        ... 21 more
sun.security.validator.ValidatorException: PKIX path validation failed: java.security.cert.CertPathValidatorException: Algorithm constraints check failed: MD5withRSA

Can you help me to find a reason?

You might want to decrease the key size for RSA (from the default of 1024, I think, to 256, for example). — keyser
– keyser, Commented Jan 19, 2014 at 14:51
"Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted." java.com/en/download/faq/release_changes.xml . I tried change Java 7 to Java 6 and it worked. — egorlitvinenko
– egorlitvinenko, Commented Jan 19, 2014 at 15:08
How to check key length: serverfault.com/questions/325467/… — Vadzim
– Vadzim, Commented Sep 15, 2015 at 7:25

JohnnyAW · Accepted Answer · 2019-12-11 13:51:07Z

53

The problem is caused by Oracle disabling hash algorithms which are no longer considered to be secure. Take a look at

JRE_HOME/lib/security/java.security

It contains the following properties:

jdk.certpath.disabledAlgorithms
jdk.tls.disabledAlgorithms

You can adjust them appropriately. For example, remove MD5 from the former and MD5withRSA from the latter.

Hint for docker images:

there is additional config file /etc/crypto-policies/back-ends/java.config in some docker images like keycloak in my case which overrides values in java.security

edited Dec 11, 2019 at 13:51

JohnnyAW

2,8761 gold badge19 silver badges27 bronze badges

answered Jan 20, 2016 at 9:31

Igor Nardin

1,64114 silver badges12 bronze badges

Sign up to request clarification or add additional context in comments.

7 Comments

Ricky Over a year ago

That's the ticket. I'm used to having to unscrew VMware components (hello, XP still exists.) I'm not used to having to dumb down java.

Brent Sandstrom Over a year ago

Is there a way to only allow an algorithm from a specific source? I should only be allowing this exception for one or two servers that I trust.

Jib'z Over a year ago

For a newer version of java the java.security file can be found at JRE_HOME/conf/security/java.security

HarlemSquirrel Over a year ago

Editing /etc/crypto-policies/back-ends/java.config on Fedora 33 using Java 11 is what I needed for Apache Directory Studio

Chris Robison Over a year ago

@HarlemSquirrel /etc/crypto-policies/back-ends/java.config was also the solution for the same situation (Java 11, ADS) on RHEL 9.

|

Community · Accepted Answer · 2017-05-23 12:18:13Z

5

keyser gave direction for answer in comment.

Problem was in key's length. In short: "Starting from 7u40, the use of x.509 certificates with RSA keys less than 1024 bits in length is restricted."

So the right way to solve this problem it is using certificates with key's length at least 2048 bits.

edited May 23, 2017 at 12:18

CommunityBot

11 silver badge

answered May 24, 2016 at 6:01

egorlitvinenko

2,7762 gold badges20 silver badges37 bronze badges

Collectives™ on Stack Overflow

SSL handshake exception: "Algorithm constraints check failed: MD5withRSA"

2 Answers 2

7 Comments

Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

7 Comments

Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related