There are 2 built in methods, and options for extending the REST API. Note that you can't use these to create new authenticated sessions or register new users.
Using a 3rd party Service for Login
There are filters that allow you to intercept the login checking and the login form. These can be used to redirect the user on login to a 3rd party site. When the 3rd party has authenticated the user and logged them in, it redirects the user back to your WP site where you can then either create a new user based on their details, or create a login session. This "user" is entirely internal and not something the visitor would use to login.
This is how most login federation works even outside of WordPress, as well as how popular plugins for implementing login via services such as auth0, Facebook logins, Google, OpenID, etc are all implemented.
Without this, there would be no way to get user IDs for a logged in user, which would break most of WordPress.
Speaking to WordPress Via The REST API
You'll want to authenticate and communicate with the REST API, allowing easy JSON based communication with the WordPress site. This should simplify things, and provide data in a machine readable fashion that Java is happy to read and process. It'll also allow you to use common libraries to communicate with WordPress instead of building the API requests from scratch
Speaking to WordPress from Java via the REST API
Built In Authentication Methods for REST API
any other way in Wordpress side to configure external user authentication but without developing anything?
No. The only authentication WordPress provides out of the box is:
- cookie + nonce based authentication, standard method out the box
- Basic Auth with Application passwords, the simplest but least secure option
Keep in mind that you can't use these to start sessions, there is no way to login with the REST API out of the box or to register, so these need to be handled separately. Application passwords have to be preconfigured and cookie based auth requires you to login first via the browser.
Additional Plugin Based Authentication Methods
There are however plugin based solutions, specifically ones that rely on widely accepted standards that aren't unique to WordPress. These implement standardised authentication protocols such as:
- Oauth1, you can use this with the REST API to authenticate from Java
- Oauth2, like Oauth1 you could use this, I'd recommend this over OAuth1, assuming your site uses SSL/TLS/Https
- SAML, you could authenticate with a SAML provider, and use that instead of WP to handle your accounts, and install a WP plugin to make WordPress use the SAML provider too. SAML is more popular with enterprise solutions and software, but more involved, with multiple types and schemes for implementing it
- JWT, avoid if possible for security reasons, not just in WP, it's attractive but fundamentally flawed, I mention it for completeness
- Raw Basic Auth, the simplest but least secure option
- Rolling your own, you could implement a REST API endpoint or two to handle user login and registration, but this is a very bad idea. There's a reason people who figured this out wrote standards such as OAuth or SAML, think of it like rolling your own cryptography, unless you have a degree in the subject you'll build something insecure with bugs, not worth the hassle
Note that these protocols are not WordPress protocols, but general standards used across numerous pieces of software.
As an example, I have a CLI application that requires the user to be authenticated, using a WordPress site to store data/users/etc. In order to login the application uses OAuth1 to communicate with WordPress and exchange details.
For your Java application, you could communicate with WordPress via OAuth2 to authenticate your user, perhaps even show an inline dialog showing a login form to approve.
Misc
Or the way by using Keyring or/and writing own filter is the right one?
This isn't a single problem, if you pick a standard then you can break this apart into a pure WP problem, and a pure Java problem. You'll get 100x as many results for implementing OAuth2 client in Java than you would for logging into WP with Java. The same for logging into WP with OAuth2.
However, Java questions are not in this stacks scope ( try stack overflow ), neither are plugin recommendations ( there's a software recommendations stack ).