metals-vscode fails to use desired java-version.

[megri]

metals-vscode does a bunch of work to find a version of java which to run:

• It looks for a JAVA_HOME env variable; the JAVA_HOME that existed at the time when VS Code was started, I think
• Failing to find that, and provided there is a java on PATH, it runs a small java-snippet in a shell process which captures the the java.home property of the java-process.
• There's also a metals.javaVersion in the plugin. I'm not sure when this is used. It lets me select either 17 or 21.
• And then there's also a metals.javaHome setting which in my installation is a blank free text field but says it defaults whatever metals.javaVersion is set to.

Note that I set the metals.javaVersion to 21 but it still used JAVA_HOME and managed to locate java 17, which I did not want. At the same time my integrated shell reported no JAVA_HOME being set, while java -version reported the desired version 21.

Since the metals language server has a different lifecycle than bloop, the java version used for metals might differ from the java-version used by bloop, which exacerbates the problem.

Expected behavior
If using either JAVA_HOME or PATH:java is desired, the values used for these should be the ones available from the project root: different tools in this space sets (usually) both of these depending on some arbitrary logic, usually from a .java-version or .tool-versions file which lists the version of java to use for the workspace, but the actual mechanism is not really important.

What is important is that both compile server and language server are kept in sync, so to not create a situation where code that looks like it should compile doesn't or compiled code refers to symbols not available in the language server.

If the metals.javaVersion setting is meant to dictate which version of java to use for the whole of the project then JAVA_HOME should not be searched. If it is only meant to be used for a component which does not provide code introspection but only shuffles data between vscode and bloop, then I think it shouldn't be a controllable setting, as the plugin could manage this version by itself.

As an aside other tools, for instance IntelliJ, keeps a list of JVMs that the user can choose from for his projects. There are no surprises, other than a JVM missing, which is easy to signal to the user with an error message. Perhaps this would be a better solution to the problem?

Search terms
JAVA_HOME

[megri]

I'm sitting here writing a small VSCode plugin that tests different ways of accessing env. With jenv, you control the JAVA_HOME property by setting a value in a file .java-version, which is activated when you cd into the directory. Not sure how exactly it works—I was never very good at OS-specific stuff—but the net result is that process.env from within an extension seems to be bound to an env where JAVA_HOME has not been initialised by jenv. Which makes sense, coming to think of it, as if you start VSCode from Spotlight how could it. This value is never updated.

What I did find was that it's possible to grab the workspace's env by forking a new process that calls env with workdir set to the project root. This is quite slow though, and opens up new questions, like how to pick the correct shell to run.

A better way might be to forgo JAVA_HOME entirely and use the specific tool's way of calling/locating the active java version, but that would require some way of finding the java version manager tool itself and using it in the correct way..

[tgodzik]

metals.javaVersion dictates what version metals should start with, though if you choose 17 and we find 21 then we will use that. Bloop should use the same version unless it was started separately, but we will try to restart it if the version changed (we might need to double check that one.)

JAVA_HOME is a pretty standard way tools use to specify the java home to be used and I don't think we should forgo that. And there should not be an issue if the version of that java is correct, metals will work correctly.

There was a lot of issues detecting Java and I'd rather keep what we have right now with possible fixes.

The algorithm is:

• find JAVA_HOME and check if that is correct
• otherwise find java on PATH and use that
• otherwise download cs and try to download Java using cs

The actual specific Java versions used by metals doesn't really matter unless it's lower than what you might need, We always add proper -release flags based on the version set in metals.javaHome, which is the java version to be used for the project.

What problems do you actually encounter if the java is not the one you expect?

[megri]

There's a multitude of problems with using JAVA_HOME:

• It's not clear where the value comes from, or where it's set, at least for me as a macOS user. I recently found that some tooling had added a .plist-file which forced JAVA_HOME to version 17 no matter what I changed in my ~/.zshrc or ~/.zprofile;
• Changing the environment in the integrated vscode terminal, which is how I control which version my build tool should use, does not change the version metals or bloop uses. I think this is because VS code gets its version on startup, and I rarely restart it;
• When working in multi-root workspaces where projects run on different java versions this becomes a nightmare since I can no longer know/trust which JDK is used to compile which project;
• It is impossible to configure a project so that other users can open them and have the correct JDK used.

To be fair having Metals use JAVA_HOME to get at least something running is nice, but it is still quite hard to understand what you're actually using, especially as your terminal might well report a different JAVA_HOME than what's detected by Metals.

However, working on different projects in as few as two separate vscode-windows using two different JDKs, I've found myself chasing environment variables around trying to understand why one of the project suddenly starts reporting that String#indent (which was added in Java 12) doesn't exist.

[tgodzik]

It is impossible to configure a project so that other users can open them and have the correct JDK used.

I think a much safer approach is just to use "-release" flags which means you will not ever publish a wrong bytecode version by default and it doesn't matter what JDK user has as long as it's higher than the release flag version. You don't have to make sure that the build tools picks up a correct java version either.

When working in multi-root workspaces where projects run on different java versions this becomes a nightmare since I can no longer know/trust which JDK is used to compile which project;

I would really want to know what problems you are encountering here. It should not be an issue especially since Bloop will report the correct errors according to metals.javaHome version or anything set up by the build tool.

So ideally, Java version should be set up by the build tools (some special variable or just release flag) and Bloop/Metals will take care of it having all the errors that it should.

[tgodzik]

And also coming back to you description:

What is important is that both compile server and language server are kept in sync, so to not create a situation where code that looks like it should compile doesn't or compiled code refers to symbols not available in the language server.

This is handled currently, there might be some issues to improve, but the errors and completions should be correct.

[megri]

I would really want to know what problems you are encountering here. It should not be an issue especially since Bloop will report the correct errors according to metals.javaHome version or anything set up by the build tool.

So ideally, Java version should be set up by the build tools (some special variable or just release flag) and Bloop/Metals will take care of it having all the errors that it should.

Issues I've seen is that Bloop is running with the wrong JDK/flag so that a project for which say JDK 17 is used, is reported as using methods that do not exist, such as String#indent which came in JDK 12. Other issues is Metals picking up the wrong JDK even though the correct JAVA_HOME is set for the current session.

My more "shell-savvy" colleague tried helping me look into this yesterday, and it seemed like either VSCode or metals uses the path / when launching, which means JAVA_HOME comes from whatever is configured in the user's home directory, depending on which tool the user is using to provision JDKs. Everyone in the team is using the same tool for this at least, but not everyone has the same JDK set in their user config.

If metals picked up the JDK from the workspace root it'd probably work better, not sure how it would handle multi-projects though 🤔

[tgodzik]

Issues I've seen is that Bloop is running with the wrong JDK/flag so that a project for which say JDK 17 is used, is reported as using methods that do not exist, such as String#indent which came in JDK 12.

That's not really possible if we use JDK 17+ by default, unless metals.javaHome is set to JDK 11 or 8. Or the export from the build tool used a lower JDK. You can confirm that in the metals doctor.

Other issues is Metals picking up the wrong JDK even though the correct JAVA_HOME is set for the current session.

As I said this should not be an issue at all since Metals just uses any correct version that it finds, it has nothing to do with the version that you might want to use for compilation.

The build tool need to export the proper JDK or you need to modify metals.javaHome manually.

If metals picked up the JDK from the workspace root it'd probably work better, not sure how it would handle multi-projects though 🤔

How it's defined? I am not aware of cases where JAVA_HOME or other env variables were different depending on directory.

[megri]

That's not really possible if we use JDK 17+ by default, unless metals.javaHome is set to JDK 11 or 8. Or the export from the build tool used a lower JDK. You can confirm that in the metals doctor.

It's possible this has happened in BSP mode. Unfortunately it's hard for me to provide a setup to trigger the behaviour as most of the issues I've seen are transient and depends on what version of JDK vs code has picked up on (VSCode) startup.

As I said this should not be an issue at all since Metals just uses any correct version that it finds, it has nothing to do with the version that you might want to use for compilation.

Well it does affect compilation, at least in Bloop-mode. Right now I want to work on a java 21 project using virtual threads but Metals automatically locates Java 17 instead:

(This is after a reload of the window with the command Developer: Reload Window)

How it's defined? I am not aware of cases where JAVA_HOME or other env variables were different depending on directory.

With mise, the active version of java (and thus $PATH:java and $JAVA_HOME) is decided on based on the contents of a file called mise.toml (mise also reads .java-version, -tool-versions etc. for compatibility reasons). This is basically how similar tools like jenv, sdkman or asdf work as well.

[tgodzik]

Did you change metals.javaVersion to 21? That should force the extension to pick up 21 and up versions.

[megri]

Did you change metals.javaVersion to 21? That should force the extension to pick up 21 and up versions.

I did change that afterwards and it worked but it downloaded its own JDK instead of using the one available in /Library/Java/JavaVirtualMachines, which is a problem in and of itself; I'm not sure I could reliably get this to work with a newer JDK since metals silently—as in without prompting me for action—downloads JDK 21 and uses that for Bloop.

Another problem is that this setting is a user setting, not available in the workspace settings, so there's no way to share the setting reliably between colleagues.

[tgodzik]

I did change that afterwards and it worked but it downloaded its own JDK instead of using the one available in /Library/Java/JavaVirtualMachines, which is a problem in and of itself; I'm not sure I could reliably get this to work with a newer JDK since metals silently—as in without prompting me for action—downloads JDK 21 and uses that for Bloop.

Maybe we should also check metals.javaHome if it's set 🤔

Otherwise JDK 21 was not on PATH or in JAVA_HOME (or at least JAVA_HOME that VS Code sees)?

[megri]

Maybe we should also check metals.javaHome if it's set 🤔

This would work for one user, using user-settings so global, and would affect all projects the user works on. It won't work for workspace settings.

Otherwise JDK 21 was not on PATH or in JAVA_HOME (or at least JAVA_HOME that VS Code sees)?

Yes, this is true. It seems like metals defaults to resolving JAVA_HOME from the system root, which in turn falls back to the JDK-version defined in $HOME/mise.yaml (if there is such a file). It would be more reasonable to default to the project directory.

But even so, even if we could get this to work perfectly with all different variations of user toolchains, I still think the process is broken by design: it's pretty much global state and assumptions all the way through. Other plugins in the vs code space, like python for instance, prompts the user to specify which virtual environment to use. While this technically doesn't solve the issue of multiple users working on the same project, at least it's letting one user be aware of which platform they're running on.

[tgodzik]

It seems like metals defaults to resolving JAVA_HOME from the system root, which in turn falls back to the JDK-version defined in $HOME/mise.yaml (if there is such a file).

I don't really understand this, from what I know env variables are defined for environment/user, not for specific directories. So it should be the same irrelevant of where we start VS Code. This might differ in different shells but not really in your environment.

it's pretty much global state and assumptions all the way through

That's true, but that's also our attempt to make Metals work without users needing to understand the details of using different Java versions etc. Which might be sometimes at odds with more advanced users. We don't want to ask beginners about that, since it most likely is not relevant to them.

I think we are dealing with two different issues here:

• a problem how to specify the Java to use for a project, I believe this should use the build tools specific way to define that. We can add something for metals, but that will not be used by a lot of people, which feels a wasted effort. We could adjust to some existing solutions, but I really don't want to create anything new.
• a problem of being able to specify JDK you want for running metals, which we actually removed since this was a lot of issues. We could try to modify that to be able to choose from resolved by coursier and some added manually.

If we have the first one, we would be able to know which JDK version to start, or alternatively we should change the default to JDK 21 which would partly fix things.

For 2. this would be a good fallback if Metals did choose something wrong by default.

[megri]

I don't really understand this, from what I know env variables are defined for environment/user, not for specific directories. So it should be the same irrelevant of where we start VS Code. This might differ in different shells but not really in your environment.

I agree with you in spirit, it's a real mess. Perhaps you already know this part but I'll go over it briefly just in case. For linux/unix (and therefore macOS, which I'm running), environment variables are set in slightly different ways depending on if you are logging onto your account, opening a terminal, and which terminal shell you are running.

In my case, the last file touched by an interactive, non-login shell is ~/.zshrc, which in turn evaluates these two lines:

eval "$(~/.local/bin/mise activate zsh)"
eval "$(~/.local/bin/mise hook-env -s zsh)"

mise activate adds a hook to the terminal prompt, which updates the PATH env variable depending on what mise finds in the mise.toml file located in the current working directory, or the nearest mise.toml, going in a straight line up from the directory towards $HOME. If no such file is found I believe it also looks in $HOME/.config/mise/config.toml. This is how it for instance can update the path to the javac command. The hook-env addition also adds/updates JAVA_HOME in the current shell's environment variables, if mise notices you've configured a java-version.

Similar tools, like jenv, sdkman etc. basically all work this way. It makes it so that an organisation can standardise on a "provider" tool, and commit these tool-configuration files to Git, to make sure that all users are running the same version of for instance java for a specific project.

So when VS Code starts it also runs some sort of "rc-file", which initialises mise and makes it set some environment variables. Since metals relies on these variables to find a JDK, it is not all that apparent which one it chooses; you have to look in the Output/metals tab and have some intrinsic knowledge to understand what it's doing. And even when you do think you understand it, it's not certain that the JAVA_HOME metal sees is the one you think you have configured. In fact I'm still not sure if metals-vscode gets a new set of environment variables every time you reload the window, or if the env is static as long as you have at least one instance of VS Code running.

It's an unfortunate practicality but these tools (mise, jenv, sdkman etc.) are meant to standardise on and simplify for new users—install mise, clone the repository, and once you've navigated to it your build dependencies are configured for you (with an additional mise install incantation.)

That's true, but that's also our attempt to make Metals work without users needing to understand the details of using different Java versions etc. Which might be sometimes at odds with more advanced users. We don't want to ask beginners about that, since it most likely is not relevant to them.

I sympathise with that. The problem for me is that even as a self-appointed "advanced user" I still can't get it to work like I think it should in some scenarios, and with less experienced users on the project the experience is even worse.

I think we are dealing with two different issues here:

• a problem how to specify the Java to use for a project, I believe this should use the build tools specific way to define that. We can add something for metals, but that will not be used by a lot of people, which feels a wasted effort. We could adjust to some existing solutions, but I really don't want to create anything new.
• a problem of being able to specify JDK you want for running metals, which we actually removed since this was a lot of issues. We could try to modify that to be able to choose from resolved by coursier and some added manually.

If we have the first one, we would be able to know which JDK version to start, or alternatively we should change the default to JDK 21 which would partly fix things.

For 2. this would be a good fallback if Metals did choose something wrong by default.

Yes, that's pretty much it.

Number 1 is a problem with several dimensions in and of itself:

• Very few build tools (at least if we're talking about mill, gradle, maven etc.) let you specify which version of Java they should use. The only one I know of off the top of my head is bleep.
• As we've talked about before, metals manages Bloop and starts it with the JDK it itself uses, so this setting doesn't come from the build definition.
• Until the community reaches an agreement on whether or not the build should define the JDK, and implements these specifications, then the dev-environment must do it. This is what IntelliJ does, and what the MS Python extension for VS Code does.
  Number 2 I don't really see as a problem with 1. solved: Metals the language-server can run with whatever version it needs. It can either download it automatically, or it could be manually configured in a VS Code setting—in fact the problem could be resolved exactly like it is today, with a few tweaks. The important part then is that this version does not dictate which JDK Bloop runs with.

My reasoning being that while metals-vscode can't necessarily figure out which version of JDK to use for a given project, we do know what it itself needs to run successfully. These dependencies should not be intermingled with a project's dependencies.

[megri]

I should add that I think there are ways of making metals-vscode approachable for fresh users while at the same time allowing more specific configuration for projects that need it, but it needs to be thought over carefully.

I also want to thank you for your patient discourse 😄

[tgodzik]

I added #1559 so we can actually override the version used by Metals.

I wonder if the solution for the issue with directory based JAVA_HOME would not be just running a command to print it within the specified directory

Something like spawn("env", [], { cwd: workspaceRoot }); and greping for JAVA_HOME or trying the same with echo, but I couldn't make echo work with spawn, it seems it's being escaped

[megri]

I wonder if the solution for the issue with directory based JAVA_HOME would not be just running a command to print it within the specified directory

Something like spawn("env", [], { cwd: workspaceRoot }); and greping for JAVA_HOME or trying the same with echo, but I couldn't make echo work with spawn, it seems it's being escaped

What I've done to get this to work from other projects is to run an interactive shell in a process, so basically (oslib-lingo) os.proc("zsh -ic", "<command + parameters I want to run>").call(cwd = workspaceRoot) (I use zsh on my work-computer and fish in private). This is required to let the tools initialise properly, add required hooks, update ENV/PATH and so on.

The problematic part, then, is to figure out which shell the user uses, and how to invoke an interactive shell with it as different shells use different syntax. I'm not sure this is a workable solution due to differences between shells but also differences between platforms — I have zero idea how this should be done in Windows for instance.

[megri]

@tgodzik
I created a small POC-extension which crudely shows how I imagine JDKs could be managed: https://github.com/megri/jdkmanager-poc

I understand this is more of a feature request, but it relates to what we've been discussing here and I'd like to see if you think it would be a good fit.

[tgodzik]

O wow, that is a full fledged extension! And you managed to write it in scala JS, that is quite impressive.

Not sure how to properly integrate this, but maybe you can actually switch metals.metalsJavaHome setting from https://github.com/scalameta/metals-vscode/pull/1559/files this way? We could have it as a separate extension or inside metals, though I guess we would need to rewrite to typescript? Or the other way, but that is much more work :(. I would actually love the extension in Scala JS, but I don't really have time to rewrite it.

[megri]

O wow, that is a full fledged extension! And you managed to write it in scala JS, that is quite impressive.

Thanks, once I got the hang of how to build my own façades it was a pretty nice experience! ScalablyTyped is a bit flaky and won't translate @types/node above version 15.x so those façades needed to be implemented by hand, but it's honestly not that much work so far.

Not sure how to properly integrate this, but maybe you can actually switch metals.metalsJavaHome setting from #1559 (files) this way? We could have it as a separate extension or inside metals, though I guess we would need to rewrite to typescript? Or the other way, but that is much more work :(. I would actually love the extension in Scala JS, but I don't really have time to rewrite it.

One way to integrate would be to depend on the JDKManager (or whatever its name will be) extension by adding it to extensionDependencies in metals-vscode. In a way I think this is the best way to integrate short-term as it lets metals-vscode to control the manner of interaction.

Speaking of integration, how do you want that part to work? From what you've told me so far I think the language server process should just start with whatever suitable JDK the user has configured, but the compilation must really happen with the workspace/workspace-folder JDK defined in the workspace settings.

This means that the process that does the actual compilation might need to be restarted when working with multiple projects, or that several compiler processes should be allowed to run at the same time, perhaps with some internal time-to-live to shut down if not interacted with for some time. For what it's worth this seems to be what mill does:

jps|fgrep MillServer|cut -d ' ' -
f 1|xargs -L 1 -I {} bash -c 'echo -n "{} " && jinfo -sysprops {}|fgrep java.version='
73314 java.version=17.0.13
73225 java.version=21.0.5
73818 java.version=11.0.25
73005 java.version=21.0.5
martin@Martins-MBP-2 ~/d/reception-activity-manager (master)> jps
73314 MillServerMain
73225 MillServerMain
73818 MillServerMain
73005 MillServerMain

I would also love it if metals-vscode was written in Scala though, perhaps I could look into that? It might take a while though, with holidays and other commitments.

[tgodzik]

Speaking of integration, how do you want that part to work? From what you've told me so far I think the language server process should just start with whatever suitable JDK the user has configured, but the compilation must really happen with the workspace/workspace-folder JDK defined in the workspace settings.

The additional extension could change the metals settings, right? So it would change java versions used by metals and ask to reload the window. That would make the dependency one way only. We could later depend from metals on it via extensionDependencies once everything is confirmed to work.

For project java version we would need to update the workspace metals.javaHome and for global stuff we could update metals.metalsJAvaHome once it's merged

I would also love it if metals-vscode was written in Scala though, perhaps I could look into that? It might take a while though, with holidays and other commitments.

Only if it's something fun for you and I would not have an expectation for you to finish it if you didn't feel like it.

[megri]

Speaking of integration, how do you want that part to work? From what you've told me so far I think the language server process should just start with whatever suitable JDK the user has configured, but the compilation must really happen with the workspace/workspace-folder JDK defined in the workspace settings.

The additional extension could change the metals settings, right? So it would change java versions used by metals and ask to reload the window. That would make the dependency one way only. We could later depend from metals on it via extensionDependencies once everything is confirmed to work.

Sounds good, I'll see what I can whip up! Need to figure out how to package this thing as well 😅

For project java version we would need to update the workspace metals.javaHome and for global stuff we could update metals.metalsJAvaHome once it's merged

If i understand things correctly the new metals.metalsJavaHome dictates which JDK to run the Language Server with; is this correct?

I'm a bit confused by the doc for metals.javaHome which states

[…] This Java version should be lower or equal to JDK version used by the Metals's server.

Why does it need to be <= the version the language server runs with? Is this because the version used by the language server is also the version Bloop uses? Would it be possible to "unlock" this dependency?

[tgodzik]

If i understand things correctly the new metals.metalsJavaHome dictates which JDK to run the Language Server with; is this correct?

Yes, but it needs to be at least 17 since metals is compiled with JDK 17

[…] This Java version should be lower or equal to JDK version used by the Metals's server.

We automatically add -release flag to support older versions, but we cannot use a future version.

[megri]

If i understand things correctly the new metals.metalsJavaHome dictates which JDK to run the Language Server with; is this correct?

Yes, but it needs to be at least 17 since metals is compiled with JDK 17

[…] This Java version should be lower or equal to JDK version used by the Metals's server.

We automatically add -release flag to support older versions, but we cannot use a future version.

Ok, I think I understand now. I think we can make the user experience quite pleasant. How about this:

1. If JDKManager knows the user has a JDK of version 17+ installed, it (quietly) set metals.metals-java-home to the highest version available; we also update this value if the user configures a newer JDK;
2. If JDKManager doesn't know of an appropriate JDK yet we ask the user for permission to download one, alternatively they can choose to do it automatically (discover), or manually and reconfigure.
3. If the project is configured to use a higher version JDK than is available, we also prompt the user in the same manner as in step 2.

This can all happen in JDKManager but I think looking ahead it makes sense to make some changes in the metals-vscode extension, for instance when it comes to asking the user to either allow the download of a newer JDK — that metals-vscode can't find the JDK I have configured in env:JAVA_HOME and just downloads something else has caught me by surprise a lot of times 😅

[tgodzik]

If JDKManager knows the user has a JDK of version 17+ installed, it (quietly) set metals.metals-java-home to the highest version available; we also update this value if the user configures a newer JDK;

That should work yeah.

If JDKManager doesn't know of an appropriate JDK yet we ask the user for permission to download one, alternatively they can choose to do it automatically (discover), or manually and reconfigure.

I would do it automatically unless turned off. It's better in that case to remove barriers

If the project is configured to use a higher version JDK than is available, we also prompt the user in the same manner as in step 2.

In that case I would also just download and only if failed involve the user