opam-query.1.3 - via opam-publish

[whitequark]

A tool to query opam files from shell scripts

opam-query is a tool that allows querying the OPAM package
description files from shell scripts, similar to oasis query.

---

• Homepage: https://github.com/whitequark/opam-query
• Source repo: git://github.com/whitequark/opam-query
• Bug tracker: https://github.com/whitequark/opam-query/issues

---

Pull-request generated by opam-publish v0.3.4

[camelus]

✅ All lint checks passed 977affe

• These packages passed lint tests: opam-query.1.3

---

✅ Installability check (6964 → 6965)

• new installable packages (1): opam-query.1.3

[rgrinberg]

A couple of deps that seem to be missing are ocamlbuild and cppo

[whitequark]

Fixed

[samoht]

missing constraint:

#=== ERROR while installing opam-query.1.3 ====================================#
# opam-version 1.2.2
# os           linux
# command      ocaml pkg/build.ml native=true native-dynlink=true
# path         /home/travis/.opam/4.05.0+trunk/build/opam-query.1.3
# compiler     4.05.0+trunk
# exit-code    10
# env-file     /home/travis/.opam/4.05.0+trunk/build/opam-query.1.3/opam-query-15078-6c00d8.env
# stdout-file  /home/travis/.opam/4.05.0+trunk/build/opam-query.1.3/opam-query-15078-6c00d8.out
# stderr-file  /home/travis/.opam/4.05.0+trunk/build/opam-query.1.3/opam-query-15078-6c00d8.err
### stdout ###
# ocamlfind ocamlopt -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild -package opam-lib myocamlbuild.ml /home/travis/.opam/4.05.0+trunk/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# + ocamlfind ocamlopt -package unix -package ocamlbuild -linkpkg -package cppo_ocamlbuild -package opam-lib myocamlbuild.ml /home/travis/.opam/4.05.0+trunk/lib/ocamlbuild/ocamlbuild.cmx -o myocamlbuild
# File "_none_", line 1:
# Warning 58: no cmx file was found in path for module Ocamlbuild_cppo, and its interface was not compiled with -opaque
# File "myocamlbuild.ml", line 1:
# Error: No implementations provided for the following modules:
#          OpamPackage referenced from myocamlbuild.cmx
#          OpamFile referenced from myocamlbuild.cmx
# Command exited with code 2.

[whitequark]

Hmm this is very confusing. It works locally. I'm not sure what's happening...

[samoht]

Maybe add a a minimal version constraint?

[whitequark]

That's not the issue here. It installs the latest versions of opam-lib and ocamlbuild already. Something's broken on opam-repository CI and I have about zero interest in elucidating what that is.

[whitequark]

Hm, actually I've just reproduced that on my own Travis CI, so it's not just an opam-repository issue...

[avsm]

There is a missing end " in this line that is causing parsing to fail.

[avsm]

any chance of a fix for this @whitequark ?

[whitequark]

done

[samoht]

Wrong checksum?

 /home/travis/.opam/packages.dev/opam-query.1.3/v1.3.tar.gz:
          - 0055739ec88a04f3abaef86c9ecba37f [expected result]
          - 74ee290379ef199b4936fbd2180ccecf [actual result]

[whitequark]

done

[avsm]

Sorry :( this is still failing to build. is it a newer version of opam-lib causing the link issue perhaps?

- File "myocamlbuild.ml", line 1:
- Error: No implementations provided for the following modules:
-          OpamPackage referenced from myocamlbuild.cmx
-          OpamFile referenced from myocamlbuild.cmx
- Command exited with code 2.

[samoht]

The failed builds are usingopam-lib.1.3.1. Not sure if this is related ...

[whitequark]

face it. opam is a disaster. it is a nightmare to use it as a library, and it is an even worse nightmare to try and publish opam packages.

[avsm]

It is quite difficult to use opam-lib in the 1.2.2 series as a library indeed, since everything was intertwined. There is hope in the future though, since @dra27 and @AltGr are working on breaking up the opam build (using jbuilder in one branch) in order to make it far easier to test plugins without opam itself being required.

On the publishing front, things will also get easier as the DataKit CI rolls out of experimental mode and into a more automatic merging mode -- it can (with the new queries available in opam2) use the solver to get an accurate reverse dependency cone, and update constraints accordingly.

I do appreciate that you're trying to publish an update in basically the most difficult part of OPAM at the moment...

[dra27]

@whitequark - your package build system is the problem, I'm afraid.

--- a/pkg/build.ml
+++ b/pkg/build.ml
@@ -4,7 +4,7 @@
 
 let builder =
   `Other ("ocamlbuild -use-ocamlfind -classic-display " ^
-          "-plugin-tags 'package(opam-lib),package(cppo_ocamlbuild)'", "_build")
+          "-plugin-tags 'package(opam-lib.format),package(cppo_ocamlbuild)'", "_build")
 
 let () =
   Pkg.describe "opam-query" ~builder [

[AltGr]

face it. opam is a disaster. it is a nightmare to use it as a library, and it is an even worse nightmare to try and publish opam packages.

Thanks for the constructive criticism anyway. To shed some light on things, opam was never constructed to be used as a library, and I have spent tremendous energy tidying things up and providing proper code hierarchy and interfaces since I joined the project. It still isn't a priority, though. I'm sorry you have trouble publishing packages, but face it, if getting the right dependencies is a nightmare, before opam times it was hell. Did you pin the package locally to test your opam file, as advised here, and did you know you can edit the metadata generated by opam publish prepare and re-run opam publish submit any number of times to update your PR ?

Two side-notes on opam query:

• opam-publish now has support for github, so you can do opam publish prepare/submit without argument, granted you already pushed your release tag
• ideally the query information would be available from opam already. Recent versions have opam show --file foo.opam --field fld:

[whitequark]

@dra27

your package build system is the problem, I'm afraid.

I've fixed that in whitequark/opam-query@3cc6408... no idea why opam-publish did not pick it up.

[whitequark]

@AltGr Yes, I did everything you said, and I've read the documentation of opam-publish before using it, too. (Although I did not know about the GitHub support, thanks for telling.)

The root cause here was opam-publish. I had a pinned and uninstalled opam-query package locally, with outdated metadata. The opam file in the directory where I ran opam-publish was fine, the opam file in the archive built by GitHub for the tag was file, nevertheless, opam-publish confused itself into picking the outdated opam file from the switch metadata (and presumably the url file as well). Go figure.

[whitequark]

Oh and while we're at it. The only reason opam-query exists is because of deficiencies in opam-publish. It wasn't (and as far as I can tell, isn't) possible to release packages using opam-publish without doing pointless busywork such as telling opam publish submit where opam publish prepare put the directory with the metadata (that has a different name every time, natch, and cannot be explicitly specified), nevermind that opam-publish, by definition, knows where that is. The in-built GitHub knowledge is a step in the right direction but it is baffling to me why the busywork cannot be eliminated entirely.

[dbuenzli]

@AltGr TBH I always found opam publish prepare to be too obscure to be used. This is why in topkg I only use opam publish submit and prepare the submission myself. I think you should at least document in the manpage how and where it gets its metadata from.

@whitequark You should have a look at topkg's release procedure see topkg help release.

[whitequark]

@dbuenzli Thanks, I should really migrate all my packages to new topkg workflow.

[AltGr]

The in-built GitHub knowledge is a step in the right direction but it is baffling to me why the busywork cannot be eliminated entirely.

Maybe the mistake I did at the beginning is that opam-publish does not need to be run from a source tree or version-controlled repo, although it seems that's the way everyone uses it. In that scope, nothing can be inferred, in particular, not the link between prepare and submit. Another assumption is that you test your package with pinning before publishing (nice for new packages, but people tend not to do it for updates, esp. now that we have good CI). The point was to allow opam pin edit to change the submitted opam file. And finally, it was designed before it became so common for all packages to include their definition. Based on the Packaging guide, I wanted it to be usable to package an existing project, directly from its released source tarball.

The context has evolved, and now the tool is blamed for not being exactly what people think it is or should be. Constructive criticisms, and suggestions, are always welcome (as are contributions!) -- but there is no need to shout.

With the Github support, you shouldn't need to specify any extra arguments to the submit command, they can be inferred like they were for prepare (I think there was a bug with that feature in a previous release, though). Since this needs running from a source tree, it is a wholly different mode from the lower-level one explained above, though.

[dbuenzli]

now the tool is blamed for not being exactly what people think it is or should be.

I read this very differently, I think the tool is blamed for being obscure in the way it operates. Reality is that it's difficult to understand where the opam metadata comes from when you opam prepare. Note that opam prepare --help and the README say absolutely nothing about it.

Regarding opam pin edit workflows I suspect it's little used in general because it becomes unclear what will happen to your changes, where they are stored and if they will be overwritten on the next opam update (I know they are saved but they end up tucked somewhere in the .opam directory). So people prefer workflows where data is sourced from the opam file of repos, where you are in control and nowadays have to commit your opam file anyway. As far as I'm concerned you could completely remove that edit feature from opam pin and I think you should change opam prepare so that it consistently sources the metadata from an opam file you specify on the cli and/or find in the cwd.

In any case I think you should concentrate on the submit part which is where all the painful bureaucracy lies in the opam-repository submission model. Generating the actual package directory to add to the repo is piece of cake. Especially nice would be to have a step in the direction of multiple package submission with high-level information (e.g. this is an incompatible release) for automated constraint update. opam-ed is certainly nice but still a bit too low-level.

[AltGr]

Regarding opam pin edit workflows I suspect it's little used in general because it becomes unclear what will happen to your changes, where they are stored and if they will be overwritten on the next opam update (I know they are saved but they end up tucked somewhere in the .opam directory).

opam will offer to copy back your opam file to the original source directory, but I get your concern, and having the opam file in the source is certainly the easiest to figure out for users. Now that most are moving to jbuilder, which requires the opam file at the root, it's even easier.
I wouldn't remove opam pin edit altogether, though: it may not be the best as a workflow, but it is still extremely useful to quickly patch a package definition (e.g. a buggy one, or to try to remove a constraint), and doesn't require a pin to a VCS.

In any case I think you should concentrate on the submit part which is where all the painful bureaucracy lies in the opam-repository submission model. Generating the actual package directory to add to the repo is piece of cake. Especially nice would be to have a step in the direction of multiple package submission with high-level information (e.g. this is an incompatible release) for automated constraint update. opam-ed is certainly nice but still a bit too low-level.

1. latest version allows multiple package submissions, but at the moment limited to the case of a single VCS repository
2. you'll be happy, I have a working opam admin add-constraint command right here. I just want to polish my formula simplification function a bit more before submitting.

[whitequark]

Especially nice would be to have a step in the direction of multiple package submission with high-level information (e.g. this is an incompatible release) for automated constraint update.

I'm going to argue that this is just papering over complete lack of opam support for any sort of principled version management. For example I follow a strict subset of Semantic Versioning (version m.n, bumping n means backward-compatible change, bumping m means backward-incompatible) and in every other package manager I've used there is a constraint like ~ 1.2 that expands to >= 1.2 & < 2.0. But not opam.

And having to carefully state these bounds manually is offputting enough that very few opam users bother to. I certainly don't do it for everything as I should although I try to, when possible.

In my opinion having completely unconstrained dependencies should be linted against (eventually as an error) because it is guaranteed that the package will break eventually, and relying on limited resources of opam admin staff is wasteful.

[whitequark]

And in case the answer is "patches welcome": sure, that is a reasonable answer for a request like cross-compilation support or something complex and niche like that. But "support for extremely widely used versioning practices", just like "support for hash functions that aren't so broken that people are generating increasingly contrived collisions as a form of recreation", shouldn't be something that external contributors provide after protracted arguments, it should come up after a cursory look at what other package managers do.

[AltGr]

opam 2.0.0 has supported sha256 and sha512 for a while now... and it wasn't so trivial without bringing in new unwanted dependencies, see ocp-sha.

And this can't be compared: we do support semantic versionning, we just don't enforce it. This is not saying we could not add sugar to make the lives of users easier when they do use it. While ~ could be nice, there is not a single feature request on opam's bug tracker asking for it (correct me if I am wrong), so maybe you should start there.

[whitequark]

While ~ could be nice, there is not a single feature request on opam's bug tracker asking for it (correct me if I am wrong)

The entire point is opam should lead the way here... But sure: ocaml/opam#2976.

[whitequark]

@AltGr Actually, you are wrong that there is no such existing feature request on the tracker. You filed one yourself!

[AltGr]

Ah, indeed, my bad! Nobody showed much interest at the time though :)

[samoht]

Yay, all green now! Thanks for your patience...

[dbuenzli]

I'm going to argue that this is just papering over complete lack of opam support for any sort of principled version management.

I'm not convinced. I still think it's better to keep upper bounds open ended. Even major versions may not break everything out there.

Note that as far as OCaml goes almost every release is a breaking one: even adding new functions to a module is a potentially breaking change (because of the M.() notation).

So at that point semver OCaml is simply breaking on almost each release and you are back to the point where you need high-level tooling to have finer control over the packages which actually do break and those that do not.

So really I'd prefer better high-level tooling to bulk control package versions than add more syntax to the opam files.

[whitequark]

Note that as far as OCaml goes almost every release is a breaking one: even adding new functions to a module is a potentially breaking change (because of the M.() notation).

I'm not arguing in favor of following semver in the strictest sense (which, in spite of what semver advocates say, is essentially impossible anyway without some sort of formal methods). What I'd like to see is a bound that works 95% of the time. In the (in my experience, rare) case where a breaking change occurs because of the local open (or the like), you can tighten it, and in the (in my experience, rare) case where a new major version is compatible with a downstream package you can relax it. But you need not do anything in the common case.

So really I'd prefer better high-level tooling to bulk control package versions

Why not both?

than add more syntax to the opam files.

This is fine as a personal preference, sure.

[dbuenzli]

Why not both?

Why not but opam development being bounded by available ressource, I'm just advocating for what I think will be the more useful in my day to day practice. However once a tool can perform the tedious bounds manipulations for me using high-level operations I don't see a clear advantage to add new syntax that has to be learned and understood.