What is the versioning policy?

[timj]

Is there a document somewhere explaining the versioning policy for astroquery.

The 0.4.8 version completely broke compatibility with the Simbad queries as implemented in 0.4.7. At first glance this seems like a patch release that shouldn't break anything so I think at Rubin we may have understood the meaning of versions. Would it be possible in the release notes to indicate more strongly that breaking changes were present for a particular query component? Should we be using exact pins for astroquery in the future?

We did get some deprecation warnings that turned out to be immediately fatal (using flux(U) as a column name) but the resulting table had columns that were completely incompatible with the old version.

cc/ @gpdf

[keflavich]

@timj could you provide more details about the failing queries?

There is a brief policy explanation here:
https://astroquery.readthedocs.io/en/latest/#installation

In short - we do not, and cannot, guarantee backward compatibility at any point because we're subject to the whims of the upstream services: if they change a keyword or do anything different, everything in astroquery necessarily breaks, and so we don't attempt to hold to normal versioning standards.

That said, SIMBAD tends to be very stable and I would not have expected this regression. We did do some cleanup/upgrades/fixing stuff that wasn't working as expected in a few PRs recently, but I wouldn't have expected it to affect your usage.

[bsipocz]

I agree that the version numbers are somewhat unfortunate, practically any tagged version is to be treated as full-fledged feature release as we do not do patch releases (we should have deleted the leading 0 a really long time ago).

That said, Simbad has been fully refactored. I don't think we anticipated breaking changes, and added deprecations at a lot of places, so if you could open an issue with the details it would be useful.

cc @ManonMarchand

[keflavich]

I'll add another informal part of the policy that we haven't discussed, but I'd like to say out loud for the record: if upstream didn't break it, we shouldn't break it, if at all possible.

[bsipocz]

And for the future: We plan to switch to date based releasing, and having much shorter release cycle. As Adam said above, we will never be able to guarantee any past releases to keep working as they may rely on a upstream services that have changed, but we do change the API with care and with deprecations whenever possible.

[timj]

Regarding the breaking changes for the Simbad queries: At a guess the old simbad query system returned RA, DEC etc in all upper case but the TAP service returns everything in all lower case. This does break all the code that is looking for RA etc in the table. RA and DEC were hms/dms strings but in the new version they are decimal degrees.

From the query side flux(U) is no longer supported and requires to be U. It looks like an attempt was made to have a deprecation warning period since I get:

  /Users/timj/work/lsstsw/build/spectractor/lib/python/spectractor/extractor/targets.py:262: DeprecationWarning: The notation 'flux(U)' is deprecated since 0.4.8 in favor of 'U'. See section on filters in https://astroquery.readthedocs.io/en/latest/simbad/simbad_evolution.html to see the new ways to interact with SIMBAD's fluxes.
    simbadQuerier.add_votable_fields('flux(U)', 'flux(B)', 'flux(V)', 'flux(R)', 'flux(I)', 'flux(J)', 'sptype',

but then it fails anyhow:

ValueError: 'flux(j)' is not one of the accepted options which can be listed with 'list_votable_fields'. Did you mean 'flux'?

This feels like an inconsistency in the deprecation handling.

The sptype and z_value fields do get migrated to the new sp_type and rzv_redshift names for the query side and issue a warning saying what the new names should be in the results.

And pm doesn't get a migration warning even though that's meant to be propermotions now.

[timj]

Regarding versions, date-based versions would be a huge improvement since it would then be obvious that semantic versioning is not involved.

[bsipocz]

semantic versioning is not involved.

😄 that's kind of true all over in python land. (I mean the major version is usually indicated by semver minor :) ).

Anyway, the timeline for the switch is soon, the packaging overhaul will be my first task once the paperwork is sorted out for the roses grant.

[ManonMarchand]

That said, Simbad has been fully refactored. I don't think we anticipated breaking changes, and added deprecations at a lot of places, so if you could open an issue with the details it would be useful.

We did anticipate breaking changes. See discussion here: (#2954 (comment) and #2954 (comment))

That said, SIMBAD tends to be very stable and I would not have expected this regression. We did do some cleanup/upgrades/fixing stuff that wasn't working as expected in a few PRs recently, but I wouldn't have expected it to affect your usage.

These changes are from ~ six months ago when we changed the API from the low maintenance and planned for shutdown one to TAP. See changelog added in #2954 and the discussion for more details on the things we couldn't prevent from requiring changes from the users side.
There is also a documentation page dedicated to these changes: https://astroquery.readthedocs.io/en/latest/simbad/simbad_evolution.html
Hope this helps?

Regarding the breaking changes for the Simbad queries: At a guess the old simbad query system returned RA, DEC etc in all upper case but the TAP service returns everything in all lower case. This does break all the code that is looking for RA etc in the table. RA and DEC were hms/dms strings but in the new version they are decimal degrees.

Yes, the new API has a different output. Blanketing this by changing the values after receiving the response was too much of a performance lost.

but then it fails anyhow:

  /Users/timj/work/lsstsw/build/spectractor/lib/python/spectractor/extractor/targets.py:262: DeprecationWarning: The notation 'flux(U)' is deprecated since 0.4.8 in favor of 'U'. See section on filters in https://astroquery.readthedocs.io/en/latest/simbad/simbad_evolution.html to see the new ways to interact with SIMBAD's fluxes.
    simbadQuerier.add_votable_fields('flux(U)', 'flux(B)', 'flux(V)', 'flux(R)', 'flux(I)', 'flux(J)', 'sptype',

You can replace by:

simbadQuerier.add_votable_fields('U', 'B', 'V', 'R', 'I', 'J')

This feels like an inconsistency in the deprecation handling.
And pm doesn't get a migration warning even though that's meant to be propermotions now.

Yes, something is wrong on these two, they should warn about the migration better. For flux(J) it's a casing issue, and for pm I'll have to investigate... Sorry for this. I could try to fix these. @bsipocz do astroquery backport changes? Otherwise we cannot do anything other than keeping this issue opened 🙁

EDIT: flux(J) works properly. Are you sure you did not type flux(j) ?? It's a nasty bug. Submitting a bugfix soon

If you notice anything else, or if there is something that you cannot do with the new API, feel free to open other issues