Fixing "uncommon" DSD-Wavpacks

[ArminiusTux]

Dear Mr. Bryant [@dbry ],

attempting to cut a DSD track by compressing it to Wavpack and repackaging (incl. cutting) that into Matroska works surprisingly well.

General
Unique ID : 285979660214772406803837314364250274400 (0xD725ADDA1BF2284EA15735CDF2E43A60)
Complete name : C:\Temp\dff\wavpack\explain_half.mka
Format : Matroska
Format version : Version 4
File size : 22.0 MiB
Duration : 4 min 15 s
Overall bit rate mode : Variable
Overall bit rate : 725 kb/s
Encoded date : UTC 2020-05-24 20:43:16
Writing application : mkvmerge v46.0.0 ('No Deeper Escape') 64-bit
Writing library : libebml v1.3.10 + libmatroska v1.5.2

Audio
ID : 1
Format : WavPack
Format profile : 4.16
Format settings : Lossless
Codec ID : A_WAVPACK4
Duration : 4 min 15 s
Bit rate mode : Variable
Bit rate : 724 kb/s
Channel(s) : 2 channels
Sampling rate : 88.2 kHz
Frame rate : 4.000 FPS (22050 SPF)
Bit depth : 8 bits
Stream size : 22.0 MiB (100%)
Encoding settings : -h
Default : Yes
Forced : No

However doing the rollback (mka -> wv -> dff) fails at the last step.

Can you confirm that MKVtoolnix (by @mbunkus) does not fully support WavPack v5 (especially for extraction)?

Since the borked .wv output (generated by mkvextract) does not play on its own BUT can be remuxed into .mka and played back again, would you consider adding a repair command line option to wavpack.exe & wvunpack.exe to salvage such "uncommon" DSD-Wavpacks?

wavpack.exe -r mkaextract.wv dsdfixed.wv

 WAVPACK  Hybrid Lossless Audio Compressor  Win64 Version 5.3.0
 Copyright (c) 1998 - 2020 David Bryant.  All Rights Reserved.

not compatible with this version of WavPack file!

Regards,

ArminiusTux

[dbry]

This does not surprise me since there is a similar issue with FFmpeg. Unfortunately my regular development system is too old to build MKVtoolnix so I couldn't test just now.

Can you try to verify the "uncommon" DSD WavPack file adding the -b option to the wvunpack command (i.e., wvunpack -vmb mkaextract.wv) to see if that fixes the issue? That will give me some idea of the scope of the problem.

Thanks!

[ArminiusTux]

Good evening,

well your command indeed restored the whole filesize but the playable track length is just 1:04 instead of 4:15

wvunpack -b ZZZZZ.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win64 Version 5.3.0
 Copyright (c) 1998 - 2020 David Bryant.  All Rights Reserved.

restored ZZZZZ.dff in 4.94 secs (lossless)

restored file:

General
Complete name : C:\Temp\dff\wavpack\ZZZZZ.dff
Format : DSDIFF
Format/Info : Direct Stream Digital Interchange File Format
Format version : Version 1.5.0.0
File size : 42.9 MiB
Duration : 1 min 3 s
Overall bit rate : 5 645 kb/s

Audio
Format : DSD
Format/Info : Direct Stream Digital
Commercial name : DSD64
Format settings : Big
Duration : 1 min 3 s
Bit rate : 5 645 kb/s
Channel(s) : 2 channels
Channel layout : L R
Sampling rate : 2 822 kHz
Compression mode : Lossless
Stream size : 42.9 MiB (100%)

extracted file (from mka):

General
Complete name : C:\Temp\dff\wavpack\ZZZZZ.wv
Format : WavPack
File size : 22.0 MiB
Duration : 4 min 15 s
Overall bit rate mode : Variable
Overall bit rate : 725 kb/s

Audio
Format : WavPack
Format profile : 4.16
Format settings : Lossless
Duration : 4 min 15 s
Bit rate mode : Variable
Bit rate : 725 kb/s
Channel(s) : 2 channels
Sampling rate : 88.2 kHz
Bit depth : 8 bits
Compression ratio : 1.948
Stream size : 22.0 MiB (100%)
Encoding settings : -h

EDIT:
Last but not least the exact command you actually requested (no file generation):

wvunpack -vmb ZZZZZ.wv

 WVUNPACK  Hybrid Lossless Audio Decompressor  Win64 Version 5.3.0
 Copyright (c) 1998 - 2020 David Bryant.  All Rights Reserved.

unpacked md5:  d4590d84df8f2b4d365c5c772105cebb
verified ZZZZZ.wv in 4.85 secs (lossless) 

[mbunkus]

Can you confirm that MKVtoolnix (by @mbunkus) does not fully support WavPack v5 (especially for extraction)?

MKVToolNix author here. mkvmerge (the CLI application part of the MKVToolNix package that actually does all the work) only supports WavPack v4. If v5 of the program also means that the format's changed somewhat, mkvmerge will most likely not work with it.

If, however, the format's unchanged, I'd be interested to know what mkvmerge does wrong. If it cannot be fixed easily, I might have to disable splitting & appending of WavPack in mkvmerge (same as I did for FLAC).

[ArminiusTux]

Hello @mbunkus,

thank you for the interest in this "niche topic".
WavPack v5 offers not only PCM compression modes but also for DSD now.

The great audio schism at the turn of the century left no clear winner, but Matroska can bridge this divide by storing the PCM-based audio streams (from DVD-Audio or Blu-ray Audio) and DSD (from SACD) in the same MKA file in a compressed and lossless manner.
As for handling WavPack v5 (DSD-type) I really can't blame mkvmerge for anything, as muxing and playback works nicely for "v5-aware" tools like foobar2000 (+plugin).

The only downside at the moment is that you can't get the wv-dsd out of the MKA with mkvextract in a compatible (V5) fashion.

Last but not least - being able to "roughly but losslessly cut" DSD streams (without transcoding to PCM) comes not only handy for certain audiophiles, but could also be beneficial to those numerous DSD online shops who could offer short samples to their customers.

Regards,

ArminiusTux

[dbry]

Here's a guide to what's new for version 5

Within the next few days I will get MKVToolNix building here (or at least experiment with the executables) and figure out exactly what's happening. I have some experience with this from FFmpeg and can give a very quick summary of the probable relevant changes:

1. An optional raw checksum has been added to each WavPack block. This is for the undecoded block and was added to improve format robustness, but to retain compatibility with old WavPack it's completely optional. However, if it's included it must be correct, and since blocks may be re-created during Matroska demuxing these can easily get messed up. The easiest thing is to simply always delete these checksums during both muxing and demuxing and I have created a simple function to do this that I can share. BTW, ignoring checksum failures is one of the things that the -b option does in wvunpack.

2. The header flag bit that indicates DSD data should have triggered parsers that did not know about DSD data to error out, but this was generally not done, so we have WavPack DSD audio embedded in Matroska files. This should not be too bad because it appears as 8-bit PCM, but unfortunately there is no way to determine the sample rate without parsing the actual WavPack blocks and get the "multiplier" for the DSD rate. This is pretty trivial to do also (it's similar to how "non-standard" sample rates are stored) and I also have code to do this.

I am happy to help with any changes required to get this working correctly. I'm not a C++ programmer, but have C snippets that do most of what's required. Let me take a look at and experiment with the relevant code and come back.

Thanks!

[dbry]

I have verified that the two issues above were in fact the problem, and have submitted a patch on GitLab to address them. This would obviously be preferable to the creation of a fixing tool...

GitLab patch

[dbry]

Hey @ArminiusTux, did you happen to get a chance to try out my changes? I suspect that they'll resolve the issues you were having.

BTW, one thing I was thinking about was adding support to sacd_extract to write directly to WavPack files (I noticed that you have a fork of it). Whenever I convert my SACDs I have to go through dff, so that would cut out a step.

[ArminiusTux]

Hello @dbry,

I was able to compile the mkvtoolnix master sources from git with your changes (5 files) for Ubuntu.
The new

mkvmerge v47.0.0.12 ('Black Flag') 64-bit &
mkvextract v47.0.0.12 ('Black Flag') 64-bit

superbly handles cutting & merging of wavpack-dsd files now.
You may lose the buried dsf tags & DSD MD5 hashvalue upon cutting and merging but not a SINGLE audio sample!
In short, MISSION ACCOMPLISHED - going into and out of the Matroska container no longer renders a wavpack stream unplayable.

Seeking (.mka only) is a bit problematic for short duration pieces but when linked to the other parts, the playback is gapless.

As for having wavpack-dsd output in sacd_extract - I would love that too. However my skills are totally insufficient to that aim (only forked setmind's project a year ago, because I wanted to make the sources compile for PowerPC).

Although your preferred SACD ripping target seems to be single files with integrated cuesheets (DIFFMASTER + CUE -> WavPack-DSD), may I redirect your attention to the latest fork by EuFlo - it offers advanced tagging for DSF.

A was a bit surprised by the fact that a single file wv-dsd (from DIFFmaster) rip offers no compression gains over individual track wv-dsd files (from DSF) counterpart.

Last but not least, I am really aching for WavPack v5 support in Kodi, LibreELEC, VLC etc.
Is there anything I can do to speed up things with FFmpeg's implementation?

Regards,

ArminiusTux

[dbry]

Hi @ArminiusTux,

Thanks so much for testing out my patch! In case you haven't seen it's been accepted into the MKVToolNix repo by @mbunkus and I've tested it again, so it's all good!

I wasn't suggesting that you put the WavPack support into sacd_extract (although that would obviously be very nice). I was more thinking about myself doing it and whether you thought it was worthwhile. I currently do everything in two operations when I rip SACDs so it's not that bad, but the more annoying thing is making sure I have space for the intermediate .dff file.

As for FFmpeg, my patch was accepted there to play WavPack DSDs a while back and the master branch plays them fine, so it's just a matter of waiting for the next release, which they seem to do on their own schedule. I guess you could use a nightly build, but I have no experience getting those other apps to use it.

Kind regards,

David

[ArminiusTux]

Hello David,

thank you for the kind words - glad I could be of help.

@mbunkus, thank you for accepting David's changes - upon its next release, MKVToolnix will become the world's first basic editing tool for DSD audio streams!

Sorry for the mixup about sacd_extract, the addition of directly ripping to WavPack-DSD would be truly welcome.
Since ripping & storing ones CD-As as LPCM (wav or aiff container) is hardly done today, I see no point in leaving the DSD audio streams uncompressed (be them from SACDs or online shops). Unless of course the backup is done for HW compatibility (e.g. audio receivers) - but even for that purpose WavPack offers easy conversion to DSD or DFF.

To make a long story short, yes do please contribute to the sacd-ripper project - may I suggest joining an existing fork though:

https://github.com/setmind/sacd-ripper
https://github.com/EuFlo/sacd-ripper

Best regards,

John

[dbry]

@ArminiusTux Yeah, the only disadvantage of going direct from SACD to WavPack is that the resulting files would not be convertible to DSF or DFF files with tags because I have no intention of converting the APEv2 tags that WavPack uses to ID3v2 tags. Of course, I could put both tag formats in there as an option (although that's kind of ugly because they could get out of sync).

What I have been doing is creating a single image file that contains a whole SACD (either the stereo or multichannel stream) and embedding both the cue sheet and the cover art in there. One file for everything! Foobar2000 handles them fine and I can export to 16/44 individual tracks for playback on my Oppo.

I took a quick look at adding WavPack support and it won't be as easy as I had hoped. But if I get some free time I'll give it a go.