HDCD Examined
On a previous webpage I gave an outline explanation of how the HDCD (High Definition Compatible Disc) system is said by its inventors to work. Here I want to examine what results we can get by looking at some examples of real HDCDs.
To do this I needed to be able to analyse behaviour in a bit-accurate sample-by-sample manner. I had written a simple program to ‘blindly’ apply the HDCD soft peak limiting and generate 24bit output files from HDCD tracks. However I also needed to probe for the use of the other methods that HDCD is said to deploy – the low-level gain adjustments and the switching of record and replay filters. This posed problems because of the ‘closed’ nature of many of the vital details. In theory a Patent is required to provide all the details needed so that a “skilled practitioner in the field” can duplicate the invention. The idea being that a Patent is a two-way agreement. The person who gets the patent has a monopoly for some years in exchange for disclosing the details so that anyone else can freely use the same ideas and methods once the patent lapses. The basic HDCD patent was granted in the mid 1990s, so it will lapse in a few years time. However amongst all the grand claims in the Patent, many critical details are omitted.
Fortunately, someone has already found out how to read the HDCD control codes. Or at least some of them! A Windows executable for this was released by C. J. Key some years ago, but without the source code. So it was for Windows users only. Then Foobar released source C/C++code files, which were said to be based on C. J. Key’s. With a lot of help from someone who is a far better programmer than myself, I was able to produce a working program that examined HDCD recordings and could be used to analyse what HDCD processes were being employed in practice. I would like to thank Nick Gorham for his kind help with this.
Alas, this code (and, I presume, the code by C. J. Key) only deals with the soft peak limiting and low level gain adjustments. No sign of any fiddling with the record/replay filters. So I am unable to comment on that. Indeed, I suspect that at present although Foobar and the Windows Media Player are said to handle HDCD, they too may not deal with that aspect. However I was able to use both the Key/Foobar approach and my own ‘blind peak expansion’ to examine the dynamics for various discs and tracks. This let me assess how close HDCD comes in use to the claimed 4-bit improvement over Audio CD. Both programs output 24bit samples to ensure any expansion of the input 16bit range is properly represented. Here I’ll just describe a few examples that seem to fairly represent the range of results I obtained from a variety of discs.
The first example is a track from a Claire Martin HDCD. This can be expected to be carefully recorded as it was made by Linn Records.
Figure 4 - Claire Martin - Linn AKD122
The upper graph in Figure 4 shows the peak music level during each 100 millisecond section of the chosen track when played as a normal CD. (N.B. For all the results shown on this web page, the red line shows the left channel, and the blue line shows the right channel.) The lower graph in Figure 4 shows the change in peak levels produced by HDCD decoding. (By which I mean the change produced by using the Key/Foobar method.)
By default the HDCD decode process scales down sample values by 6dB with respect to the maximum possible (0dBFS). Hence if the track were not HDCD encoded the result would be a flat line at -6dBFS. In general if we see the level in a “Played as HDCD - Played as Audio CD” plot rise above -6dB we can expect this to be the result of HDCD soft peak compression being expanded during loud peaks. If the level moves below -6dB we can expect that to be due to HDCD alterations in the gain during long low-level passages of music.
In this case we can see that most of the time the change in level produced by HDCD is -6dB. However, when the music peaks from the disc reach above -3dBFS, decoding expands the peaks. This produces the occasional brief rise above the -6dBFS line. All being well, this restores the shapes of the peaks that would be ‘squashed’ when the disc is played as being a CD on a normal player.
When I tried my blind peak expansion program it gave exactly the same results as the above. That implies two things about the Claire Martin track.
- The blind program doesn’t find or obey any control codes. It just assumes any excursions above -3dBFS in the input data from the CD require expansion. The Key/Foobar decoding only performs expansions when it has detected HDCD control codes that enable this. So getting the same results from both processes tells us that effectively the entire track is ‘coded’ for soft peak limiting to be enabled.
- Taken together, the results show that at no point was the music affected by any low level HDCD gain changes. Either this was disabled for the track during recording, or the level never went low enough to prompt the effect being employed.
So in the case of the above track ‘blind’ peak expansion would have worked as well as full code-controlled HDCD dynamics expansion. In practice it seems to make sense for the soft peak limiting to either always be enabled (or disabled) during a complete piece of music. Otherwise, when played on a normal CD player the sound level might jump up or down by 6dB whenever it was turned on or off! This would be quite noticeable. It may also be significant here that some users of HDCD players report hearing a ‘click’ when they begin playing an HDCD. This may be a result of the peak limiting and associated change of 6dB in overall gain being deployed in some designs of player. Although – as seems usual for HDCD – it is hard to tell because practical details of how players work are scarce.
Looking at the lower graph in Figure 4 we can also see that the degree of soft limiting was slight. Typically around 1dB. Hence any ‘improvement’ in dynamic range due to HDCD in this case would only have been 1 or 2 dB. Rather less than than would be possible from full use of even 1 extra bit (6dB) let alone the HDCD claimed maximum possible – 4 bits leading to a maximum claimed improvement of 24dB over normal CD. Hence it might have been simpler for this example if it had been made as a conventional Audio CD with the overall level turned down by just 2dB. That way people playing the result in a normal CD player could have enjoyed it without any soft peak limiting imposed by HDCD!
Figure 5 Joni Mitchell - Don Juan’s Reckless Daughter
Figure 5 shows the results of examining one of the Joni Mitchell discs that I purchased thinking it was a normal Audio CD. It was advertised as a CD and there are no HDCD logos externally visible. Yet it is an HDCD, not a conventional Audio CD. Inside the leaflet it mentions in tiny text “... mastered using the Pacific Microsonics HDCD system”. But if you bought the disc shrink-wrapped or over the net you wouldn’t know what you were getting. Despite that, Figure 5 indicates that it uses both the soft peak limiting and low level gain alterations of the HDCD system! Peaks are expanded by up to about 4dB and a few brief quiet sections are altered by up to 3dB. So in principle you can say it extends the dynamic range by about 7dB compared with normal CD replay. (Just over 1 bit worth.) Although some of the low level adjustments seem to be in the gaps between tracks so it is debatable if they contribute much...
In this case my impression is that when played as a CD it sounds distinctly different to when played as an HDCD. As a CD it seems to suffer from noticeable peak compression. This was what prompted me to examine the leaflet carefully and find the mention of HDCD. If I hadn’t been investigating HDCD I probably wouldn’t have noticed this. I would have just put it down as one more Audio CD suffering from the pop/rock business’s obsession with loudness and peak compression. The shame is that most people will listen this as a CD and not be able to hear the sound properly. Whereas if the level had been wound down 3-4dB it would probably sound better when played as the Audio CD customers probably assume they are buying. (And I suspect most retailers assume they are selling an Audio CD.)
Figure 6 Liszt - Mephisto Waltz Reference Recordings RR-82CD
Figure 6 shows an example of a recording of the Liszt Mephisto Waltz that was made by “Prof Johnson” who is a co-inventor of HDCD. The results clearly confirm that it includes the use of both soft peak limiting and low level extension. It is a good recording and sounds excellent, both when played as a normal CD and as HDCD. In this case the low level extensions are applied to a significant amount of the music which has some long low-level passages. Comparing this with the Joni Mitchell example where the HDCD sounds good, but seems compressed when played as a CD, it implies that the HDCD process needs to be applied with great care and judgment to be able to successfully squeeze a pint-and-a-half into a pint pot without the result sounding unfortunate on CD. Again, excellent as this example sounds, it does concern me that a normal CD made with just a 3dB reduction in overall level might have sounded as good as HDCD. But I can see more sense in applying HDCD to such ‘classical’ material than to much rock/pop which often tends to have a relatively limited dynamic range. There, poorly judged added peak compression may be a bigger minus for most listeners.
Figure 7 Kodaly - Dances of Galanta
In the case of the Claire Martin track recorded by Linn Records the result shows signs of HDCD soft peak compression, but not of any low level gain changes. However figure 7 shows a slightly odd example, taken from just the first track of CKD234. Although given an HDCD logo the track shows no sign of being encoded for any dynamics alterations. (This is indicated by the flat line at -6dB when we compared Played as HDCD with Played as Audio CD). In a few places the ‘blind’ peak expansion assumes a peak above -3dBFS on the CD should be expanded. This produces the spikes in the bottom plot in Figure 7. This implies that these blind expansions are presumably incorrect and aren’t specified by embedded control codes. This track shows no sign of its dynamics being changed by HDCD.
At the same time as I purchased Joni Mitchell’s Don Juan’s Reckless Daughter I also bought two other Joni Mitchell ‘Audio CDs’. These were Blue (Reprise 7599-27199-2) and Both Sides Now. Although Don Juan gave no external visible indication of being an HDCD, looking carefully at these I found that both have an HDCD logo on the back.
Blue is, indeed, an HDCD. Using the same programs that gave the above results I found that this disc has a fairly light application of HDCD soft peak limiting. So the dynamics of some peaks are slightly increased when its control codes are detected and obeyed. But the alterations are slight and seem to have little audible impact. However when I analysed Both Sides Now I found no sign of any HDCD control codes. And when processed all that happened was that the overall level was changed by -6dB – just as if it were a normal Audio CD. Despite the logo, this disc seems not to be an actual HDCD.
Figure 8 Joni Mitchell - Both Sides Now (Comes Love)
This is quite strange as the disc is also measurably peak level compressed as can be seen by looking at Figure 8. This shows the statistical distribution of how often a given peak level occurs during track 3 (Comes Love) on this disc. The left-hand plot shows the dynamics when the disc is simply treated as a normal Audio CD. The striking feature here is the sharp upward peak as the we reach 0dBFS. This tends to be a sign of clipping and/or peaks being compressed. Usually indicating that the CD was made at too high a level. Now if this were the HDCD soft peak limiting in action, and this was correctly indicated by the relevant control codes, then the HDCD analysis would ‘expand’ this squashed part of the distribution to occupy a range that extended over more like 6dB at the top of the pattern. But the graph on the right shows this simply doesn’t happen.
When I tried blind peak expansion on this disc I got mixed results. Some of the time the result gave a more ‘natural’ dynamics distribution and sounded better. But in other places some peaks sounded distinctly nasty and had obviously been distorted by blind expansion. So is this actually meant to be an HDCD, as labelled?... I can only guess. But my suspicion is that the material may have been HDCD processed at some point. But was then altered afterwards before the result ended up on the disc.
One vital requirement of HDCD is that the control codes that allow an HDCD player to ‘correct’ the alterations made when mastering the disc must be preserved. Any alterations to the sample values after the HDCD process will ‘scramble’ the codes and HDCD control will be lost. Now it may be that this disc never was HDCD at all, and the presence of the logo is an error. But an uncomfortably plausible alternative is that it was mastered as an HDCD but someone else then took that and decided to ‘improve’ it (or so they thought!) in some way. In the process, destroying the control codes. And any alteration may also tend to slightly derail the process of even simple blind peak shape correction as the values were altered again after the nonlinear HDCD changes!
I can’t tell if this is the case or not. But it would be consistent with what I found. Mind you, many ‘normal’ pop/rock CDs have had their peaks squashed because of the mindless obsession of some in the music biz with LOUDNESS. Blind peak expansion might make some non-HDCDs sound ‘better’ simply by reducing the severity of the waveform compression.
Conclusions... that generate more questions than answers!
So is HDCD worthwhile in practice or not? I suspect this depends on your personal starting preferences and assumptions. But to me in present circumstances it does look doubtful.
On the positive side, some HDCDs do sound excellent when played as HDCD and still sound good when played as a normal CD. And some discs – e.g. Don Juan’s Reckless Daughter – do sound better to my ears when played as an HDCD than as a CD. However I can’t help feeling that it might have sounded just as good if it had simply been mastered onto CD at a slightly lower level so that the peaks hadn’t needed to be HDCD squashed to fit. That way the sound would have been available to a wider audience to appreciate. As it is, people who play the disc on a normal Audio CD player may simply find the sound has been degraded by the peak compression they get as a side-effect of the way the HDCD process has been applied. Particularly unfortunate if examining the outside of the box before purchase gives the buyer no sign that a disc isn’t a normal CD. As it is, those who can play HDCD may welcome the disc if they realise what it is. But those who without an HDCD player might need to be more cautious.
In some other cases the HDCD seems to add little, at the expense of producing some peak compression for those listening as a CD. And the existence of HDCD may also expose us all to oddities like discs that say they are an HDCD, but then don’t decode as one, due, perhaps, to errors and oversights by those making the disc.
In practice this seems to be something we’d have to assess case-by-case as a decision based on a mix of how well the particular HDCD was mastered, and what the balance of advantage for HDCD replay versus impact on CD replay will be. The problem being how any buyer would know any of this in advance...
On a more basic level there are some more hard-to-answer questions. Is an HDCD really ‘compatible’ with Audio CD? For me the answer would be that it depends on how heavily features like peak soft limiting and employed. However given that I found that the increases in dynamic range were all far less than the 24dB implied by the claim of 4 bits improvement. And that for most rock/pop music the low-level gain extensions would hardly ever be used... Is the game worth the candle give the possible effect when most people only have normal CD players? In practice, for much popular music you could probably only approach even 6dB (i.e. just 1 bit) of range extension by making a disc that sounds distinctly compressed when played on a normal CD player. A bargain made even odder by having control codes added to the least significant bits, reducing the range in the process!
If every maker of Audio CDs and players had adopted HDCD the situation might have been different. Then there would be no concern that most people have a non-HDCD player and might get a degraded result. But that isn’t the situation we have reached. Instead we have people moving to downloading files and bypassing CD entirely. If Audio CD is a weak link, then people have found better ways to get around it.
Beyond that is an even more fundamental question. Since HDCD is now a rarity and most audiophiles are tending to move to ‘high resolution’ files, is the entire idea outmoded? It never really caught on in the first place. Its status may matter most to those people who have a small collection of HDCDs who wish to go on being able to hear them as HDCD in the future. New high-quality HDCD players are scarce. And the main patents will lapse in a few years. Alas, since many vital details of how the full system works seem to be ‘commercial in confidence’ that may mean playing discs in the future may become even more of a problem.
And when considering patents, another question springs to mind. In the July 2012 issue of Hi Fi News magazine (on page 99) Keith Howard relates an interesting story. This is about Ed Meitner’s patent for ‘adaptive filters’. Apparently no-one reached any agreements with him about using his idea that the filter used when recording or playing digital material should change according to the nature of the material. However his idea strikes me as pretty similar to the idea that HDCD might include a choice of filters that change according to the material. Has HDCD in practice ever actually used this, as claimed in the main HDCD patent and paper? If so, why no agreement with Ed Meitner? Or has it never actually been used – perhaps because it might require such an arrangement? I have no idea. Once again this seems to be an issue that isn’t clarified by the public information I have been able to find.
Which I must admit sets me wondering? Are the HDCD patents actually valid? It can be argued that they lack the required details that a patent should contain to be valid. And there have been various cases where a patent has been challenged successfully in court. But who would bother now to contest this or be able to ensure that the necessary details are made public so that anyone who owns HDCDs can play them once the patents lapse?
On all the above, I guess you have to pay yer money and take yer chance. And hope that what you choose is what you think it is!
