Issue with multi processing and thread sync

[Henrizzle]

Hello,

Not sure who is reading this. So it was early on this year when I was remixing some old songs and I noticed sometimes certain tracks with volume automation would disappear in the render with export. Then recently I notice lots of pops and scraps. Today I notice pops and scrapes were making its way into the export. So after some thoughts and experimenting I disabled multi processing. Now the export works correctly, and oddly enough the tracks sound cleaner and the bass is better, so maybe even if it wasn't something hugely noticeable it had some artifacts with multiprocessing.

So with this fact one can deduce that either thread synchronization is not correctly implemented or buffers are not allocated and aligned to cache size thus possibly causing corruption when two or more threads performs operations on a cache page that includes buffers from more than one source.

I hope this makes it to the developers, I can provide projects that show this issue. If this can't get solved I will have to use a different DAW. I have used Cakewalk since the late 90s and I think it is great until now.

Thanks!

Henrizzle

 

[bvideo]

Hi Henrizzle,

"... disabled multiprocessing ..." How exactly?  Did you do that in the BIOS? Did you disable "plug-in load balancing"? Did you set CPU affinity to just one processor? Something else? (I ask with curiosity because of a prior experience of mine.)

Also, do you have "64-bit Double Precision Engine" disabled or enabled? Do you use an audio buffer size of 256 or greater or smaller? Do either of these changes make a difference?

Does Sonar after September 2016 do the same?

To reach Cakewalk support directly, see "How do I report a problem with a Cakewalk product?". It says "please contact us at support@cakewalk.com."

[Henrizzle]

Hi bvideo,

You can disable the multiprocessing from the advanced preferences menu. In this case the rendering routines are "step locked" as it can only execute serially. You can see in the performance meter there is only one processer in this case and it can no longer render in real time with large projects due to CPU load.

It doesn't matter buffer size, I have the offline render buffer set to largest possible. The offline render or "export" does not care about buffer size as this is not done in real time and takes as long as it takes to complete the render. If anything, I think the artifacts will probably be bigger and more noticeable with bigger buffers as they are likely to be buffer sized.

64 bit precision on or off has same issue.

I have tried everything, in the end I have deduced the multiprocessing is the problem as I can get consistently a good mix using single processing.

I have installed my old Sonar X1 producer, unfortunately somehow some of the plugin GUIDs are different so it can;t find them while they are clearly available in the menu, so I can't just load the same project to get the same mix. It will take me awhile to build the same project using all the same plugins... so I will try that later after I finish some tasks at hand.

Currently I am exporting my catalogue of songs using single processing, this is the only way I can be sure there are no artifacts, overall the mixes sound a bit clearer, I'm guessing even when there are no huge artifacts to notice there are small ones that typically effects how tight the bottom end is as low frequencies have long waves and are particularity vulnerable to small imperfections.

Edited July 24, 2023 by Henrizzle
Additional thoughts.

[Max Arwood]

What cpu, memory and sound card? How many tracks? Any virtual instruments? Have you frozen any tracks to lower cpu usage?

[Henrizzle]

I have been a software engineer for over 25 years. I can tell you this is a problem with sync between the render threads as this is a non-real time render for "exeport". So CPU, sound card does not come into play here. Rather there are sync objects like semaphores and mutex to sync threads that are not implemented properly or more than one buffer is on the same cache page being flushed by different threads at the same time.

These projects are hollywood style mixes with all real instruments so there are lots and lots of tracks and busses. Currently this project, I am looking at 49 tracks and bus assignments out to "AC". There is native plugins as wells as UAD2 DSP. None of this is the issue as the single processor render does it perfectly.

Edited July 24, 2023 by Henrizzle

[Henrizzle]

BTW I installed my Sonar X1 Producer again... OMG this UI is an eye-sore. The new Cakewalk UI is so much easier to look at HAHA!

[scook]

Installing older versions of SONAR and CbB may overwrite shared files needed by the newer versions of the DAW.

When installing SONAR and related DAWs, the most recent version should be (re)installed last.

 

[Henrizzle]

I got both latest Cakewalk and Sonar running on same system. Only issue is Sonar makes reference to GUIDs differently for some plugins so it can't find them while they are clearly there in the FX menu. Anyhow, I don't wish to tackle this GUID issue at this time as there is no need to. I'm exporting my catalogue of songs using single processor as there is no telling whether there were small artifacts from the multi processor exports I did before... that's REALLY scary. You don't want to find out there are artifacts after you've published them and hear them on streaming service :0

Edited July 24, 2023 by Henrizzle

[Henrizzle]

Okay I sent a message to support@cakewalk.com lets see how it goes.

[bvideo]

There are differing degrees of multiprocessing. One special case is "plug-in load balancing", where the chain of plugins on a single track is handled by multiple processors, one per plugin.  I was wondering if you tried disabling that. I'm sure one of my projects had a problem with it, but only with certain plugins and certain settings thereof. I asked you about buffer size because plug-in load balancing is disabled when the buffer is smaller than 256 (by default). So I am still curious if you have tried those things.

Aside from plugins and audio drivers, the core of Cakewalk & Sonar is shared by dozens (hundreds? Thousands? ???) of users, not many reporting playback or rendering corruption. That's why I'm asking you these questions about a corner case I believe is there, and has been there since plug-in load balancing was introduced. (My early report).

[Henrizzle]

Hi bvideo,

I tired it all, with and without plugin load balancing. I even edited the aud.INI to set AllowOfflineMixThreads to TRUE and FALSE and it didn't actually make offline render single thread.. the setting doesn't work. I thought at least I can make the offline renders single thread without having to change the multiprocessing to single by using this option... but nope... so there are lots of things that doesn't work it seems.

I think there are not many users that actually run that many tracks and busses and plugins to show the problem. While others are getting little artifacts without knowing it. I did a binary compare of the output between single and multiprocessor, even when you don't hear oblivious artifacts there are differences in the output binary. So I'm still doing some research, it's possible that mastering is using a random number generator to do dithering for 24 -> 16 bit that could account for the differences. I think the next step is to take one of the stock demo projects and render in multi and single processor then do a diff of the output.

I'm out of time today. Was trying some mixes in Studio One, so far it seems good but I compared the data between source -> output -> input, it seems to have some filtering that changes the data more than I think it should when there are no plugins or any sort of processing at all. I ran the same test on Cakewalk... all same hardware, the result is a a little more reasonable.

Here are the specs for the PC, this is offline rendering at this point so ASIO driver is not to blame but I am using Orion32HD Gen3.

Device name    WIN11_20230609
Processor    Intel(R) Core(TM) i9-9900K CPU @ 3.60GHz   3.60 GHz
Installed RAM    64.0 GB
System type    64-bit operating system, x64-based processor

Edition    Windows 11 Pro
Version    22H2
Installed on    ‎6/‎9/‎2023
OS build    22621.1992
Experience    Windows Feature Experience Pack 1000.22644.1000.0

49 tracks, 27 busses, about 90+ plugins, some native some UAD2 DSP.

Edited July 25, 2023 by Henrizzle

[Henrizzle]

I also have to mention. Other issues with multi processor rendering is that sometimes when you have gain automation on a track for example, the track disappears. I noticed this first time earlier this year, those I fixed those by removing the automation and editing the track. This also point to where the render threads are out of sync.

Edited July 25, 2023 by Henrizzle

[Max Arwood]

I run 100 tracks 150+ plugins and have had buses to AT. I could not render my songs without using multi processors power. I would suggest you set your buffers to 2048, turn on multiprocessor, try 64 bit both ways  off an on. There are several people here that do professional orchestration and movie soundtracks on older computers than what you use. Sometimes there are  unruly plugins that can cause odd problems. Lurssen Mastering, Ozone, AR TG mastering plug-in. There are others as well. I hope we can help you figure this out. Scook is great at figuring things out. 

Edit: Bitfilpper is great at figuring things out too!

Edited July 27, 2023 by Max Arwood
Added Bitfilpper!

[bitflipper]

Henrizzle's said he's already got buffers at max, and has already experimented with 64-bit option off and on. I doubt he's going to be able to configure his way out of this.

I'm confused about the assumption that this is a thread synchronization issue. I know how semaphores and mutexes work, and they exist to negotiate resource contention, race problems, memory and I/O bandwidth restrictions - basically real-time issues. They should not be a factor in normal rendering, especially in a project that's almost entirely audio. 

Perhaps a more telling symptom is the dropouts in the exported audio. This shouldn't happen. In 20 years using Sonar/CbB, I have experienced dropouts while recording, dropouts while playing back, but never in an exported file. I agree that this is data corruption, but it's happening at a lower level than the application.

Henrizzle, have you tried a real-time render?

[Xoo]

12 minutes ago, bitflipper said:

Henrizzle, have you tried a real-time render?

Have you also tried a disk and memory check?

[Max Arwood]

How many virtual instruments are you using? How many channels are you using in them?

I have had glitches in exports. Computer maxed out with plug-ins and virtual instruments. Remedy - Freeze all virtual synths and tracks with plugins. Your bus can also put a strain on your CPU. No way to freeze the bus, but you could do a stem export and bring the stems back into a final project file. Real time helps when you are at the top of your processor power. I couldn't imagine an i9 straining on anything. 

Hopefully one day we can freeze folders \ buses without having to do it manually. 

 

[Henrizzle]

No virtual instruments, all PCM tracks. Real time render is worse, you can her the pops and scrapes. This is not a voodoo do a disc check sort of thing, it is clearly related to multiprocessing threads synchronization.

 

Edited July 30, 2023 by Henrizzle

[Larioso]

"Thread syncronization", I am thinking different thread scheduling models in settings Configuration File.

- 1=wait for single object type, which is legacy as stated

- 2= I believe is default since long

You did not mention as I read, so just an idea to troubleshoot for devs to get more info...and maybe solution not to handle rendering different than mixing and the rest....

 

I got a new 16+4 core i7-12700F beginning year, and had some crackling issues and also tried running different number of threads/cores in Sonar, also in that config at top.

- I would try 2 cores, 4 cores and something like that to narrow down when issues appear

- setting is MixThreadCount but really is number of cores to use

- it works directly as audio engine starts exiting preferences, so no need to restart Sonar/Cakewalk

 

My solution was to get another usb 3 interface as pci card, so computer had two usb root hubs. So no problem with Sonar at all.

- newer computer mb seems to use Intel usb 3 controller, which makes a single usb root hub only

- just stating for clarity, nothing to do with current OP

Edited July 31, 2023 by Larioso

[Jorge Arias]

On 7/26/2023 at 4:53 PM, Max Arwood said:

I run 100 tracks 150+ plugins and have had buses to AT. I could not render my songs without using multi processors power. I would suggest you set your buffers to 2048, turn on multiprocessor, try 64 bit both ways  off an on. There are several people here that do professional orchestration and movie soundtracks on older computers than what you use. Sometimes there are  unruly plugins that can cause odd problems. Lurssen Mastering, Ozone, AR TG mastering plug-in. There are others as well. I hope we can help you figure this out. Scook is great at figuring things out. 

Edit: Bitfilpper is great at figuring things out too!

Among the problems with plugins is the recent one with those from Soundtoys, which were temporarily corrected with a beta (5.4.2) and which produced similar problems during playback and recording.

[David Baay]

On 7/24/2023 at 10:35 AM, Henrizzle said:

There is native plugins as wells as UAD2 DSP

This issue is almost certainly plugin-related, and eliminating UAD from an offending project would be a good place to start troubleshooting. Cakewalk has had various issues with UAD over the years, at least some of which had to be fixed by UAD as I recall.

On 7/24/2023 at 10:37 PM, Henrizzle said:

it's possible that mastering is using a random number generator to do dithering for 24 -> 16 bit that could account for the differences.

Again, I would be more inclined to attribute any variability in the output file to FX plugins that have random elements, intentional or otherwise. 

Standard operating procedure for troubleshooting an issue like this is to start stripping the project down until the problem goes away. A good way to do this with a big project is to take a binary approach: Disable half the buses/tracks/plugins and check the export to determine which half has the issue, then disable half of that half, etc.

One way or another this is most likely to be a project-specific issue that can only be properly diagnosed by getting hands on with the project and all the plugins (and the UAD hardware if the problem can't be reproduced without it).