Slow project loading

[satchmo-x]

I believe this is a bug, so I'm wondering if anyone else is having this problem or can reproduce it. I recently upgraded my computer (December 2022), audio interface (March 2023), and the Cakewalk 2022.09 version was released shortly before that (2023.09 has the same problem). Somewhere in that timeframe, project loading times changed. It's now very slow, with huge spikes in CPU load until Cakewalk finishes opening the project, accompanied by a high ramp up of the AIO cooler fans. This happens with all newer project when opening from the saved copy, even for projects with zero content. After the project is finished loading, everything seems normal. It takes almost 20 seconds to load, which may seem quick, but the computer only takes about 15 seconds for a "full" Windows boot from power on (fast start disabled). Also, old projects (pre ProChannel?) load almost instantly.

I suspect it's related to how Cakewalk or ProChannel and the new CPU are interacting.

CPU: Intel i9-13900k (upgraded from Intel i7-6850k)
MoBo: ASUS Maximus z790 Hero (upgraded from ASUS X99-DELUXE II)
Audio interface: RME Fireface UFX III on USB3 port (upgraded from Focusrite Thunderbolt Clarett 8Pre)

To reproduce:

- Open a new  project from one of the default templates (Basic used for the example file, but I usually use the 4-track tempate)
- Save the project. (normal or bun, doesn't matter)
- Close the project.
-Open the project.

The new project, save project, and close project steps all take less than a second each. The open takes about 19 seconds with a high fan ramp up on the AIO cooler as noted above. I never experienced this on my previous computer.  Project loads were very fast before the upgrades, I believe around 2 to 5 seconds if I recall correctly.

I've attached a BUN file I just made using the steps above which exhibits the problem on my system.

Thanks for any replies or insight.

Regards,
Tim

slow_test_bun.cwb

Edited October 1, 2023 by satchmo-x
Added a word for clarity

[gustabo]

Loaded it and it loaded in about a second on an  i5-8265U CPU @ 1.60GHz laptop with 8gb ram.

[OutrageProductions]

Ditto with @gustabo re: <1sec timing on i9 @3.4Ghz desktop w/64Gb RAM.

[John Vere]

Have you ever run Resplendent Latency Monitor?  That’s a good start to troubleshoot computer issues. 

[satchmo-x]

5 hours ago, OutrageProductions said:

Ditto with @gustabo re: <1sec timing on i9 @3.4Ghz desktop w/64Gb RAM.

What model i9? I'm wondering if the efficiency cores the 13900K has and the way they go in to sleep/parked mode and/or the way all the cores ramp up and down is part of this. 13900K has been a weird CPU in that regard. I had to cap the wattage in the BIOS to keep it sane regarding it's tendency to thermal throttle cores.

More testing continues to point at ProChannel being involved, as well. I tried opening a bunch of old projects without ProChannel, and they all load in 1-4 seconds, depending on the complexity.

2 hours ago, JohnnyV said:

Have you ever run Resplendent Latency Monitor?  That’s a good start to troubleshoot computer issues. 

Thank you for the suggestion. This isn't a latency issue, though. For most projects, I'm able to run stable at 2.8 ms round trip with this setup (32 sample buffer at 48KHz). The problem I'm having has only to do with long load times for previously saved projects. Once open, those projects seem to run fine, even at low latency.

[Will.]

We've all had this issue before. I certainly have at least twice. 

[satchmo-x]

5 hours ago, Will. said:

We've all had this issue before. I certainly have at least twice. 

This is happening on all newer projects (I think with ProChannel) every time a project is open since the day I got Cakewalk running on this computer. So, 6-7 months of consistent, repeatable behavior of nearly 20 second project open times on a CPU that is in theory 5 times more powerful than my system that didn't do this. On some projects, the status window on the bottom left sticks at "Creating UI". Lord Tim's "Time to Fly" is one that fits this pattern, with 2 seconds of all the other messages, and 18 seconds on "Creating UI". On other projects, it sticks at "Almost done" for 18 seconds. The included "Living for Now" demo from Sophia Jane does this, as does the blank project I attached above.

That leans to indicating a weird interaction with the software and the new parts of this setup. I suspect it's the 13th gen Intel i9 with efficiency cores. I can't swap to the old interface, as the Thunderbolt 4 ports on my new motherboard aren't backwards compatible with Thunderbolt 2 devices to eliminate that.

[John Vere]

You should have looked. It’s not for audio latency it’s for system latency. It’s a standard tool everyone uses to see how their computer is doing. 
 

There are other tools as well. I had same issues last spring are all of a sudden everything seemed slow to load or even play.
It turned out my SSD data drive where Cakewalk projects were stored was slowing down. I ran the Samsung Magician which determined that. With SSD drives it’s not how full they are as it’s  how many GB have been written and deleted.  
$50 each for a couple of 500 GB Evo drives The Magician will clone your old drives in less than 20 minutes and my computer is now working better than ever. I’m going to do this annually from now on. 

Edited October 2, 2023 by JohnnyV

[Will.]

1 hour ago, satchmo-x said:

This is happening on all newer projects (I think with ProChannel) every time a project is open since the day I got Cakewalk running on this computer. So, 6-7 months of consistent, repeatable behavior of nearly 20 second project open times on a CPU that is in theory 5 times more powerful than my system that didn't do this.

You might have a faulty disk. My issues where my projects load long was always with a plugin. I had since deleted that effect permanently. 

You can test this by pressing the bypass shortcut for effects with the lettet "E" on your keyboard, save the project and reopen it in that state with all your effect bypassed. 

If it is a favorite synth: make a copy of the project and just replace the  the synth with any other one save that project and reopen it again until you find the culprit. 

[satchmo-x]

2 hours ago, JohnnyV said:

You should have looked. It’s not for audio latency it’s for system latency. It’s a standard tool everyone uses to see how their computer is doing. 
 

There are other tools as well. I had same issues last spring are all of a sudden everything seemed slow to load or even play.
It turned out my SSD data drive where Cakewalk projects were stored was slowing down. I ran the Samsung Magician which determined that. With SSD drives it’s not how full they are as it’s  how many GB have been written and deleted.  
$50 each for a couple of 500 GB Evo drives The Magician will clone your old drives in less than 20 minutes and my computer is now working better than ever. I’m going to do this annually from now on. 

I'm familiar and have used it. I don't think that's the issue, but since you believe there may be something there, I ran it while looking atSamsung Magician, getting screen shots, browsing the web, watching YouTube, connecting to AWS Workspaces virtual desktop, and opening and closing multiple Cakewalk projects that exhibit the problem. Also while this capture of LatencyMon was running, I updated Samsung Magician, as it was very out of date since I don't let the service run at startup. That's likely the reason for all the pagefaults. Attached is the report with my computer name removed, as well as the Samsung Magician screens for the OS/System and Music drives.

EDIT: Samsung Magician 8.x is a no go on my box. It's the reason for the page faults. Tried multiple times, and it installs, but the UI won't launch, sticking in a page fault crash loop. I tried the --disable-gpu-sandbox switch, and still no joy. I re-installed Magician 7.2, and it works fine again.

latency_mon_20231002.txt

Edited October 2, 2023 by satchmo-x

[OutrageProductions]

12 hours ago, satchmo-x said:

What model i9? 

9900k

I've researched well the newer procs, 11th &12th gen are OK, 13th showed significant code anomalies. At least untill Win11 gets its act together a little more. Win 10 is rick solid for me, so I'll only upgrade kickin' & screamin'.

[satchmo-x]

1 hour ago, Will. said:

You might have a faulty disk. My issues where my projects load long was always with a plugin. I had since deleted that effect permanently. 

You can test this by pressing the bypass shortcut for effects with the lettet "E" on your keyboard, save the project and reopen it in that state with all your effect bypassed. 

If it is a favorite synth: make a copy of the project and just replace the  the synth with any other one save that project and reopen it again until you find the culprit. 

My install doesn't have that keybind, but I tried bypassing all effects on the 2 track template using the "FX" button. It still has the issue. So, I created a new project from "Empty Project" that starts with no tracks and all effects bypassed, and it still takes 18 seconds to open. By default, ProChannel is on Master, Metronome, and Preview. I don't know how to remove ProChannel from tracks or busses, so I deleted all the busses. It still takes 18 seconds to open. I open another old project from pre-ProChannel days (created in Sonar Producer 8 or prior), and it opens in about a second.

Attached are bundles for empty project, and empty project with busses removed. Both take 18 seconds to load. I moved the bundles to another drive besides the Music drive, and had them create their folders on that other drive, as well. Still 18 seconds to open.

slow_load_empty_project.cwb slow_load_empty_project-no_busses.cwb

[satchmo-x]

45 minutes ago, OutrageProductions said:

9900k

I've researched well the newer procs, 11th &12th gen are OK, 13th showed significant code anomalies. At least untill Win11 gets its act together a little more. Win 10 is rick solid for me, so I'll only upgrade kickin' & screamin'.

Thanks for the CPU info. I'm still suspecting something with 13th gen Intel/efficiency cores combined with newer Cakewalk projects (ProChannel being at the top of the sus list).

I'm sticking with Windows 10, too. That's a risky jump for a system I need to be stable.

[mettelus]

21 hours ago, satchmo-x said:

accompanied by a high ramp up of the AIO cooler fans

This is the most concerning part of your post. Have you overclocked your machine at all? I have a Corsair cooler and had to disable the Corsair LINK 4 app because that was actually tweaking the UEFI on its own and setting targets well above what the CPU was capable of (like 5.3GHz) even if it had been delidded, which it is not. The machine was cycling high and low like a roller coaster till I disabled that app. Every time it took off the fans would kick on in about 3 seconds. Based on that experience, check to see if you have a cooler app running. After disabling that and giving control back to the ASUS MOBO, I have had no issues. I picked up on that by checking the UEFI manually, and it kept getting changed to a value I had definitely not set for it.

For me, that bun file unpacked and loaded in about 2 seconds, saved as a cwp, closed and reloaded, and was the same.

[satchmo-x]

49 minutes ago, mettelus said:

This is the most concerning part of your post. Have you overclocked your machine at all? I have a Corsair cooler and had to disable the Corsair LINK 4 app because that was actually tweaking the UEFI on its own and setting targets well above what the CPU was capable of (like 5.3GHz) even if it had been delidded, which it is not. The machine was cycling high and low like a roller coaster till I disabled that app. Every time it took off the fans would kick on in about 3 seconds. Based on that experience, check to see if you have a cooler app running. After disabling that and giving control back to the ASUS MOBO, I have had no issues. I picked up on that by checking the UEFI manually, and it kept getting changed to a value I had definitely not set for it.

For me, that bun file unpacked and loaded in about 2 seconds, saved as a cwp, closed and reloaded, and was the same.

Thanks for the reply and testing the file. The 13900k is weird... it basically internally automatically overclocks and underclocks, parks cores, etc. It's really interesting to watch in HW Info in general and during benchmarks. Single cores are able to hit thermal limits without damage and they throttle individually, and that's actually how it's supposed to work. Throw on ASUS "magic sauce" motherboard stuff, and it gets even weirder. I've built a lot of computers over the last 30 years, from desktops to high end servers, and I've never seen anything like it. It freaked me out at first until I researched it. The only overclock I have on is XMP and whatever the CPU does automatically internally. I limited the CPU to 125W in the BIOS, and that tamed 95%+ of the AIO ramp up that would happen constantly. I couldn't have it doing that during recording, and that setting fixed it. It still ramps up when cores are pegged, though. It's just how this CPU is supposed to work :: shrug ::

On that note, when watching Task Manager while opening any newer CW project, a single logical processor on one of the hyperthread capable performance cores is pegged while the AIO is ramped up, and all the other 22 1/2 physical cores/31 logical procs are almost completely idle, as shown in the attached screenshot. So, I'd guess a single thread in CW is putting a lot of processing in to loading... something(?) related to ProChannel or something else in the newer project files? Figuring out how to distribute the load across cores? I honestly don't know. It's just weird that it's only with newer projects.

EDIT: As a note on general behavior, once loaded, when playing Lord Tim's "Time to Fly" at 32 sample buffer, the AIO fans stay at the silent setting. I do get a late buffer or two, but dropping to a 64 sample buffer results in zero late buffers, engine load sits around 25-30%,  and the CPU looks like it's not working hard. That project is a great acid test.

Edited October 2, 2023 by satchmo-x

[mettelus]

Even if distributed, there are processes that have to run end-to-end on one core/thread, so you may be seeing one of them. That CPU spike is nutty though, and much like what I saw when my cooler app was enabled (just on all cores for me). You may want to try something simpler at first, like going into the UEFI and saving your current profile, then go into EZ Mode and just set that to normal and see where that lands you on performance. I tend to default to 1.03 for XMP settings anymore, since once overheating begins I have seen a slew of page faults get thrown (LatencyMon will see these too). Once that starts the performance tanks out fast, so you may be seeing the computer trying to decipher the gibberish that one core is throwing at it.

[satchmo-x]

5 hours ago, mettelus said:

You may want to try something simpler at first, like going into the UEFI and saving your current profile, then go into EZ Mode and just set that to normal and see where that lands you on performance.

Good suggestion. I ended up with weird results. I updated my BIOS first, as it was several versions behind. Bad troubleshooting practice to introduce more than one variable, but it seemed to... well, it certainly changed things. I now wish I hadn't done that first and just played with the settings in the old BIOS.

So, after the BIOS update, I've tried with:
- default BIOS settings, which sets  Multi-Core Enhancement/MCE, i.e. the over-wattage setting, to "Auto" which is basically "On". This is ASUS's recommended setting.
- normal, but with MCE disabled. This is the 125W cap for the CPU, which is really 125W base, 253W boost. This is Intel's recommended setting
- I tried both of those MCE settings with XMP Enabled and also Disabled.

All of those settings don't seem to make any difference. Behavior is the same in all 4 setting combinations. The BIOS update seems to have changed things up, though. Now, new projects take 18 seconds to load the first time after I open Cakewalk, but then, they potentially open fast 1 to 4 times after that, then they'll take 18 seconds again on that 2nd, 3rd, 4th, or 5th time. Doesn't seem to be a rhyme or reason to it. I can open different projects or the same one 5 times in a row, rapidly or with some delay on opening/closing/both. It just seems random when it's going to open slow again. Closing/reopening Cakewalk will always make the project load slow again. I watched the CPU temps in HW Info, and surprisingly, the cores didn't even approach overtemp or thermal throttle. The max was 73C, and these cores don't throttle until 91C. Old Producer 8 and older projects still open quick every time. 

LatencyMon is now indicating cakewalk.exe as the highest page fault offender, but this is just during file open/close, not playback:

- Process with highest pagefault count:                 cakewalk.exe
- Total number of hard pagefaults                       862
- Hard pagefault count of hardest hit process:          302
- Number of processes hit:                              13

[OutrageProductions]

Silly question here... but;

if you have spinning drives, have you inspected the fragmentation lately?

And if SSD, what is their health/error rate?

[satchmo-x]

1 hour ago, OutrageProductions said:

Silly question here... but;

if you have spinning drives, have you inspected the fragmentation lately?

And if SSD, what is their health/error rate?

Not silly, I appreciate the attempt to help.

All Samsung SSD, no noisy slow spinning disks allowed on my workstations.  I also try to watch the TB written and minimize it. All drives have overprovisioning enabled.

C drive (system/OS/music applications): 2 TB 980 Pro w/heatsync, 9 of 1200 TBW. All SMART stats in "Good" state with zero errors.
D drive (music projects, libs, some plugins): 1 TB 850 Pro, 2 of 300 TBW. All SMART stats in "Good" state with zero errors. Zero relocated sectors.
N drive (EastWest Hollywood Orchestra library/old game drive): 1 TB 850 Pro, 34 of 300 TBW. All SMART stats in "Good" state with zero errors. Zero relocated sectors.
E drive (new game drive): 4 TB 860 Evo, 36 of 2400 TBW. All SMART stats in "Good" state with zero errors. Zero relocated sectors.

I'm about to replace the music drive with a new 4 TB 870.

[mettelus]

14 hours ago, satchmo-x said:

LatencyMon is now indicating cakewalk.exe as the highest page fault offender, but this is just during file open/close, not playback:

- Process with highest pagefault count:                 cakewalk.exe
- Total number of hard pagefaults                       862
- Hard pagefault count of hardest hit process:          302
- Number of processes hit:                              13

Wow, I cannot recall cakewalk itself ever throwing page faults on me, so that is just plain weird to see. Did this also happen with the 22.09 version or is it specific to the 23.09?  @Jim Roseberry is a good person to reach out to as well, since he custom builds the newer machines and keeps up with the intricacies of them. I am certainly not up to date with the i9s, but he knows the details for tweaking them.

Edited October 3, 2023 by mettelus