[Fixed in v7.1] v7 upgrade problem with Sync

[Pauven]

I've read through your write-up, and I have the impression that you are a bit confused on how data moves around. So let's try to level set.

For CME:
NAS <-> MM <-> MM API <-> CME --> NAS <-> CMC

CME pings the My Movies API for data, most of which comes from the MM DB, but MM will scan your NAS folders as well to retrieve the Dune info. CME then writes this data (XML and artwork) to your NAS media folders. Later, when CMC does a folder scan/sync, it reads this data from your NAS media folders, pulling in the data that was written by CME.

In this case, CME and CMC both only work if your NAS is online. If your NAS is offline, then at best CMC will use previously scanned results, but I believe this only works if the "Keep Offline Data" is checked in CMC's settings (this was created for users who put their NAS to sleep), so doing a CMC sync with your NAS offline would result in either errors or your collection disappearing.


For CCC:
NAS <-> MM <-> MM API <-> CCC <-> CMC

CCC pings the My Movies API for data, again most of which comes from the DB, but MM will scan your NAS folders as well to retrieve the Dune info (usually twice, though, because CCC pulls additional data via the API). CCC then writes this data to a local CCC database. CCC never accesses your NAS directly. Later, when CMC syncs with CCC, CCC simply shares it's local DB data and artwork with CMC (either just the changes or the full DB as-needed, both super fast).

In this case CCC and CMC both work fine with your NAS offline (only playback needs it online).

This loops us back to your test with the NAS offline and the CCC export being super quick. My Movies was still trying to scan your media folders for the Dune data, but because your NAS wasn't reachable this step failed super quick, and the export continued. You can see how much faster this was without the folder scans. And CCC didn't care, because CCC discards the Dune data anyway, so the end result is essentially the same, only much much faster.

Now to be clear, I'm not recommending you have your NAS offline, since My Movies sometimes flags your collection as missing if it can't get to it, so this is dangerous territory. But it does perfectly illustrate that the API performance penalty is the folder scanning for Dune info, which I've asked Binnerup 3 years ago to provide an option for disabling. This issue affects some setups worse than others, and for some reason it seems especially bad on your setup.

CCC v7 did include a number of fixes/enhancements that result in it pulling even more data than v6, so it's possible that this made v7 even slower than v6 - again this isn't CCC being slower, simply the MM API being taxed harder and working slower. Though for the most part, I think it should perform very similarly to v6.


Regarding your performance delta between CME and CCC, this is much harder for me to explain - or even comprehend. Yes, CCC should be a little slower, but not that much slower. I use a shared code base for CME and CCC, it's 90% the same program, the main difference is that CME writes to your media folders, and CCC writes to it's own local DB. Additionally, CCC will grab more data, by making additional API calls, and in your tests this likely means 2 calls per movie instead of just 1. But these calls are sequentially process, not simultaneous, so the My Movies API should just process the 2nd one similar to the first.

Actually, often I find CME to be slower than CCC, simply because it takes CME longer to write data to the NAS versus CCC storing it locally, so even though CCC pulls more data, it stores it more efficiently.


Now, here are some performance considerations for an optimal installation of My Movies and CCC:

If possible, My Movies should be installed on your NAS server (this is how I have it, with my NAS box being an Unraid server that also runs VM's, so I created a Win10 VM for My Movies, so all communications are on the same box, not over the network).

If that's not possible, then My Movies should have a fast, close connection to your NAS server. Gigabit recommended, and at most 1 switch between your My Movies server and your NAS.

CCC should only be installed on your My Movies server. Do NOT run CCC on a different PC and have it connect via the network to your My Movies server. Yes, technically this works, but it has poor performance. Best to have them both on the exact same Windows PC.

Also, do NOT run CCC on a My Movies Client PC that runs locally but still connects via the network to the My Movies Server. This too creates a performance bottleneck, as the My Movies Client plays man in the middle, taking the API requests from the local CCC and forwarding them over the network to the My Movies Server, and then receiving the response from the Server and relaying it to the local CCC application.

I mention all of this since you've recently upgraded to new, faster PC's but you still have your old PC's, and I'm not sure of your overall installation architecture anymore. It may be that you've done something against these guidelines and that could be causing the performance issues.

So an ideal MM + CCC + CMC installation would be MM Server on the NAS, CCC on the same PC with MM Server, zero MM Clients (simply not needed), and CMC on your HTPC's. Your CMC HTPC's connect to CCC on your My Movies Server, and your My Movies server should have a very efficient connection to your NAS media folders (either because it's on the same hardware, or has a very fast network connection).

[robert5733]

I know I said I was taking a hiatus, I took the day to take care of some other things and gave this a couple hours tonight. No more all nighters though.

First I will clarify some things from your post. I semi sort of think I have most of it figured out from tutorials.

In this case CCC and CMC both work fine with your NAS offline (only playback needs it online).

This loops us back to your test with the NAS offline and the CCC export being super quick. My Movies was still trying to scan your media folders for the Dune data, but because your NAS wasn't reachable this step failed super quick, and the export continued. You can see how much faster this was without the folder scans. And CCC didn't care, because CCC discards the Dune data anyway, so the end result is essentially the same, only much much faster.

So if things go as expected, I don't need other information, so my best bet is to disconnect the NAS and do the CCC export and forget the MM and API etc... and get the 33 min export. I lose nothing except a long wait. Unless MM starts some kind of faulty behavior due to NAS being offline, hopefully the quick 33 minutes offline status won't raise flags. I wish Binnerup would work with you. Losing Media Center is going to lose a large customer base, you would think they would want to work with you to get your customer base. Or maybe they are stealing your code so they can make their own version of Media Center.

CCC v7 did include a number of fixes/enhancements that result in it pulling even more data than v6, so it's possible that this made v7 even slower than v6 - again this isn't CCC being slower, simply the MM API being taxed harder and working slower. Though for the most part, I think it should perform very similarly to v6.

I am going to experiment with all versions, I will explain later.

Regarding your performance delta between CME and CCC, this is much harder for me to explain - or even comprehend. Yes, CCC should be a little slower, but not that much slower. I use a shared code base for CME and CCC, it's 90% the same program, the main difference is that CME writes to your media folders, and CCC writes to it's own local DB. Additionally, CCC will grab more data, by making additional API calls, and in your tests this likely means 2 calls per movie instead of just 1. But these calls are sequentially process, not simultaneous, so the My Movies API should just process the 2nd one similar to the first.

Actually, often I find CME to be slower than CCC, simply because it takes CME longer to write data to the NAS versus CCC storing it locally, so even though CCC pulls more data, it stores it more efficiently.

Yes this is baffling me. I don't remember this problem before(year ago). These new tests should shed some light on my setup. Only thing I can comprehend is that it gets the information and writes it then goes for more and has to wait, writes it then goes for more and so on so on. I don't think that is what it is doing and not what you explain is the process. Perhaps the cache or buffer gets full and it has to wait for more space to get more data and then has to wait for access. Maybe that is why my old PC seems to be different than my new PC, something about cache or buffer. That is why I want to test my old PC through different configurations, already put my new one through the gammet.

Now, here are some performance considerations for an optimal installation of My Movies and CCC:

If possible, My Movies should be installed on your NAS server

Not possible with my NAS

If that's not possible, then My Movies should have a fast, close connection to your NAS server. Gigabit recommended, and at most 1 switch between your My Movies server and your NAS.

CCC should only be installed on your My Movies server. Do NOT run CCC on a different PC and have it connect via the network to your My Movies server. Yes, technically this works, but it has poor performance. Best to have them both on the exact same Windows PC.

Also, do NOT run CCC on a My Movies Client PC that runs locally but still connects via the network to the My Movies Server. This too creates a performance bottleneck, as the My Movies Client plays man in the middle, taking the API requests from the local CCC and forwarding them over the network to the My Movies Server, and then receiving the response from the Server and relaying it to the local CCC application.

Pretty much have this scenario, will explain shortly.

I mention all of this since you've recently upgraded to new, faster PC's but you still have your old PC's, and I'm not sure of your overall installation architecture anymore. It may be that you've done something against these guidelines and that could be causing the performance issues.

I am replacing my old PCs with new ones. The old ones are not connected, nor even turned on, they are in a cabinet until I get this set up and then they will be sold. I am only bringing one back out for testing purposes but not having any other PCs on while doing so.

So an ideal MM + CCC + CMC installation would be MM Server on the NAS, CCC on the same PC with MM Server, zero MM Clients (simply not needed), and CMC on your HTPC's. Your CMC HTPC's connect to CCC on your My Movies Server, and your My Movies server should have a very efficient connection to your NAS media folders (either because it's on the same hardware, or has a very fast network connection).

Here is my setup. I have a NAS storage drive which is JABOD. It has it's own operating system? that allows connection over Gigabit network. Wired straight to a Gigabit switch (only used for complete setup where I will have a second computer accessing the first for CCC). At this time, only one PC connected to switch with Gigabit ethernet. This PC is running MM and CME_CCC and PowerDVD 22(without updates removing SGX). This will always be my media center computer, without internet access, except when required) and not used for any other purposes. Once this PC is set up, I will add another PC for the bedroom to access CCC and play movies through the Switch to the primary PC. That is the reason for the switch in the configuration.

Now, as I have said earlier, I gave it a couple hours tonight. My primary PC is set up using CME and CCC(two folders in the C:\, one called CMC_CME and one CMC_CCC) with the beta v6 CMC in both folders. I use this to store metadata from CME for the future, and to run CCC to watch movies. It is serving that purpose right now and works great. Only errors I receive is the screensaver. Not worried about that. Worried about the slow CCC export, want to experiment and see if I can get that resolved.

I spent a couple hours and set up my old PC with a fresh windows install, PowerDVD 17(didn't have a problem before SGX concerns), and installed MM(tried to use new test account but forgot about the 2500 points required for API), used a backup of my good account. Made a backup right here so I can restore and try over and over.

Here is my question(finally) and concern. I know that only CME rights to the NAS, CCC does not. I can run test exports, but they are not real exports. I don't want to write to the NAS over and over. I want to try v6 beta on this PC and see if there is a difference with CCC, compared to new PC. I know I can simply jump right to CCC and try exporting, but I want true analysis. That means starting with CME. Can I copy the CMC_CME folders from the working PC(new one) and put them in the folder for CCC_CME of the test PC(old one) and it will not see any changes for export and not have to write to the NAS, and CMC_CME will just open synced(or is it the registry that will cause it not to see\sync etc...). CCC I am not worried about since it does not write to the NAS.

Then I want to upgrade that v6 to v7 and test the same results.

Then I want to start fresh again, reboot to backup, and use fresh v7 folders. Same question about copying files(except for CMC file) and test results

Bottom line is CME not writing to the known working files I have on the NAS and new PC.

[Pauven]

...so my best bet is to disconnect the NAS and do the CCC export and forget the MM and API etc... and get the 33 min export. I lose nothing except a long wait. Unless MM starts some kind of faulty behavior due to NAS being offline, hopefully the quick 33 minutes offline status won't raise flags.

The problem with this strategy is that the only way for CCC to access the API is for the My Movies General Service to be running, and this is the service that scans for collection changes (like titles disappearing).

Perhaps you'll get lucky, that by having the entire server unreachable that MMGS won't start removing titles. But don't fool yourself into thinking that My Movies isn't aware the titles are "gone", it knows, it definitely knows. The question is will it trigger the title removal logic...

I wish Binnerup would work with you. Losing Media Center is going to lose a large customer base, you would think they would want to work with you to get your customer base. Or maybe they are stealing your code so they can make their own version of Media Center.

Honestly, they're welcome to my code if they so wanted to run with it. But my gut tells me they are abandoning the desktop market, migrating towards a mobile collection management app. They've been vocal about saying the mobile app is their main revenue source, explaining the reason they don't continue to develop on desktop is that the market has dried up.

Additionally, I changed CMC to be free because the licensing solution was costing me more money than licensing revenue brought in, the CMC user base is pretty tiny, so it's actually more cost effective to just give it away. In hindsight, I wish I had made it free from the beginning, it would have helped grow the user base and in turn that might have had a positive influence on Binnerup's plans with desktop MM. Of course, I never expected that Binnerup would be so unwelcoming to a 3rd party replacement for WMC. There's been a handful of positive recommendations to use CMC over the years, but for the most part they seem to pretend CMC doesn't exist.

Only thing I can comprehend is that it gets the information and writes it then goes for more and has to wait, writes it then goes for more and so on so on. I don't think that is what it is doing and not what you explain is the process. Perhaps the cache or buffer gets full and it has to wait for more space to get more data and then has to wait for access. Maybe that is why my old PC seems to be different than my new PC, something about cache or buffer. That is why I want to test my old PC through different configurations, already put my new one through the gammet.

Decades ago I was supporting a DOS application for a hospital network, and our team embarked on a host "server" PC upgrade project. I think we upgraded from 66MHz PC's to 133MHz PC's, hoping for a nice performance boost. Instead the application ground to a halt and had all kinds of issues. Turns out the PC's were too fast. We had to work with the vendor on code updates to handle the faster processing.

It's crossed my mind that something similar could be occurring here. A faster PC might hammer the server harder, and maybe it's exceeded some kind of threshold on one of the components (most likely related to the NAS, since turning it off drastically improves performance).

Years later I was doing stress testing of a large Unix application deployment for The Home Depot. As we ramped up the load, simulating hundreds and thousands of users, we kept hitting a threshold where everything fell apart, performance ground to a halt, and the software essentially crashed. It took many weeks of troubleshooting, during which we identified that some servers worked fine, while others exhibited the issue. Eventually we traced it to NICs that were misconfigured, running at 10Mb half-duplex, instead of 100Mb full-duplex. This choked network communications enough, and added latency to the overall processing, that the end result is that the servers started timing out and retrying communications only making network traffic worse.

So similarly it's crossed my mind that the issues you're seeing could be due to a network/NIC configuration issue. When you start doing things thousands and tens of thousands and even hundreds of thousands of times, repeatedly back-to-back (i.e. full export via My Movies API), it's little things like a NIC that can have a profound impact.

While I doubt that your issues are NIC or CPU speed related, I think the mindset here is to consider all possibilities, even ones that seem unlikely.

Here is my setup...

So if I read all that correctly, you have one NAS and one PC, connected via gigabit through a switch. The PC is running My Movies Server, CCC, CMC, and PowerDVD.

While I typically like to see the My Movies Server and CCC running on a separate, dedicated PC, distinct from your HTPC's, your single PC solution should be pretty optimized for performance, so I can't fault your architecture.

Worried about the slow CCC export, want to experiment and see if I can get that resolved.

In all of your various tests and updates, I'm not clear on how long this is taking now with your NAS online. If a full export is in the range of 2-8 hours, for a collection of your size, this might just be "how it is" and not indicative of an issue. Originally you were projecting 270+ days or something like that, and it seems you resolved that original issue by correcting the datetime. So what is the current full export time?

I can run test exports, but they are not real exports. I don't want to write to the NAS over and over. I want to try v6 beta on this PC and see if there is a difference with CCC, compared to new PC. I know I can simply jump right to CCC and try exporting, but I want true analysis.

That's what the CME Test Export button accomplishes - it simply turns off the writes. It still exports the data from My Movies, but doesn't write it to the media folders. It's been years since I created all this, so I'm a little foggy on one aspect - I don't remember if it still exports the artwork, but it definitely exports the XML data, and that's usually 90% of the API processing load.

Can I copy the CMC_CME folders from the working PC(new one) and put them in the folder for CCC_CME of the test PC(old one) and it will not see any changes for export and not have to write to the NAS, and CMC_CME will just open synced(or is it the registry that will cause it not to see\sync etc...). CCC I am not worried about since it does not write to the NAS.

No, not exactly. CME and CCC write some data to the Windows Registry, including the settings and the timestamp of the last Changes Only export. So if you copy it from one folder to another on the same PC, it runs the same. But if you copy it to a completely different PC, the registry values can/will be different.

This matters most if you are trying to do a Changes Only export, as the timestamp would be different for determining how far back to look for changes. So on your new PC if you ran an export yesterday, then running a Changes Only today would only looks for changes since yesterday's run. But if you turn on your old PC that's been offline for a month, doing the same Changes Only export will look for changes since the last run, i.e. a month ago.

If you are doing a Full Export, then the registry mostly doesn't matter. You would still need to double-check CME's export settings to make sure they match on both PC's - there are options for determining what data is written and where it is written, and since these settings are in the registry you need to make sure they match manually in the CME settings panels. But the timestamp is irrelevant to a full export.

If you just want to get timings, do a Test Full Export in CME on both PC's, times should be comparable. I wouldn't advise comparing a test export to a real export, simply because a real export adds in writing data to your NAS, which is simultaneously being scanned by MM, so a lot of extra activity makes a real export not equivalent to a test export, at least performance wise.

But for the most part, there's little downside to doing a real full export multiple times. There is a little extra drive wear from re-doing all the writes, but we're only talking about a few gig of data, pretty small even compared to writing a single DVD.

[robert5733]

In all of your various tests and updates, I'm not clear on how long this is taking now with your NAS online. If a full export is in the range of 2-8 hours, for a collection of your size, this might just be "how it is" and not indicative of an issue. Originally you were projecting 270+ days or something like that, and it seems you resolved that original issue by correcting the datetime. So what is the current full export time?

The export time in the log I mentioned earlier shows CME exporting at 3 seconds per title, after 4 titles there is a time query. It is instantaneous with no lag in progress. Unfortunately, I don't have the same logs for the CCC export to look at. Lost them with the restores. But there was a lag in them I remember. Will get new logs later as I mention below, my future on this.

So here is the dilemna I'm facing now. I added 5 movies in MM. I exported CME, Finished lickity split, maybe 45 seconds tops. Then synced CMC(CME) and it synced in less than a minute. Perfect all is well

Exported CCC and it took like 2 min 40 sec per title. Abysmal. Opened CMC(CCC) and the titles were already there. Perfect.

Difference this time is that the NAS is running. So it got the information from the NAS this time instead of CME which answers one of my earlier questions. Next time I will turn the NAS off and do the export and see if goes back to CME.

Wait a minute, this is too much. Although my watch timer said 2 min 40 sec on it for one title as it scrolled through the log, the finished log says 3 min 19 sec for the entire 5 title export. So that would be like 13 hours to finish the 5 titles scrolling on the screen, but the finished CCC log would say 15 minutes from start time to finish time. Even if the scrolling was different than the actual writing, 3 min per title would be like 18 hrs to export. Guess only a real export will tell the truth. I disabled logging in MM so I won't have that log to look at.

If this is the case and you fixed getting the data from CME in v7, then I think I want to stay on v6 unfortunately. I'm sure it has to do with MM but I don't hold hopes of them fixing it. If I had to do a full export, that would take me like 70 hours. I just don't know what is going on here. I know you say three hours, the only way I could verify that would be to do a full export. I like my 33 minute export from CME.

I started a test export and let it run for 35 minutes or so.

The scrolling export shows 12 titles in a half hour so far, so not sure how reliable any of this is. The computer would have to be on till the screen shows export done which might be 70 hours, but the actual export might have been done sooner according to the time stamp on the CCC export log. And this is all theoretical since it is only a test export. I aborted at 15 titles, and have to wait for the program to respond, it took a couple of minutes.

Disconnected the ethernet to NAS and test export. Scroll is super fast, can't even read the text going by. It Is getting it from CME.

This is what I'm trying to figure out.

[robert5733]

You can skip all My tests below if you like, here is the end result in v7

Wait it had a slow start on full export(not test), only 2 movies in a long period, but then just started exporting rather quickly with NAS in standby and no link lights on NAS or switch LEDs. Could be it started once NAS went to standby. Maybe it never got it from CME in the first place when the NAS was offline. It must be just getting info from MM and not the NAS. There is nowhere else to get the files from on my fresh restore.

I screwed up and rotated the NAS to check link LEDs on the back and hit the menu button waking the NAS, CCC is now taking 2 min 45 sec again per title. It Was blazing at over 1000 titles in 30 min. Currently no HDD LED activity indicated only exporting at about 3 minutes per title. Will see what happens when it goes back to standby. Waited 45 minutes for return to standby, before pulling the ethernet. Back to fast exporting.

So the timestamp has to do with time syncing between the NAS and MM according to the log. Forgot to set the time after the restore is the first problem. It caused the slow export because it kept changing the time jumping from 4pm to 7pm to 4pm to 3pm and many more, with the frequent timestamps. Then the NAS went to slepp. Only did the time stamp at each login and export was blazing. Then I woke the NAS and timestamps started frequently (no time changes this time) just like in the past, and export crawled. I pulled the ethernet and timestamps went back to logins only and it was blazing again.

Now I have a place to begin finally. It must be the NAS doing the time requests, since they only happen when it is on/connected. Two settings I think could play a role is NTP time update and Twonky rescan for changes in shared folders.

By the way total export time on this slower machine was 2 hrs, and that's counting the 45 minutes of nothing.




Here are the results of my first test run with the old computer.

v6

Old PC, fresh windows install, defender firewall disabled, fresh MM restore, no CMC_CME folder, only CMC_CCC v6, NAS connected. Ran CCC test export

Same result as new PC, about 3 minutes or so for each title to scroll on log, blue circle spinning and CCC not responding. Attempting abort, at waited for 4 titles to scroll on log(10-15 minutes) with no abort, just blue circle and not responding. clicked abort again, waited for the log to scroll 3 more titles (10-15 minutes) before force closing CCC.

Added CMC_CME folder and copied all the artwork etc... from the old export(to give CCC somewhere to get it) to the CMC_CME folder, disconnected NAS. Ran CCC test export.

scrolling about 3 titles per second, obviously getting it from CMC_CME. CCC fully responsive to abort command. Log says 183 titles processed in 2 min 16 sec.

v7

Tried just deleting CMC_CCC v6 and CMC_CME v6 and placing CMC_CCC v7 in the C:. Reconnected NAS.

Same exact result as v6 above.

Added CMC_CME folder and copied all the artwork etc... from the old export(to give CCC somewhere to get it) to the CMC_CME folder, disconnected NAS. Ran CCC test export.

Same exact result as v6 above.

Restored computer to fresh install.

no CMC_CME folder, only CMC_CCC v7, NAS connected. Ran CCC test export.

Same exact result as v6 above. Noticed lights for HDD on NAS are not blinking and the NAS is in standby mode. So not accessing it for data.

MM log says only one title exported during this time period taking 2 min 38 sec. Had to force close CCC.

Added CMC_CME folder and copied all the artwork etc... from the old export(to give CCC somewhere to get it) to the CMC_CME folder, disconnected NAS. Ran CCC test export.

Same good result as v6 above. MM log says 42 titles exported in 54 Seconds.

Curious thing in v7. Got errors for export on 3 titles for missing metadata in MM. They were Harlem Nights, Boomerang, and Coming To America. I don't have these movies, and they are not in my MM database. Didn't happen in v6. MM log has no reference to Boomerang in it.

So the program appears to work fine with NAS disconnected, that leads me to think there might be a problem with the NAS or MM during CCC export. Curiously though, CME in v7 still exports fast when connected to NAS and does not have the 3 titles missing error, so speed of NAS appears fine leaving it to MM as the culprit.

[robert5733]

Turned off NTP time updates. Turned off Cloud access, turned off Mac Backups, Turned off AFP, Turned off iTunes.

Checked Logs and they are still timestamping. While taking a break, the NAS went to standby, link LEDs are still flashing for NAS and MM is still logging timestamps.

Tried CCC export while NAS is in standby, Slow export as usual. Force closed it.

Checked logs and still major timestamps. Disconnected ethernet and checked logs. Still timestamping every 12 seconds.

Maybe it changes with a full export instead of a test export. Reconnected ethernet and tried a full export. Same slow export.

Disconnected Ethernet, within a minute fast export began.

So it is not the NAS that is initiating the timestamp. Don't know why without the NAS it blazes through the export. The timestamp is coming from MM. As was seen in the high debugging log. It is initiating processes with timers. Not timestamping(just the name I gave it at the beginning of this).

I guess this completes what we can do with troubleshooting, it is something going on in MM, and we can't do anything about it.

I will keep constant backups of MMCM in case something goes awry, but my only choice at this point is to disconnect the NAS for CCC export. Still don't see why it is different in CME and CCC, one works and the other doesn't. I know there is more info in CCC to pull, but it is still pulling the info with the NAS disconnected using MMCM data.

So I am done unless something else goes bad.

[Pauven]

Wait it had a slow start on full export(not test), only 2 movies in a long period, but then just started exporting rather quickly with NAS in standby and no link lights on NAS or switch LEDs. Could be it started once NAS went to standby. Maybe it never got it from CME in the first place when the NAS was offline. It must be just getting info from MM and not the NAS. There is nowhere else to get the files from on my fresh restore.

Unfortunately I can't tell from your description what program you're talking about. I think you're talking about CCC v7, but that's really just a guess.

If so, please understand that CCC never pulls data from CME, and CME never pulls data from CCC. And neither CCC nor CME ever pull data from the NAS - with one exception on CME.

You can configure CME to "Force Update" the artwork, which means it will be overwritten even if the exported file already exists (from a previous export). If the Force Update options are not enabled, then artwork will only be exported for files that do not already exist in your NAS folders. There's a related option for Force Updating the XML files too. So in this regard, CME does look in the folder before it writes any files, just to see if a file of the same name exists, but it doesn't "use" or even evaluate the file, it simply notes if it exists or not. So I think you may be talking about this, but I'm not sure.

For CCC, I took away those options. CCC will ALWAYS do a force export of all files (XML data and artwork) anytime a title is exported. This makes more sense because it ensures more accurate, up-to-date data, and it's also being written to the local CCC folder, not to your NAS, so there's no risk of overwriting your NAS data.

All exported data comes from the My Movies database, even the artwork. Those jpg's are stored in what's called a blob format inside a DB table. While the My Movies API is pretty slow to retrieve the XML metadata, it's surprisingly fast to pull the artwork, almost real-time.

So long story summarized: all CME and CCC exported data comes from the MM API, which in turn comes from the MM DB. Additionally, CME has options to skip exporting if files already exist, but any exported data always comes from the MM API & DB.

I screwed up and rotated the NAS to check link LEDs on the back and hit the menu button waking the NAS, CCC is now taking 2 min 45 sec again per title. It Was blazing at over 1000 titles in 30 min. Currently no HDD LED activity indicated only exporting at about 3 minutes per title. Will see what happens when it goes back to standby. Waited 45 minutes for return to standby, before pulling the ethernet. Back to fast exporting.

I find this very interesting. It sounds to me like MM is not able to wake up your NAS, and since CCC doesn't wake it up either (since it never accesses it), then this could explain why your CCC exports were previously fast - as long as your NAS was sleeping CCC exports were fast. But in your recent testing, you've had your NAS awake, which has exposed a performance issue when MM accesses it to retrieve the DUNE data.

This would also make me think that CME would always be slow, since CME accesses your NAS to perform reads & writes, and presumably this would wake it up. But at the same time, since CME exports less data from MM, perhaps it's not as bad because MM is doing less work and accessing the NAS less often.

So the timestamp has to do with time syncing between the NAS and MM according to the log.

Sorry, I think you've mis-interpreted what the timestamps are. For reference, I believe this is what you're referring to as the Timestamps:

Code: Select all

2024-01-04 16:28:20 . 30 - Common.Database.ExecuteNonQuery - 'UPDATE tblCollectionChanges SET nvcProcessClient = 'DESKTOP-TOMF3C2' WHERE ISNULL(nvcProcessClient, '') = '' AND DATEDIFF(SECOND, datTime, GETDATE()) >= 0'

So what this does is update a My Movies table named "CollectionChanges", and it sets the field "ProcessClient" to your PC's name (DESKTOP-TOMF3C2), for all records where the ProcessClient is empty and the record's timestamp is older than the current time.

This table is basically a checklist. Whenever a title experiences a change (either you changed some data, or maybe an online contribution was downloaded), an entry is added to this table with the timestamp of when the change was applied, and the ProcessClient value left empty. This new record is an unprocessed record.

Then the My Movies General Service uses this query to find these new records based upon the ProcessClient being empty and the date being old, and then it processes these records (i.e. exports the data to the NAS, something My Movies does automatically to create the mymovies.xml files and some artwork), and then flags it as processed with this query (by changing ProcessClient to this PC's name). This is now a processed record, and won't be picked up for processing again. Just a simple checklist.

I mention all this because you seem to be focusing in on NTP time updates as a possible issue, and I think you're going in the wrong direction. Yes, the DB time needs to match the MM client's time for accurate comparisons, otherwise GetDate() (which is the DB's current date/time) will be offset from the record's datTime value which was created by a different MM process.

It honestly shouldn't cause a performance issue if these are not synced, rather it would just cause a delay in processing (i.e. it won't even show up for processing for several more hours). I think your earlier result where you saw a speedup after correcting the time was a coincidence, and led you to believe you were onto a solution.

Think about it like a Dr.'s appointment - you're supposed to show up at 2pm, and you show up at 2pm Pacific, but you're 3 hours late because the appointment was at 2pm Eastern. The Dr. sees you anyway, and your visit lasts 15 minutes, the same as if you had showed up on-time. So the time sync issue doesn't cause a performance issue, it simply time shifts when the work gets performed.

Also, this DB query has NOTHING at all to do with your NAS. This is not syncing data to your NAS, this is not even accessing your NAS. It is simply MM General Service querying and updating data in the MM DB. It's the result of these queries that may lead to work being performed (MMGS exporting data to your NAS).

So the timestamp has to do with time syncing between the NAS and MM according to the log.

No, it has nothing to do with your NAS. It is only MM updating records in it's own DB, housekeeping.

Now I have a place to begin finally. It must be the NAS doing the time requests, since they only happen when it is on/connected. Two settings I think could play a role is NTP time update and Twonky rescan for changes in shared folders.

Again, no, it has nothing to do with your NAS. Think about it, the query is a MM query updating a MM table, and this gets logged in the MM debug log.

Your NAS doesn't even know what MM is, it simply has no concept of My Movies. So how would a NAS try to update the MM database, and the entry shows up in the MM log. Your NAS doesn't even know MM exists, much less how to communicate with it via an API or any other means. It's simply not possible.

And further, MM doesn't care one bit about the time on your NAS, it's simply irrelevant. MM cares about it's own time (on the PC) and the MM database time, and possibly timestamps on files that live on the NAS. But those timestamps on those files come from MM, which created them, not the NAS time.

The reason you're associating the timestamps with the NAS being available is because the My Movies General Service will wait for the NAS to be available for scanning/writing before doing any work. You take your NAS offline, and MMGS halts, which in turn stops these timestamps because MMGS isn't doing any work. You bring your NAS back online, and MMGS resumes doing work, scanning and writing to your media folders, and logs this work as being done in it's own DB via these timestamps.

Turned off NTP time updates. Turned off Cloud access, turned off Mac Backups, Turned off AFP, Turned off iTunes.

Checked Logs and they are still timestamping. While taking a break, the NAS went to standby, link LEDs are still flashing for NAS and MM is still logging timestamps.

Tried CCC export while NAS is in standby, Slow export as usual. Force closed it.

Checked logs and still major timestamps. Disconnected ethernet and checked logs. Still timestamping every 12 seconds.

This is exactly what I'm saying - the timestamps have nothing at all to do with your NAS. It is housekeeping, a checklist that My Movies General Service uses to see if any work needs to be performed. These timestamp entries don't even reveal if work was done, merely that MMGS was checking off the work that was performed, so it doesn't get done again. They are a good thing, MM working as intended, do not indicate a performance problem, have nothing to do with NTP, the cloud, Mac/AFP/iTunes, or anything else.

The fact that they are timestamping is a GOOD THING. It shows MMGS marking work as complete. The 12 second interval is how often it scans for and completes work. When they are 12 seconds apart consistently, that probably indicates there was no work done. When the interval is longer, i.e. 5 minutes between, then that probably indicates there was nearly 5 minutes of work being performed.

There's only 2 ways to stop the timestamps, turn off MMGS (which is performing the timestamps), or turn off the MM DB/SQL Server (which is where the timestamps are being updated). In both cases, you break My Movies and the My Movies API, because you turned off critical services. And the lack of timestamps is a BAD THING - they're like a heartbeat indicating a healthy system, and you're trying to kill the heartbeat because it beats every time a muscle is moved.

Added CMC_CME folder and copied all the artwork etc... from the old export(to give CCC somewhere to get it) to the CMC_CME folder, disconnected NAS. Ran CCC test export.

scrolling about 3 titles per second, obviously getting it from CMC_CME. CCC fully responsive to abort command. Log says 183 titles processed in 2 min 16 sec.

No, CCC doesn't care if artwork is there or not, it will always export, always overwrite, and never checks to see if it already exists. This is by my design.

CCC gets all artwork from MM.

All you proved here is that by disconnecting NAS, you are turning off the DUNE data retrieval scan, and so MM works fast (all data comes from MM DB). When NAS is online, MM performs the extra media folder scan to gather data needed for DUNE devices to connect to your NAS, and THIS results in the performance slowdown.

Curious thing in v7. Got errors for export on 3 titles for missing metadata in MM. They were Harlem Nights, Boomerang, and Coming To America. I don't have these movies, and they are not in my MM database. Didn't happen in v6. MM log has no reference to Boomerang in it.

These are probably referenced in a box set or movie collection, and CCC is trying to export the full collection. This could indicate bad data in My Movies, possibly even some corrupted data. CCC can't hallucinate these movie names, it only knows about them because MM referenced them, and CCC is simply trying to export them.

So the program appears to work fine with NAS disconnected, that leads me to think there might be a problem with the NAS or MM during CCC export. Curiously though, CME in v7 still exports fast when connected to NAS and does not have the 3 titles missing error, so speed of NAS appears fine leaving it to MM as the culprit.

I agree with half of your conclusion.

CME doesn't export Movie Collections, this is only in CCC, so if the 3 titles are part of a Movie Collection, that would explain why the errors only show up in CCC.

Since the slowdown only occurs when your NAS is online, and only affects CCC not CME, then this 100% indicates it is My Movies slowing down during accessing your NAS. CCC NEVER accesses your NAS.

I think you're probably right that your NAS is fine, since CME runs quickly and your NAS is handling simultaneous accesses from CME and MM at the same time.

So the question becomes, why does MM accessing your NAS during CCC exports cause a performance slowdown, yet it doesn't happen during CME?

I think there are two potential considerations here.

a) Perhaps CME accessing your NAS is helping. I know that seems counterintuitive, but maybe CME is somehow opening up a communications channel, and this helps MM when it also accesses your NAS. Honestly, this seems unlikely, but I can't rule it out. At the very least, since CME is accessing the same media folders at the same time, it's encouraging the NAS to be responsive to data requests in those directories, so maybe that is helping.

b) CCC exports more data, and does so by requesting more data, via more API calls. This puts a bigger burden on MM to respond to more queries. Additionally, MM stupidly repeats the NAS media folder scan on one of these extra calls. So where CME causes MM to scan your media folder once, CCC causes MM to scan it twice.

"b" seems more likely to me as related to the core issue, but it still seems like, at worse case, the export would be half as fast as CME. So if CME took 2 hours, CCC would take 4, if the MM folder scanning for DUNE performance penalty was extrapolated linearly.

But what you're seeing is the entire export grind to a halt. This probably indicates some type of locking/blocking is occurring. Maybe the NAS reached it's access limit, and is causing the slowdown. Or maybe MM can't handle all this processing.

Now, I can say that what you're experiencing is not normal. You and perhaps one other user are the only ones that have encountered these types of issues.

As far as how you resolve this, well it does seem to be a My Movies issue. CCC not responding to you clicking buttons is a symptom, not a cause, it's simply waiting on MM to respond. And the only reason your NAS is accessed during a CCC export is when MM retrieves the data for DUNE.

So it is not the NAS that is initiating the timestamp. Don't know why without the NAS it blazes through the export. The timestamp is coming from MM. As was seen in the high debugging log. It is initiating processes with timers. Not timestamping(just the name I gave it at the beginning of this).

Correct, seems you were already arriving at the proper conclusion. And to say it yet again, it's MM doing a folder scan to retrieve data for DUNE devices. Why that data isn't already stored in the DB is beyond me, seems very inefficient, but Binnerup suggested it was to ensure the data was real-time accessibility data so that DUNE's didn't error out when trying to play media.

I guess when you use DUNE to browse your collection, it requests the title data in real-time via the API - for only that 1 title. The DUNE related data is added to tell DUNE how to log into your NAS and play the associated media. Because this "could" change (i.e. your NAS could be offline, paths and passwords could change), the MM API generates it in real-time by scanning the media folder right when the title data is requested. That way DUNE knows if your NAS if online/offline, and the current credentials and paths for playing the media.

When I think about it from the DUNE perspective, this behavior makes sense. But it's simply not needed for CCC. I even delete the DUNE data from the XML, because it includes passwords in plain text and represents a security risk, and CMC doesn't need any of this DUNE data anyway. So this behavior is extremely undesirable for CCC/CME/CMC users.

I guess this completes what we can do with troubleshooting, it is something going on in MM, and we can't do anything about it.

What you can do is complain to Binnerup that the media scans for DUNE data on API export are killing your ability to use your My Movies data. Complain in that same My Movies forum thread I posted earlier. The solution is that Binnerup needs to provide a method for turning off the DUNE data scan when the title and movie data are queried via the API. Let your voice be heard. The more people that complain, the more likely we are to get an actual solution.

While I don't suffer the performance penalty, I HATE that My Movies spins up my sleeping hard drives on my NAS when CCC simply wants data that's already in the DB. It wastes power, and it causes wear and tear on my precious data drives.

And while I agree that disconnecting your NAS to improve CCC export speeds won't harm CCC/CMC data (it's 100% the same), I worry that you risk triggering the My Movies deleted title processing logic, so what you're doing is not without risk. Please, complain to Binnerup. We all need a solution, you more so than anyone else.

[Pauven]

Something I keep forgetting to mention is that mapped drives are not supported by My Movies.

While your NAS is on the network, you are accessing it via mapped drive letters. I know it seems logical to do this, and yes it seems to work okay with My Movies, and definitely is mostly fine for CCC/CME too. But Binnerup says you should NOT do this.

The most obvious reason not to do it is that you could have mappings not match from one PC to another. For example, if your My Movies has the NAS as drive V:\, but your HTPC has it as M:\, then that mismatch will cause playback errors and possibly write errors. So to even have any chance of success with mapped folders, every PC must use the exact same mapped folders.

But even then, there's another reason not to do it - My Movies doesn't support it! Instead, you're supposed to use UNC network paths.

For example, I have my Unraid NAS named "Tower". Instead of mapping drive M: to \\Tower\Movies, I would instead simply access \\Tower\Movies (the UNC path) directly in My Movies.

CCC, CME, and CMC should also use the same UNC paths, to be consistent with whatever you used in My Movies.

I know you're thinking "but mapped drives work fine for me!". Well, maybe they don't. I have had other users report weird behavior when using mapped drives instead of UNC paths, and if I recall correctly, I think one or two users also were experiencing some performance issues. Not quite what you're experiencing, but not completely unrelated either.

Additionally, since Binnerup doesn't support the used of mapped drives, they're likely to resist providing you support until you follow their guidelines.

So you really should switch to using UNC paths instead of mapped drives. Luckily, this is very easy!!! Change one title from V:\T\xyz... to \\NASname\T\xyz, and My Movies will ask you if you want to change all movies to new path, simply say yes and let it update.

This easy change brings you into compliance with Binnerup's guidelines, and might even resolve the performance issue.

[robert5733]

Ok, get off that subject for now. Maybe it is time to upgrade my NAS with the new PC. If I went with a USB external drive, connected to the primary PC, will that still work with my idea of two computers accessing CCC on the primary? Secondary PC would now have to access the movie through the primary PC to the USB. Will that tax the primary PC (streaming to secondary) to the point you can't watch movies on both PCs at the same time due to the primary working to hard? With the ethernet, both computers only access the NAS for file retrieval and playing, so the secondary PC is not really taxing the primary PC.

Reason for asking is that I bought the NAS in 2015. The first two hard drives were also bought then. They have probably already outlived their expectancy, and I will have to upgrade at some point. Either new hard drives (sata) in the NAS or a complete overhaul. SSD is just too expensive for 10Tb, so I was thinking Seagate STKP16000400 USB, but that is still HDD, or a new NAS, Synology DS220j(HDD). The old NAS takes forever to boot up and is a pain to access or maintain. I have an external USB I keep a backup of the NAS on already, so failure is not a concern, well other than cost to replace...lol...

What is your thoughts on it? Is a dedicated NAS better than an external USB? will USB work without undue stress on the primary, or even work at all? I could do two USB external 5Tb HDDs(enough for now), I have the ports for it, would just have to do a lot of remapping of drives(not like I haven't been doing that already), but it would be easier to transport and increase write performance, I don't think read speed is a factor. I am in a stable environment for six months and then travel in an RV for six months. Then I would just have to add another drive when I reach capacity. That would require a hub to get the three ports, but that shouldn't make a difference.

Sorry for some basic questions, but as mentioned some time ago, my tbi has taken away alot of what I learned and knew about basic networking, as well as a lot of other things.

[robert5733]

While your NAS is on the network, you are accessing it via mapped drive letters. I know it seems logical to do this, and yes it seems to work okay with My Movies, and definitely is mostly fine for CCC/CME too. But Binnerup says you should NOT do this.

The most obvious reason not to do it is that you could have mappings not match from one PC to another. For example, if your My Movies has the NAS as drive V:\, but your HTPC has it as M:\, then that mismatch will cause playback errors and possibly write errors. So to even have any chance of success with mapped folders, every PC must use the exact same mapped folders.

I had already changed to UNC quite some time back due to incompatibility with something, somewhere. Did you see something different in my logs or something? Maybe I missed something somewhere. The Y:? reference I made was just something I found and discussed about the MM logs timestamp thing.