Re: v7 upgrade problem with Sync
Posted: Sun Jan 07, 2024 2:47 pm
I've read through your write-up, and I have the impression that you are a bit confused on how data moves around. So let's try to level set.
For CME:
NAS <-> MM <-> MM API <-> CME --> NAS <-> CMC
CME pings the My Movies API for data, most of which comes from the MM DB, but MM will scan your NAS folders as well to retrieve the Dune info. CME then writes this data (XML and artwork) to your NAS media folders. Later, when CMC does a folder scan/sync, it reads this data from your NAS media folders, pulling in the data that was written by CME.
In this case, CME and CMC both only work if your NAS is online. If your NAS is offline, then at best CMC will use previously scanned results, but I believe this only works if the "Keep Offline Data" is checked in CMC's settings (this was created for users who put their NAS to sleep), so doing a CMC sync with your NAS offline would result in either errors or your collection disappearing.
For CCC:
NAS <-> MM <-> MM API <-> CCC <-> CMC
CCC pings the My Movies API for data, again most of which comes from the DB, but MM will scan your NAS folders as well to retrieve the Dune info (usually twice, though, because CCC pulls additional data via the API). CCC then writes this data to a local CCC database. CCC never accesses your NAS directly. Later, when CMC syncs with CCC, CCC simply shares it's local DB data and artwork with CMC (either just the changes or the full DB as-needed, both super fast).
In this case CCC and CMC both work fine with your NAS offline (only playback needs it online).
This loops us back to your test with the NAS offline and the CCC export being super quick. My Movies was still trying to scan your media folders for the Dune data, but because your NAS wasn't reachable this step failed super quick, and the export continued. You can see how much faster this was without the folder scans. And CCC didn't care, because CCC discards the Dune data anyway, so the end result is essentially the same, only much much faster.
Now to be clear, I'm not recommending you have your NAS offline, since My Movies sometimes flags your collection as missing if it can't get to it, so this is dangerous territory. But it does perfectly illustrate that the API performance penalty is the folder scanning for Dune info, which I've asked Binnerup 3 years ago to provide an option for disabling. This issue affects some setups worse than others, and for some reason it seems especially bad on your setup.
CCC v7 did include a number of fixes/enhancements that result in it pulling even more data than v6, so it's possible that this made v7 even slower than v6 - again this isn't CCC being slower, simply the MM API being taxed harder and working slower. Though for the most part, I think it should perform very similarly to v6.
Regarding your performance delta between CME and CCC, this is much harder for me to explain - or even comprehend. Yes, CCC should be a little slower, but not that much slower. I use a shared code base for CME and CCC, it's 90% the same program, the main difference is that CME writes to your media folders, and CCC writes to it's own local DB. Additionally, CCC will grab more data, by making additional API calls, and in your tests this likely means 2 calls per movie instead of just 1. But these calls are sequentially process, not simultaneous, so the My Movies API should just process the 2nd one similar to the first.
Actually, often I find CME to be slower than CCC, simply because it takes CME longer to write data to the NAS versus CCC storing it locally, so even though CCC pulls more data, it stores it more efficiently.
Now, here are some performance considerations for an optimal installation of My Movies and CCC:
If possible, My Movies should be installed on your NAS server (this is how I have it, with my NAS box being an Unraid server that also runs VM's, so I created a Win10 VM for My Movies, so all communications are on the same box, not over the network).
If that's not possible, then My Movies should have a fast, close connection to your NAS server. Gigabit recommended, and at most 1 switch between your My Movies server and your NAS.
CCC should only be installed on your My Movies server. Do NOT run CCC on a different PC and have it connect via the network to your My Movies server. Yes, technically this works, but it has poor performance. Best to have them both on the exact same Windows PC.
Also, do NOT run CCC on a My Movies Client PC that runs locally but still connects via the network to the My Movies Server. This too creates a performance bottleneck, as the My Movies Client plays man in the middle, taking the API requests from the local CCC and forwarding them over the network to the My Movies Server, and then receiving the response from the Server and relaying it to the local CCC application.
I mention all of this since you've recently upgraded to new, faster PC's but you still have your old PC's, and I'm not sure of your overall installation architecture anymore. It may be that you've done something against these guidelines and that could be causing the performance issues.
So an ideal MM + CCC + CMC installation would be MM Server on the NAS, CCC on the same PC with MM Server, zero MM Clients (simply not needed), and CMC on your HTPC's. Your CMC HTPC's connect to CCC on your My Movies Server, and your My Movies server should have a very efficient connection to your NAS media folders (either because it's on the same hardware, or has a very fast network connection).
For CME:
NAS <-> MM <-> MM API <-> CME --> NAS <-> CMC
CME pings the My Movies API for data, most of which comes from the MM DB, but MM will scan your NAS folders as well to retrieve the Dune info. CME then writes this data (XML and artwork) to your NAS media folders. Later, when CMC does a folder scan/sync, it reads this data from your NAS media folders, pulling in the data that was written by CME.
In this case, CME and CMC both only work if your NAS is online. If your NAS is offline, then at best CMC will use previously scanned results, but I believe this only works if the "Keep Offline Data" is checked in CMC's settings (this was created for users who put their NAS to sleep), so doing a CMC sync with your NAS offline would result in either errors or your collection disappearing.
For CCC:
NAS <-> MM <-> MM API <-> CCC <-> CMC
CCC pings the My Movies API for data, again most of which comes from the DB, but MM will scan your NAS folders as well to retrieve the Dune info (usually twice, though, because CCC pulls additional data via the API). CCC then writes this data to a local CCC database. CCC never accesses your NAS directly. Later, when CMC syncs with CCC, CCC simply shares it's local DB data and artwork with CMC (either just the changes or the full DB as-needed, both super fast).
In this case CCC and CMC both work fine with your NAS offline (only playback needs it online).
This loops us back to your test with the NAS offline and the CCC export being super quick. My Movies was still trying to scan your media folders for the Dune data, but because your NAS wasn't reachable this step failed super quick, and the export continued. You can see how much faster this was without the folder scans. And CCC didn't care, because CCC discards the Dune data anyway, so the end result is essentially the same, only much much faster.
Now to be clear, I'm not recommending you have your NAS offline, since My Movies sometimes flags your collection as missing if it can't get to it, so this is dangerous territory. But it does perfectly illustrate that the API performance penalty is the folder scanning for Dune info, which I've asked Binnerup 3 years ago to provide an option for disabling. This issue affects some setups worse than others, and for some reason it seems especially bad on your setup.
CCC v7 did include a number of fixes/enhancements that result in it pulling even more data than v6, so it's possible that this made v7 even slower than v6 - again this isn't CCC being slower, simply the MM API being taxed harder and working slower. Though for the most part, I think it should perform very similarly to v6.
Regarding your performance delta between CME and CCC, this is much harder for me to explain - or even comprehend. Yes, CCC should be a little slower, but not that much slower. I use a shared code base for CME and CCC, it's 90% the same program, the main difference is that CME writes to your media folders, and CCC writes to it's own local DB. Additionally, CCC will grab more data, by making additional API calls, and in your tests this likely means 2 calls per movie instead of just 1. But these calls are sequentially process, not simultaneous, so the My Movies API should just process the 2nd one similar to the first.
Actually, often I find CME to be slower than CCC, simply because it takes CME longer to write data to the NAS versus CCC storing it locally, so even though CCC pulls more data, it stores it more efficiently.
Now, here are some performance considerations for an optimal installation of My Movies and CCC:
If possible, My Movies should be installed on your NAS server (this is how I have it, with my NAS box being an Unraid server that also runs VM's, so I created a Win10 VM for My Movies, so all communications are on the same box, not over the network).
If that's not possible, then My Movies should have a fast, close connection to your NAS server. Gigabit recommended, and at most 1 switch between your My Movies server and your NAS.
CCC should only be installed on your My Movies server. Do NOT run CCC on a different PC and have it connect via the network to your My Movies server. Yes, technically this works, but it has poor performance. Best to have them both on the exact same Windows PC.
Also, do NOT run CCC on a My Movies Client PC that runs locally but still connects via the network to the My Movies Server. This too creates a performance bottleneck, as the My Movies Client plays man in the middle, taking the API requests from the local CCC and forwarding them over the network to the My Movies Server, and then receiving the response from the Server and relaying it to the local CCC application.
I mention all of this since you've recently upgraded to new, faster PC's but you still have your old PC's, and I'm not sure of your overall installation architecture anymore. It may be that you've done something against these guidelines and that could be causing the performance issues.
So an ideal MM + CCC + CMC installation would be MM Server on the NAS, CCC on the same PC with MM Server, zero MM Clients (simply not needed), and CMC on your HTPC's. Your CMC HTPC's connect to CCC on your My Movies Server, and your My Movies server should have a very efficient connection to your NAS media folders (either because it's on the same hardware, or has a very fast network connection).