Decorum for the Forum:
  • Be nice. If you want to be mean, try Reddit.
  • No Piracy. If you want to be a thief, there are dark places on the internet dedicated to that.
  • No Cracking. Discussions on AnyDVD, DeUHD, DVDFab, UHDKeys and similar tools are not permitted here.
  • No Spamming. If you want to make a buck, work smarter... somewhere else.
  • No Adult Content. Half the internet is dedicated to adult content. This half isn't.

Privacy Policy: Click Here to Review (updated September 30, 2020)

Awesome Release

General support forum for Chameleon MediaCenter (formerly MM Browser)
Post Reply
Teddyboy
Posts: 268
Joined: Wed Apr 25, 2018 10:30 pm
Location: San Diego, Ca

Awesome Release

Post by Teddyboy » Sat Nov 02, 2019 9:45 pm

Great job Paul. All the chapter based episodes came in without a problem. After the initial conversion to the new version syncing appears to be much faster. One small niggle was that CME found a error in one of my looney tunes discs. No episode mappings, but that can be easily fixed.
Thanks for all your hard work.
Mechanical Engineer, Solidworks junkie, Reef aquariums, sci-fi, weight training, orchids, Ferns, Kaiju, model kit building, balboa park, scripps aquarium

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Sun Nov 03, 2019 10:44 am

Hmmmm, I'm trying to imagine why syncing would be faster. v3.0 is grabbing a few additional fields from the XML, which should actually make it slower, not faster (but we're talking milliseconds here, so not noticeable). Perhaps I made another tweak or fix I just don't recall at the moment.

Regardless, glad to hear it's working so well for you. Thanks.

On a related note, I recently discovered that my development language's default XML parser is sloooooow, especially with larger files (like TV Series). I've found a 3rd party XML library that is supposed to be the fastest available, and I will be testing it out. If it works well, I will purchase a license for it (I like to use the little bit of money that comes in from license sales to improve CMC). In theory, this would improve the speed of syncing in CMC, which will also affect CME as I begin to migrate that process over.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Teddyboy
Posts: 268
Joined: Wed Apr 25, 2018 10:30 pm
Location: San Diego, Ca

Re: Awesome Release

Post by Teddyboy » Fri Nov 08, 2019 11:59 am

When I went to use my htpc, CMC said there was a new version available. I immediately upgraded to the new version. Before I began using CMC it said I had to upgrade the metadata to the new version. That took some time to do. Once it did that I was able to use it. In order for the chapter episodes to show up in CMC I had to run CME after the initial metadata upgrade. Doing a full sync after regenerating metadata for tv series sync appeared noticeably faster then in the prior release. I can’t back it up with timing a full sync with the prior release against the current release. Too me it just appeared like it was faster.

Regardless I think this is the best release yet. I am looking forward when I will no longer have to run a sync when you move to a database. I’ve never been a fan of metadata files, but it works. Aside from my holy grail feature request of a integrated player. I would like to see parental controls fully implemented. I don’t have children, but I would like to restrict what people can see.

Thanks Paul for your hard work.
Mechanical Engineer, Solidworks junkie, Reef aquariums, sci-fi, weight training, orchids, Ferns, Kaiju, model kit building, balboa park, scripps aquarium

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Fri Nov 08, 2019 12:37 pm

In prep for adding XML parsing and DB export functionality to CME, this week I've been cleaning up and modularizing the XML parsing routines in CMC. These routines have remained mostly unchanged since the earliest versions of CMC, except for tacking on new functionality, and they certainly were an ugly unmanageable eyesore. This is work that I actually thought wouldn't happen until early next year, and I'm happy to say I'm almost done with this cleanup.

I've still got a little more cleanup and optimization to do, plus add in a few more fields from the XML files (I've already added a few). Once I get this done, my plan is to test the 3rd party XML parser and see if it is worth the price of admission.

Once all that is done, the next step is to add these new XML parsing routines into CME. There's still a lot of work to do there, but I certainly feel I'm a few months ahead of [my own] schedule. I've also already made some nice enhancements to CMC v3.1, exciting stuff I'll share later.

One thing that is causing me brain fits is how to handle tracking watched status with CME creating the DB. I'll explain the challenge below in case anyone has ideas or thoughts.

Today, CMC creates CMC_Watched.txt flag files in the media folders to track which Movies and Episodes you have watched, and these files get imported during a sync (to keep all clients in sync with each other). CME will also scan for these files, and update My Movies via the API to the new Watched status, which in turns creates new metadata files, and CMC automatically removes the CMC_Watched.txt files when the new metadata with the same Watched status comes through.

The simplicity of this approach, all metadata files and Watched flag files get output to the same media folders, doesn't really fit into the new model with CME creating the DB file directly.

In the CME DB Master model, no metadata gets written to your media folders by CME (unless you want it to). Instead, all of the API data is parsed immediately and a DB file is created. A CMC client will ping the CME Master to see if a new DB is available, and download the latest if changed. If it hasn't changed, or CME is offline/not reachable, then CMC continues by loading the current local DB.

So when you watch a movie and flag it as Watched, how do we get that status back to CME and into My Movies? If CMC continues the current process of writing CMC_Watched.txt flag files to the media folders, that feels a bit at odds with the overall new approach. If CMC only tracks the Watched status in each PC's local DB, then that gets blown away and lost with each DB refresh. If CMC immediately contacts the CME Master with the Watched status change, that requires CME to always be running, something I'm trying to avoid.

One idea that seems a bit better than the others is for each CMC to track Watched status changes locally, and anytime it communicates with CME for an updated DB it could also share the Watched status changes. But I still see a gap with this solution. Say you have 3 CMC PC's, and a guest watches a movie on PC #3 (in a rarely used guest room), then shut down that PC before it syncs again with the CME master. That means that CME and the other 2 PC's are unaware of the Watched status change. A month later you power up PC #3 and it finally syncs with the CME Master and sends over the new Watched status. Since a month has gone by, that's really old data to be syncing in, and perhaps it is no longer what you want as the current status.

The most robust solution still seems to be for CMC to create the CMC_Watched.txt flag files in the media folders, which allows for a sort of offline sync capability between each CMC PC and CME. As much as I wanted to get away from creating files in media folders, I'm a bit stumped on finding a better solution for this.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Sat Nov 09, 2019 11:55 pm

Just a teaser. In v3.0, a full sync of 1758 titles is taking around 57 seconds on my machine.

In the new v3.1 version I'm working on, 43 seconds. That's about 33% faster.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Sun Nov 10, 2019 2:16 pm

Whoohoo! I just blew that time out of the water!

I noticed that updating the GUI was actually holding sync performance back. This is because I was updating the GUI with every directory being synced, and the ability to update the GUI is highly affected by system performance and GPU - on my very high-end machine I could only update the GUI about 50-60 times per second.

So I split the sync into a separate thread. Now the GUI updates are handled by the main thread, while the sync runs at its own speed in a separate thread, unconstrained by how fast the GUI can be updated.

My new record? 36 seconds!!! That's 58% faster than before. I think the improvement might be even better on machines with less graphical horsepower, since now the sync will run as fast as it possibly can with the main speed limits being XML file reading speed and CPU speed.

I think it is amazing that the My Movies API takes nearly an hour to export this data, and CMC imports it all in half a minute.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Tue Nov 26, 2019 4:37 pm

Head's up: I've just posted the first release candidate for v3.1. I would appreciate testing feedback.

Over the past couple weeks I've improved v3.1 even more. Got my full sync time down below 20 seconds. SmartSync is faster too. Improvements have been even better for other testers.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Teddyboy
Posts: 268
Joined: Wed Apr 25, 2018 10:30 pm
Location: San Diego, Ca

Re: Awesome Release

Post by Teddyboy » Tue Dec 03, 2019 11:02 am

Paul,
Slowdown. ;) I haven’t had a chance to play with it yet. I had five days off, and I have no idea where the time went. You’ve done a tremendous amount of work. I really like what you’ve done so far just by reading the threads.
Mechanical Engineer, Solidworks junkie, Reef aquariums, sci-fi, weight training, orchids, Ferns, Kaiju, model kit building, balboa park, scripps aquarium

User avatar
Pauven
Posts: 2777
Joined: Tue Dec 26, 2017 10:28 pm
Location: Atlanta, GA, USA
Contact:

Re: Awesome Release

Post by Pauven » Tue Dec 03, 2019 2:38 pm

Teddyboy wrote: Tue Dec 03, 2019 11:02 am Slowdown. I haven’t had a chance to play with it yet.

Better buckle up, progress waits for no one... 8-)


Teddyboy wrote: Tue Dec 03, 2019 11:02 am I really like what you’ve done so far just by reading the threads.

If only everyone was so easy to please.
President, Chameleon Consulting LLC
Author, Chameleon MediaCenter

Post Reply