Metadata and Plex update

From the Plex Blog here

Plex » Metadata Update

So what exactly is metadata? Defined on Wikipedia as “data about data”, it turns our collection of media from a drab list of files into an interlinked web of facts and pictures, and imbues each item with a rich set of properties. Your files might have structure, but painting them with metadata converts that simple collection into a multi-dimensional universe of relationships. You can now set about answering complex questions like “Do I have any romantic comedies from the 1990′s that I haven’t watched in at least a year, starring Julia Roberts, and not featuring Whoopi Goldberg?”

Everyone loves metadata, and we didn’t anticipate the extreme load the Plex/Nine release would put on a number of sites when we launched. Tens of thousands of early downloaders, eagerly rescanning their huge personal media collections, contributed to massive amounts of traffic to multiple sites.

As a result, we’ve had to spend quite a bit of time since the release focusing on stabilizing our sources of metadata, optimizing the metadata agents (the bits of code that go out and get your metadata), and adding infrastructure to support all of our new users. Here’s a summary of what we’ve done:

• We’ve brought up a massively powerful machine to serve as our TheTVDB proxy cache. All requests to TheTVDB go through this machine, and it serves over 99% of all bytes out of its cache, which means we’ve reduced the data load on the parent site by a factor of 100x. At peak, we were serving over 500 requests/second, and sending out 320Mbps. Darrin, one of our super-talented Plex engineers, worked literally day and night to get this running after the release, and we also appreciate the help and support from the TVDB guys!
• For movies, we’ve moved to using data that is accessible with an API or through structured data dumps. Specifically, we’re using metadata from Freebase, Wikipedia, and TheMovieDB (as well as a few others for extra artwork, such as MoviePosterDB). This ensures the best availability and stability of the data.

If you’re not familiar with Freebase, you should check them out. It’s one of the few sites in recent memory that’s totally blown me away. The people who designed it are very, very smart people and the amount of data available is unbelievable. If you check out the page for the movie 300, you’ll see it links to 33 reviews of the movie, 6 other sites (such as Rotten Tomatoes), and then has a veritable cornucopia of data including cast, genres, subjects, filming locations, award lists, and more. All of that data is available via a sophisticated API, or via weekly database dumps.

We’ve processed the most recent Freebase data dump into a form that’s most suitable for our agent to consume. Additionally, we’ve enhanced the Wikipedia agent to support multiple different languages for the summaries. Finally, much more data from TheMovieDB is being pulled in by that agent.

In summary, massive amounts of data, all structured (no more “scraping” sites that can change at a moment’s notice), and all completely up to you as to how you use them. Like TheMovieDB summaries? Drag it to the top of the list of agents. Prefer your summaries in Swedish? Make sure Wikipedia is above TheMovieDB, so its internationalized summaries will take precedence. Have two French movies for your mother-in-law? You can manually set the language preference to French for just those two movies, and she’ll offer to babysit her grandkids while happily reading the summaries in French.

These agent changes have been pushed, and you will have them within the hour, unless you check sooner with Plex Online > More > Check for Updates.

Get your settings exactly how you like them, shift-click the refresh button to get new metadata for all your movies, and then sit back and watch the metadata flow in. (N.B. At this point in time, poster/art selections are “sticky” so once set, it won’t change unless you rescan a section from scratch).

Here’s a summary of what the different movie agents now provide, so as to allow you to prioritize them accordingly, through the settings option shown below:

• Freebase: Genres, content ratings, studio, directors, writers, actors, tag-lines.
• Wikipedia: Multi-language summaries, directors, writers, actors, studio.
• TheMovieDB: Summaries (more plot oriented), content ratings, directors, writers, actors, studio, tag-lines.
• MoviePosterDB: Lots of movie posters, at lower resolution than TheMovieDB.

So as an example, if you hate the Wikipedia summaries, and prefer English plot summaries, drag TheMovieDB above Wikipedia. If you leave Wikipedia enabled, summaries that aren’t found from TheMovieDB will be filled in by Wikipedia.

If you want your summaries in Swedish, you’ll need to enable Wikipedia and have it higher in the list than TheMovieDB. Note that currently, in order to change languages, you’ll need to create a new section with the new language setting. Alternatively you can “fix match” on an individual item and manually set the language.

Lots of you have asked: How can we help? Luckily this is quite easy; let’s say you have a movie that’s missing data, or has incorrect data. You can head to one of those sites above and add the missing data, and then everyone in the community will benefit, including users of other apps that access those sites. This really is a case where each one of you has the power to help hundreds of thousands of other people!

The most immediate “turnaround” from this data would be through TheMovieDB, which we access through a well-designed API. We cache requests for 4 hours, so if you add data, you will not see the new data for at most this amount of time. (Note that we are also working to improve TheTVDB refresh times, which are now between 24-48 hours.)

Also, if you’re a developer, please check out our repository for agents. They are easy and fun to write, and we’re really looking forward to seeing the creative things you come up with. Oncleben31 has already written an agent for Allociné for French users, and the ever talented Sander wrote one for MovieMeter, for our Dutch users.

In the near future, we’ll allow you to fully customize any of the data for your media and lock it in place, so that it won’t be overwritten by new data from the Internet. So, for example, you can lock all your titles and summaries, but let the ratings and genres continue to expand and improve over time.

Your media has a bright future inside Plex, and metadata is the key.

WOW - Plex is going to be embedded into 2011 LG TVs!

I'm at a loss - this is just beyond impressive, and came out of nowhere:

Plex and the Future of Television

September 02nd, 2010 | Category: Status,Wow | Author: elan

3diggsdigg

This week has been a big one for TV-related announcements. Apple annouced their revamped Apple TV, and Boxee was quick to reply with their own thoughts on the matter. It turns out that now is the perfect time for us to explain our vision for the future of Plex and television as well.

Today, Plex requires that you have a Mac connected to your TV. As sexy as they are, a Mac Mini is $699. And let’s face it, you probably have a few televisions, so it becomes an expensive proposition to Plex-ify your house, especially now that you can stream your library all over with Plex/Nine. Of course, on the flip side, a Mac Mini is a powerful computer that can be used for other meaningful tasks like 3D modeling, genome sequencing, or World of Warcraft.

Another approach is something like the Boxee Box. It’s $199, which is much better, but still prohibitive for many, and it’s completely specialized. No way you could send your kid off to college with a Boxee Box, although you have to admit, it would probably help his or her social life. Additionally, releasing a custom piece of hardware is not a trivial endeavor, if we even wanted to attempt such a thing. Embedded systems are hard, and the XBMC codebase from which Plex and Boxee are both derived is a large and complex one. And really, at the end of the day, do you want yet another specialized box sitting by your TV?

Even if you get the design right, you have to be able to price it appropriately. With not outrageously different hardware from Boxee, the new Apple TV is half the price. Apple also has a two-fold advantage over Boxee: They are going to be selling their Apple TV in much higher volume (which means lower cost to produce), and – critically – they can subsidize the cost of the device because they make money every time you put your arm around your date and click “Watch” on a movie. Also, let’s face it, it’s a typical Apple product: it works perfectly as long as you don’t stray outside their ecosystem. Your files have to be in their limited range of supported formats, and you only get access to the online content they sanction. It’s not an open platform in any sense of the word, and trust me, I was the first person hoping to be able to run our Plex iOS app on it.

The optimal solution, of course, would be a box that was free, infinitely small, and required no cables. Well, we’re extremely proud to be able to introduce to you, for the first time, the Plex Box, with exactly those characteristics.

How is this possible? Well, we actually have one more “one more thing” to announce: We’re working with LG Electronics (the second largest TV manufacturer in the world) to integrate the Plex platform into their 2011 lineup of Netcast™ connected TVs and Blu-ray devices. So early next year, when you buy an LG Netcast™ TV or Blu-ray player, you will have Plex functionality built-in. Specifically, it will connect to a cloud version of the Plex platform for online content, and, if you happen to have a Plex Media Server running anywhere in your house (after all, who doesn’t have a computer in their house?), you can access your local and online content, in a rich interface, with full metadata. I’ve seen it, and it looks awesome.

I’ve been talking a lot about the importance of getting the architecture right for our platform, and this is a perfect example. Thin clients (LG TV, iOS devices), a smart media server, and plug-ins that can run in the cloud. A single integrated interface to access online content, local content, and personal content.

I can’t even begin to tell you how exciting this is to us. LG chose our platform in no small part because it is OPEN, and that is what makes it special. We have developers all over the world creating plug-ins, helping us evolve the platform, and using it creatively. We wouldn’t be here without them, and it’s been an absolute pleasure working with them over the years. I also have enormous respect for LG, who have great products, massively talented engineering, and forward-thinking management. I’ve been to Korea twice in the last year, and their engineers are super-smart, highly knowledgable, and a delight to work with. They “get” where TV is going, and I have to make a confession – the first time I saw their Plex interface, talking to a remote Plex Media Server and flawlessly streaming content, I had to pretend I had something in my eye. This is a team completely committed to revolutionizing the way we enjoy content, and clearly willing to take chances in doing so, as evidenced by working with a small team like ours.

This is also a massive win for content providers. Yesterday, writing a Plex plug-in would make their content available on a Mac, or a television powered by a Mac. Yesterday, they could suddenly make their content available on 100 million iOS devices. And tomorrow (early next year, technically), they will be able to get their content onto millions of LG TVs and Blu-ray devices. This, friends, is an unprecedented time in history. The distance between content provider and consumer has never been this close or frictionless, and it’s incredible to be a part of.

So what does this mean to you, our dear users? You’ve been so supportive over the years, and this is great news for you as well. It means, first and foremost, that we’ll be able to focus more resources on the development. This will be a full-time job for me and others on the team, which is – honestly – a dream come true. The Plex Media Server is the heart and lungs of the platform, and we’ll be making it rock solid and adding some really, really cool new features. We’ll be bringing it to more platforms, to make it available everywhere. There will be more content providers investing in writing Plex plug-ins, so your online content choices will grow. And next year, if you’re upgrading your TV, or or buying an LG Blu-ray player, you’ll have the ability to get Plex, built in, at no additional cost. Fully integrated into killer consumer electronics gear, exactly as it should be.

And *that* is cool.

It’s been a long journey this past year. Now you finally know all of the cool stuff we’ve been working on, and it’s so great to be able to share it with you. We’ve re-architected our platform for the future, and thankfully, most of that work is behind us. Now we can focus on making Plex more stable, more usable, and overall more AWESOME.

Plex 9, the 24 hour status report from Plex

The official word from Plex on how the release is going as well as some minor issues being addressed:

State of the Release

September 01st, 2010 | Category: Status | Author: elan

0diggsdigg

Just over 24 hours since the release, and this has been an extremely exciting time for all of us. We simply can’t tell you how much we appreciate the outpouring of positive feedback on the release. Over Twitter, Facebook, email, and in the forums, the number of positive comments were astounding to us, and definitely made all the hard work worthwhile. We all want to say, collectively: Thank You.

A few salient things about the release:

• The demand on our server was completely unprecedented. The OS load went over 30 for long periods of time, and there were HTTP and database issues. Isaac jumped in and quickly and skillfully spread our services out over a few spare slices and was able to return things to normal. Isaac, you rock.
• Our mirrors (a big thanks to them!) were hammered badly as tens of thousands of downloaders tried to get the latest release, so in the morning we moved the main download site to S3 to ease of the pressure on the mirrors and get the app into people’s hands faster.
• At this point, all services should be performing well, and we’re continuing to monitor and make adjustments as needed.

We are also tracking at this point a number of issues with the release (nobody’s perfect, right?), and we wanted to give you a quick summary of the more common issues, along with workarounds or resolutions whenever possible. Note that these are not the only issues, just the ones at the top of the list at the moment:

• CRASHES ON STARTUP: We’re tracking these here. The most common reason (fixed in the next release), is if your computer doesn’t have a name. You can easily workaround this by going to System Preferences > Sharing > Computer Name and put in a clever, well thought out name like “Macadamia”. There’s also a crash we’re seeing on Leopard, which will also be fixed in the next release. Please post your crash reports in that thread.
• SCANNERS: First and foremost, we’d like to help you get all your media into the library. There were issues with M4V files stopping a scan, an issue parsing date-base episodes, and an issue with .AppleDouble folder which we weren’t ignoring. We’ve pushed bug-fixes for these issues, and your Plex Media Server should update automatically the scanners within the hour. The good news is that you simply have to do another scan (assuming you don’t already have it set to automatically scan). You might be asleep at this point, and you might wake up with a bunch of missing episodes tucked comfortable into your library, and that, my friends, is magic. MAGIC! (If you’d like to track the progress on the scanners, you can follow our Github repository here.)
• TV SHOW METADATA: Slightly less about magic, and more about laws of large numbers, I’m sorry to report that we, um, melted down TheTVDB today. My sincere apologies to them, and my apologies to you all, as we’ll be without metadata from them until we figure out how to reduce the load. We are working with them as we speak, and hopefully will be able to bring that back online shortly. The good news is that the Alexandria library system has been architected for this exact scenario, and you’ll still be able to scan your episodes into the library, and play them, there just won’t be show and season posters or summary data (until the next time you’re sleeping, when we’ll silently push another update and you’ll wake up with lots of posters). We really like magic, have I mentioned that?
• LOCAL MEDIA AGENT: This is the one that picks up existing thumbs and fanart and such. There were a few bugs in it, and we think we’ve fixed them, but we need to test more. If you’d like to take the new agent for a spin and know what you’re doing, feel free to check it out here. (Note that you’ll need to remove and add the section to get the new art to “stick”, expect improvements here.)
• AFP SHARE ISSUE: There seems to be an issue adding folders which live on a remote NAS. A bit baffling, but we’re looking into it.

There are of course other issues, but these are the ones we’d like to get resolved as soon as possible. So, how can you help? Why, I’m glad you asked. If you’re having trouble getting your media into the library, please post in the forum with your exact directory and file layout (screenshot or ls -lR from the terminal).

A few other tips:

• If you’re seeing pausing when playing video in the iOS app, there are a few simple things you can do. Don’t select “auto” quality setting, and make sure you select a quality that’s appropriate to your network speed. Also, make sure you’re not trying to play 720p content (highest quality) to an iPad/iPhone4 if you have a slow server (less than 2.2GHz). Use this as an excuse to upgrade (“Honey, but don’t you want to watch So You Think You Can Dance without a pause ever time she does a pirouette?”). And finally, if you take the wireless router and tape it to your chest while using the iOS app, this improves reception.
• If you have a huge library, you might want to let the Media Server take a bit of “alone time” while scanning for the first time. I mean, come on, it’s walking through all your files, computing hashes, extracting thumbnails, generating automatic fanart, analyzing the media, getting it all into the database, talking to the Internet for metadata, downloading that metadata and getting it associated with your media. Multiple agents are working together to contribute data. There are hamsters running around everywhere. So just sit back, work on your golf swing or water the garden or something. Lotus position and staring at the Plex Media Manager works well.
• A positive review on the App Store directly contributes to Barkley’s diet. Like every time we get a five star review, I walk into the kitchen, get him a delicious venison jerky treat and feed it to him. Literally. Just remember, only YOU can make Barkley gain 10 pounds.

And now, I really need some sleep.

Plex 9 released!

Plex 9 and the iOS app are released! Download them by clicking this link here. Thanks to all the great people at Plex who worked on this massive upgrade!

Oh - One More Thing

Those crazy cats at Plex are really doing it up! One More Thing! Charge up your iOS devices, its sooner than you think. Click the picture:

Plex .9 arrives Sept 1

Here is the teaser photo from Plex! Sept 1 it is. Coincidentally, or not - the same day as the Apple media event. They've also posted a Q&A which I am re-posting here:


Click here for original link posting.

The Road to Plex/Nine

from Plex by elan

It’s been a long time in development, and we’re finally putting the finishing touches on the first public release. An incredible amount of effort has been spent developing this new version, and I’d like to take a bit of time introducing you to it. I’ll start by answering a few frequently asked questions:

Are you guys being bought by Apple? If you pay attention to how previous acquisitions have gone, you’ll already know the answer to this one. An Apple acquisition is usually followed by immediate disappearance of the service/product in question, sequestered away no doubt to a secret sub-basement in Cupertino. The fact that we’re still here talking and the website is still running is a sure sign that we have not been acquired by Apple. (The date is entirely coincidental, we went “one week out” and ended up there.)

Will my favorite new feature/bug be added/fixed in this release? Probably not. We’ve been uniquely focused (much to the surprise of some, who think we’ve just been playing Angry Birds) on getting the Alexandria library up and running. The only new feature in this release is the official inclusion of Ryan’s hardware-accelerated video decoding.

Can I run the new version alongside the old version? The answer is “mostly”. When you install Plex/Nine it will set aside your old Application Support directories so as not to step on anything. If you hate it and want to go back to Plex/Eight, you can just rename the directories, whack Plex/Nine, and crawl back into that comfortable old pair of Plex/Eight jeans.

Is this new release completely bug free? Um, no. This is the first release of a massive from-scratch rewrite of the library, and it’s safe to say there will be bugs, and even things that used to work that don’t work yet with the new version. However, we hope some of the cool new stuff coming your way will more than make up for it. This release has been through 17 (really!) builds which have gone out to our elite group of super-testers, but there are certainly lots of bugs. Remember what your mom told you: “Hope for the best, expect the worst, and take what comes with a smile.”

Why the @$%#%$ did you guys have to rewrite all that stuff? I’m really glad you asked. Our vision for the future of Plex is a distributed architecture, as opposed to one giant monolithic program. A simple picture might help here:

In order to realize this vision, we had to effectively do two things: (1) rip out massive amounts of code in the client and make it into a “thin client” (essentially just player and skin), and (2) design and build the Alexandria library system into the Plex Media Server.

We consider these major changes to be fundamentally important to the future of Plex. We’re building the foundation for some truly incredible things, and building a strong foundation takes a lot of time and work. Continuing the metaphor, the first release really only shows you the foundation and perhaps a few ground-level floors. However, once you see the speed at which we can add amazing new features, you’ll better understand why we had to take so much time to get this right.

Can you walk me through an example of how this new library is so much better than the old one? One thing I really like is the decoupling of the scanning from the metadata. The scanners are responsible for looking for media, usually on disk, and performing “structural identification” (e.g. Lost, season 3 episode 14″). Media analysis is also performed to get the resolution of the file, pull out a thumbnail, etc. The media is added to Alexandria right at the start of scanning. Note that at no point yet has TVDB or any other Internet site been required. So you can scan new episodes into your library even if the site is down, and see them next to other episodes. What’s more, you can write a custom scanner in a few lines of Python code, which you might want to do if you have a custom filesystem layout for your media. Flexible!

Now the agents, in my opinion, is where things get incredibly cool. They elegantly solve the problem of “how do I bring metadata from the Internet and connect it with my media?”. The agent functionality, along with the other parts of the new plug-in Framework coming in Plex/Nine was designed by James, and I have to be honest, he’s kicked some serious ass.

Plex/Eight used the XBMC scrapers to accomplish that task, and if you’ve ever tried to write one (or even modify an existing one), you’ll know that it’s not an easy task. As Jamie Zawinski once said: “Some people, when confronted with a problem, think ‘I know, I’ll use regular expressions.’ Now they have two problems.” (The only thing worse than regular expressions, mind you, is regular expressions encoded in XML.)

Plex/Nine’s metadata agents use the same (well, an enhanced) version of the Plex Plug-in Framework you’ve come to know and love. This means that powerful XPATH, Unicode, and HTTP functions are available in an easy-to-use manner, and you can write an agent in as little as a few dozen lines of code.

One of the other fundamentally neat things about agents is that multiple agents can contribute to a piece of media. I might want to get my ratings from Rotten Tomatoes and my descriptions from IMDB, for example. The other day Scott said “the movie descriptions on Wikipedia are really good!” and an hour later, this is what my own configuration looked like after I dropped his new agent in:

The “Local Media Assets” agent is the one responsible for finding posters, banners, fanart and such sitting around next to your media. The Wikipedia one is the next highest priority agent, which means that the metadata combiner will prefer its summaries over others.

I think I fell asleep during that last answer. Yeah, I don’t blame you.

How open and extensible is this awesome new library? Well, you tell me. The database is stored in SQLite format, and there is a really nice REST-ish HTTP/XML interface for all areas of the library (which is what the Plex client uses). Want to build a dashboard widget which displays your recently added movies? Easy! Fun! And given that the Alexandria scanners are completely written in Python and support custom-written ones, this new library is much more easily customizable and user-tweakable than the old one. We’re really excited to see what interesting agents and scanners you come up with!

I think I’m out of questions. Good, because I need to get back to Angry Birds.

iTV - We'll find out next month

I believe this is a very high chance of being released next month, and could possibly change things for TV the way iPhone did for mobile phones and touch screens -
SO many possibilities

Kevin Rose, founder of Digg, has heard from sources within Apple that the company will launch its all-new iTV set top box, powered by the iOS operating system, in September for $99.

Rose posted on his blog that he believes the revamped and renamed Apple TV hardware will "change everything" for a number of reasons, including the addition of TV applications written for the iOS operating system. Like the iPhone and iPad, Apple's new iTV will have access to the App Store where users will be able to download software to run on their device, he said.

With these applications, he said, content providers will be able to provide iTV owners "a la carte" stations that Rose believes will change the way people watch content on their television.

"With Apple's iAds, content producers (eg. ABC/NBC/etc.) can directly monetize and distribute their content," he said. "This will eventually destroy the television side of the cable and satellite industry, as your only requirement to access these on-demand stations will be an internet connection. Say goodbye to your monthly cable bill."

Rose also said he has heard that the $99 device will make it easy for users to share pictures and videos "with the push of a button." He suggested the device could include a feature that would notify users when new family photos or videos are available to be viewed.

The Web entrepreneur also said the new iTV will turn Apple's iPad into a remote control, as the "preferred input device" for the set top box. The iPad will also allow users to edit videos, control games, and "extend the interactive television experience," he said. He gave an example of watching football while viewing other camera angles on the iPad.

Rose also made mention of Google TV, the search giant's already-announced Android-powered set top box set to launch this fall. He said that people should keep an eye on Google's product as well. "This is going to be a hot space in 2011," he said.

Rose has reported numerous Apple rumors in the past with varying degrees of accuracy. In 2008, Rose revealed a leaked photo of the fourth-generation iPod nano in advance.

His iPhone-related leaks have been hit or miss, as he correctly predicted the addition of copy and paste to iOS 3.0, but incorrectly said that the iPhone 3G would do video chat. That feature eventually came two years later with the iPhone 4, in the form of FaceTime.

Late last year, Rose also said the then-unannounced iPad would have an emphasis on e-books, and would prove to be a "Kindle killer," referring to Amazon's e-ink device. Apple's iBooks application was available at launch for the iPad.