With the Automatic Content Enrichment (or ACE) project, we wanted to offer a solution which would address the problem signalled to us by many of our business partners offering video streaming services and monetizing their content on the internet. Many of these providers were looking for a way to analyse their video content (both live streams and uploaded files) in real time and display some additional, contextually relevant content alongside it – or even substitute some parts of the original content with completely new content. We wrote about our initial goals and capabilities when developing the technology in an earlier post on the blog.

Our solution, naturally, lends especially well to targeted advertising applications, e.g. advertising breaks can be filled with the exact length of contextually targeted ads that are different for each household. However, we envision many other conceivable use cases of content enrichment. For example, additional information about the actors/products/football players can be generated and displayed automatically whenever a specific image/face is detected in the content. 

We got the patent!

Due to the innovative technology behind the invention and unique character of the solution itself, we decided to apply for a patent. It took a really long time since the submission of our patent application on 03.08.2018 and a lot of paperwork until the receiving of the official patent document, but it was worth it. 

Once positively reviewed by the Patent Research Department of the Patent Office of the Republic of Poland, the patent was officially granted on 6.07.2020 – you can see the scan of the official certificate below.

BSG patent

What are the basic requirements to get your product patented?

You may be wondering why it isn’t so common for software companies to patent their technology. Firstly, getting a piece of technology patented is no mean feat and requires not only patience but also great knowledge and skills in the area. It’s the inventor’s responsibility to properly articulate the purpose and workings of the invention, and provide ample evidence that it is not only unique and patentable, but also valuable to the general public. 

For example, one of the key requirements from the Patent Office of the Republic of Poland is that the application under consideration must be presented clearly and comprehensively enough for a skilled person (expert in the area) to implement the invention independently. This basically means the patent claim should be expressed in a clear and concise manner, and supported with detailed evidence and documentation.

Can you patent anything?

The Patent Office will determine whether the submitted solution can be considered an invention in the first place. What’s important and often reiterated by them is that only patentable inventions can be patented. Wait, what? Yes, you’ve heard it right. This might seem like a circular definition of sorts, but we’ve sorted it out a long time ago. To be patentable, the invention must be a unique technical solution to a problem.

At the same time, not all inventions can be protected. For example, according to Art. 29 of the Patent Law of the Republic of Poland, patents are not granted for inventions the use of which would be contrary to public order or morality. However, it is not considered contrary to public order to use an invention simply because it is prohibited by law.

What made ACE a patentable invention?

Our mission at Better Software Group has always been to create software that creates new business opportunities for our clients, and increases their bottom lines. 

Automatic Content Enrichment was created to allow video broadcasters and video content publishers to monetize their content by more accurately matching the advertising message, thus improving users’ experience and increasing the profits from advertising for the publisher.

Upon analysis, the patent office concluded that Better Software Group’s Automatic Content Enrichment officially meets the criteria for a patentable invention. This results from the fact that the technology successfully meets all the requirements:

1. Uniformity of the solution 

The invention meets the uniformity requirement as specified in the Polish Patent Law.

2. Industrial applicability 

ACE is widely applicable in the industry. 

3. Sufficient disclosure 

The Office found that the solution had been disclosed clearly and exhaustively enough for an expert in the field to implement it without additional creative work on their end. 

4. State of the art 

ACE was deemed an invention based on an exhaustive report presenting the current state of the art on the subject of content analysis with neural networks. 

5. Novelty 

The Office confirmed that our invention meets the criterion of novelty (there is no other known solution in existence that serves the same purpose). 

6. Inventive level

An invention can be considered to have an inventive level if it does not seem obvious to an expert in the field. The solution does not have an inventive level if a specialist familiar with the state of the art would, in a professional and routine way and no creative input on his part, be able to come to a proprietary solution using the usual logic of reasoning. 

For example, the following inventions would not be granted an inventive level:

  • An invention that’s the same as another, yet smaller (e.g. Walkman is not an invention in itself. However, the name is a trademark.);
  • An invention which is a combination of several known solutions or elements but does not add value in any way.
  • An invention which has specific components changed to known equivalents.

What is ACE and why do video publishers need it?

The biggest challenge faced by creators and broadcasters is the inability to perfectly match the content and the moment of broadcasting of advertisements to the user’s preferences. There currently are no solutions on the market that enable the effective and fully automatic analysis of video content and parameters. 

The use of artificial intelligence in the media creates a series of new amazing opportunities for our clients. At BSG, we always try to be the forefront of innovation, finding and implementing the latest technologies in the solutions we create. The project developed in cooperation with Warsaw University of Technology will give us the opportunity to actually apply something that not long ago was just a pipe dream. – Bartłomiej Lozia, Co-CEO of Better Software Group.

Automatic Content Enrichment (ACE) addresses this problem using neural processing and makes it possible to, among others:

  • Replace selected advertising breaks with personalized content from the ad server. The users, unaware that they are watching a targeted advertisement,  will receive it as a part of the live content;
  • Analyse of video materials;
  • Provide contextual ads targeted to the currently broadcast video content;
  • Mark the beginnings and ends of individual programs in live broadcast, in order to sell information to the owners of nPVR and CatchUP services;
  • More precisely match the program to its EPG (electronic program guide) position.

program guide

Electronic program guides are menu-based systems that display scheduling information for current and upcoming TV broadcast programming

How does ACE work?

In essence, we wanted to develop intelligent technology that would allow publishers to effectively analyze video materials published on the web, both live streams and files. 

To get more into the weeds, the encoded audiovisual data is sent from the source server as a recording (i.e. a file recorded prior to the broadcast) and received by the processing server, decoded into audiovisual data and analyzed by a neural network implemented on the processing server. This is what happens under the hood:

  • The neural network is configured to detect specific images or sounds, or a series of images or sounds (movies), e.g. to detect a specific advertising content. In other words, the neural network “watches” the retransmission and analyzes it. If it detects that it is viewing a specific advertisement, it informs the sender of the content about this fact (e.g. for reporting and attribution purposes) or just records the fact that the advertisement was broadcast. 
  • Also, the exact time (keyframe) of the start of the advertisement and its end (end-of-ad keyframe) in a given retransmission is recorded. Then the processing server, provided with the information that a given advertisement is part of a retransmission, may, depending on the configuration, save this fact (for statistical purposes) or decide to replace it with a more current one (e.g. new, up-to-date advertising content) or remove this content from transmission. 
  • The replacement or deletion of a given content is performed in such a way that it does not interfere with the content of the retransmission. What’s important, the program is appropriately shifted or shifted in time so that the displayed image is continuous and undisturbed. The changed data stream is sent to a user equipment configured to broadcast the decoded audiovisual data, e.g. a digital data player.
  • The processing server is configured to analyze (view) all channels encoded in the transmitted data stream from the source server. 

The technology behind ACE

ACE uses inference algorithms and modern machine learning techniques. We implemented Scale-Invariant Feature Transform (SIFT) and Gradient Location and Orientation Histogram (GLOH) for the image, and FLUX mechanism for audio data – spectrum stream, spectrograms or mel-based cepstral parameters (MFCC). The whole process is based on a deep, convolutional neural network.

What are the possible use cases of ACE

There are many possible applications of ACE, but while we are aware of a few extremely useful ways to implement the technology, we believe the examples below are only the tip of the iceberg and there are much more to come. 

Use case #1: Better ad reporting

Thanks to ACE, the sender of the context content (advertisements) can get information about whether his advertisement was displayed on the user’s screen and at what time.

Use case #2: Update archival content with up-to-date materials

The advertiser may also use it to update or enrich the advertising content of programs recorded in the past, even archival ones. Thanks to image analysis, ACE makes it possible to adjust the existing content (commercials in ad slots) to the content of the broadcast program.

Use case #3: Use specific images to trigger ads

The processing server can be configured to detect specific objects, e.g. buildings, and on this basis match advertisements to them and to the place of displaying the broadcast. In this situation, for example, a user from Japan watching a program about Paris in the advertising block will see an advertisement for flights from Japan to Paris instead of flights from another country that were advertised in the originally recorded broadcast.

Use case #4: Better ad targeting

The biggest challenge faced by creators and broadcasters is the inability to perfectly match the content and the moment of broadcasting of advertisements to the user’s preferences. There currently are no other solutions on the market that enable such an effective and fully automatic analysis of video content and parameters. 

Summary

If you think your product might benefit from Automatic Content Enrichment, drop us a line and see how we could leverage our technology to boost your bottom line or create completely new monetization use cases for you!