A Nightmare becomes true – learning from an experienced Rails Developer

I am writing this to give myself air and because I think you can still learn a lot from incidents like this I have experienced here as an old hare.

Imagine: You are solely responsible for a complex software. You have to introduce new essential features for the system. You have slowly brought the new features into the system via “feature switches”. Many already use the new features. Now comes the time when you can start switching off old features and all users more or less compulsively are lead to the new features. This is probably the most difficult moment of a product launch. There is no easy return. Just at this moment happens what all developers and system administrators are afraid of. The system dies and there is a complete data loss for several days.

This is exactly what happened with the UlangoTV 2.0 introduction.

The whole story is so exemplary and classic that I think it will interest many of the technically skilled, how it could come to it and how one can liberate oneself without damage from such a catastrophe.

A fatal crash with big data loss – how can it happen?

Most web and app servers use Linux systems because they have proven to be particularly stable, secure, powerful and inexpensive over the years – so we too. Many administrators are proud that their systems have not been restarted for more than 6 months or more and that they are just stable. Backups are made regularly – today mostly by the providers via snapshot on the lowest level of the raw partitions. To this end, regular DB backups are made with the DB tools, which guarantee the necessary transaction security.

The DB backups are a special problem, if the databases become too big as in our case – over 50GB. Importing such data requires a lot of time and leads to a great downtime in the event of a disaster. The only way to achieve high availability is the introduction of redundancy. How this helped in our case relatively quickly back on the legs, more below.

Now to the crash in more detail. In ongoing operation with increasing system load, it suddenly happens that some processes are no longer finishing their work in time. The load swells, the system begins to swap until practically everything hangs. You quickly identify load causes, services are turned off. But suddenly this does not help anymore because there is obviously a blockage situation in the system (A waits for B, B waits for C, C waits for A – deadlock – you know what I mean). Very bad, if the blockade obviously lies at the bottom of the filesystem. Now the moment has come, where only a reboot can help. And then it happens: The system can not be rebooted because there are inconsistencies in the filesystem. This is usually not too bad, because a modern file system carries enough redundancy in itself to repair itself. Unfortunately, however, this was also not possible in our case. No chance to make the system work again. So a backup had to be restored – from the last day – a few hours before. After an hour we realize that the filesystem is already too broken here, that it is no longer usable. Another backup back – another hour – does not work either. Now it’s slowly becoming critical. In the meantime, we are preparing a new system into which we can upload our backup – uploading the compressed data: 22 hours !!

So a backup back – the weekly backup 5 days back. Hurray – it works. I decide to relinquish the import of the DB and to manage with the data loss of 5 days somehow. Phew

Restoration of data from redundant sources

When it comes to recovering from a data loss as quickly as possible, all kinds of sources can help. This is where the strengths of the Ruby on Rails programming language we use come into play, with the help of which AdHoc programs could be written very quickly. In our case, we had three sources: 1. Central log files, which were redundantly stored on other servers. 2. Our external order data at PayPal and 3. Our Riak-based key-value storage for channels and streams.

From the logfiles, it was relatively easy to restore user data – apart from the passwords, of course. With an AdHoc program the data were imported and the users were notified by e-mail to reset their passwords.

The restoration of the order data proved to be much more difficult, since the order information was also lost, and it was not always possible to associate the payments with users.

Finally, relatively easy was to restore our central database for streams and channels. We had transferred this data to a so-called key-value store (Riak) a long time ago, in particular in order to distribute the load on requests as well as the redundancy achieved over several servers. It is the key to a virtually unlimited scalable system. It is a technology that is used today in all large systems and was used for the first time by Amazon in a larger style (Dynamo).

Conclusion

For us, the following are the main conclusions

  1. More frequent reboots of the system to detect “creeping” destruction of data, which then also migrates into the backups, early.
  2. Storage of the DB Backups close to the server in order to make it available as quickly as possible.
  3. Improved centralized logging (syslog daemon)
  4. Relocat even more data – especially users and order data – in distributed KV stores to minimize the “single point of failure” situations.

So – now I’ve got rid of it and I feel much better and ready to tackle new challenges at UlangoTV!

Facebooktwittergoogle_pluspinterestlinkedinmail

UlangoTV 2 – Yellow – Blue – Magenta now makes sense!

Now it’s colorful! From UlangoTV 2.0, IPTV streams and channels will in the future be divided into 4 color categories of usability to provide us and our users with a legally secured status. For a long time, we have been working on how to bring a version of UlangoTV to the App Stores, which can not lead to complaints from content owners and is completely safe for the users. Result is a color scheme, which we now introduce with this version. The colors are already known from our logo: yellow, blue and magenta. In addition, color red, with which we identify channels, which are adult or for which there is an official request, do not show these channels.

In the Yellow basic version the app is free for all registered users, but will be delivered with advertising in the future. In the Blue version, users can access more channels via their own search queries, but they are likely to be legal broadcasts too. The magenta version also covers streams, which are only covered by short-term test streams of possibly illegal broadcasters.

Yellow: Streams for all registered users free!

Gelb: Streams für alle registrierten User kostenfrei!

Yellow streams have been known to us for more than 6 months. Registered users can see these streams for free. Adult streams are excludes from this category.

For yellow streams, we assume that they are sent legally by broadcasters. Typically these ‘official’ streams are without license problems. Restricted to yellow streams the app can therefore be used by anyone without problems.

Once we learn that a sender  is protected with exclusive global distribution rights, the corresponding streams are red for all users.
As soon as we learn that a channel is protected in a region with exclusive distribution rights, it will be red for registered users from the region.
As soon as we learn or see 😉 that a station is adult, all the related streams are red.

Yellow streams take a special status in that they no longer participate in the short-term periodical quality control. So we can not remove them so easily from the system. This is done manually only by administrators, if there are a lot of corresponding user reports. The yellow streams run in the long term, but their quality is often less good.

The higher resolution streams are often found among the blue and magenta.

Yellow streams are free for all registered users..

Blue: Streams for Premium or Trial with unknown license status

Blau: Streams für Premium oder Trial mit ungeklärtem Lizenzstatus

Blue streams have been known to us for at least 6 weeks. Also these are usually ‘official’ streams without license problems. Trial users can see these streams, but you must explicitly give us search orders so that we can search for them on the Internet.

After 6 months, blue streams will automatically turn yellow.

For blue streams, too, we assume that they are legally sent by broadcasters. 6 weeks availability on the net is a long time and one can assume that the operators of the server are either particularly bold or have a license for the distribution.

However, we are cautious in these cases, and the user must give us a special search order on the “My Search Orders” page of our website, which we then execute in his responsibility to release the streams. He can take back the release at any time.

Analogous to the yellow streams:

Once we learn that a channel is protected with exclusive global distribution rights, the corresponding streams are red for all users.
As soon as we learn that a channel is protected in a region with exclusive distribution rights, it will be red for users registered from the region.
As soon as we learn that a channel is adult, all the related streams are red.

In the settings on the website you can adjust, that only yellow or also blue streams are visible. Yellow is the default setting.

Blue streams require at least a trial subscription, which registered users can only use at most once within six months. In special cases, e.g. now with the introduction of 2.0 this 6-month interval is reset.

Magenta: Probably illegal streams only for Premium users under tight restrictions

Magenta: wahrscheinlich illegal gesendete Streams nur für Premium User unter Einschränkungen

Magenta streams are usually short-lived and have only been known to us recently. These streams are likely to originate from unlicensed sources. Premium users must give us a special search order to search for such streams at their own risk. The streams with the best quality can be found in this category, but with high fluctuation, since they are mostly test streams with a very short period of validity.

After 6 weeks magenta streams turn blue automatically.

For magenta streams, we are even more cautious and the user has to give us a special search order, which we then execute in its responsibility to unblock these streams. He can take back the release at any time.

In the settings on the website one can adjust that yellow, or also blue or also magenta streams are released. Yellow is the default setting.

Only premium users can see magenta streams. A premium subscription can be ordered via the UlangoTV Shop on the website.

Red: Protected streams for users who know what they are doing.

Rot: geschützte Streams für User, die wissen, was sie da tun.

There are always users, who find ways to make protected streams visible. The method is simple: one searches the network for appropriate m3u lists and then hand it over to one of the many available players. Also UlangoTV can be used as a player in this way. The premium user can raise his self-restriction to the red level in the “My Search Orders”. Legal restrictions differ in various parts of the world. The user himself is liable to comply with local restrictions.

 

General Remarks

General remarks

The color of a stream is indicated in the app by a colored button.

The color of a channel is displayed on the website by a colored button. Yellow is not displayed in the app, in order not to disturb the appearance of the lists unnecessarily. All channels also appear in the search engines. In the case of the blue and magenta streams, it is explicitly pointed out that a user can only access these stations via search queries or uploaded m3u lists.

You can test and use the functionality of the system without paying. Effectively, the app is free for free (yellow) streams – which are currently at least 2000 streams

It is now solely the responsibility of the users, what they take from the search results for IPTV streams. With the color scheme, we are the only service provider to differentiate the set of streams and make it visible and controllable for everyone, what is to be included in the search results. Because we deliberately do not distribute URLs and make the streams visible only in a “black box player” UlangoTV, we also prevent the possibility that a user uses the URLs to create illegal copies or for “re-streaming”. This allows us to meet the requirements of content owners, so that the app can now be distributed without any doubt in the app stores like any other IPTV player.

Facebooktwittergoogle_pluspinterestlinkedinmail