Site News

[Maiku_Ando]

Aug 31st - Sept 4th, AKA What Happened??

Over the weekend the main server we use for this site suffered a hard drive failure, we have therefore had to re-install the server's OS from scratch. The anime and site code were stored on a separate drive, which appears to be ok. However, the databases which the site runs from were lost. The only backups were also stored on the same drive.

Most of the site is now back up and running, the DDLs and MATV for Fansub Groups are all working including adding new torrents/releases, accounts were re-created so contact Maiku_Ando on irc if you need the pass changing. Animes are being restored all the time, hopefully most of them will be back by the end of weekend.

The forum was also lost in the hd crash and still needs some work but its now working so please re-register and feel free to add in posts or threads again.

If you wish to talk to us you can also join us in #matv on irc.rizon.net.

 

[Maiku_Ando]

Missing / Outstanding Content
Items in this list currently do not show up on the site, and with the exception of Ichiban Ushiro no Daimaou cannot be downloaded.
(Ichiban Ushiro no Daimaou can be downloaded using our DDL links on www.ryuumaru-fansubs.com).

The following items are present on the server but do not have release data (need to be manually fixed/added):

Anime:
[AS1619] Ichiban Ushiro no Daimaou Eps 1 -12. Done.
[SIZE=10pt][AS640] Da Capo Second Season[/SIZE] Eps 14 -26. fixed.
[AS1011] [SIZE=10pt]Lovely Complex Eps [/SIZE]1-9,11-24. fixed.
[AS1382] [SIZE=10pt]Hayate no Gotoku!! (S2) Eps 1-25.[/SIZE] Fixed. (SS-Eclipse releases).
[AS1267] [SIZE=10pt]Nogizaka Haruka no Himitsu Eps 1-12.[/SIZE] Re downloading eps 1,5,7,8,10,11,12
[AS1474] [SIZE=10pt]Princess Lover (DVD) SP1.[/SIZE] Done.

Dramas:
[SIZE=10pt][AS70] You're Under Arrest - Live Action[/SIZE] Done/fixed.
[--] Maid In Akihabara (all) Done/fixed.
[--] Maid Deka (all) Series page and EP1 Done/fixed.

The following are missing files, this is becuase we downloaded RAWs insted of subs:
(This is not related to the HD crash, but I'm listing it here as the original thread has gone).
Maid Deka Eps 2-End.

.

 

[Maiku_Ando]

Site News: '403 Forbidden' Errors Sept 4th - Present

We have noticed some problems with the DDL server giving 403 errors when it shouldnt, this is due to a misconfiguration when the images vhost was added, it appears to be counting connections to the images vhost as well when checking download connections. If you recieve this error, simply wait 5-10 seconds and refresh the page and it will give you the file. Unfortunatly the server is busy and the fix will require a process restart, I dont want to kill everyones downloads so please bear with it and it will get fixed at the next quiet period.

 

[Maiku_Ando]

Site News: 27 Sept - Short Downtime

We needed to poweroff the server for 2-3 mins to remove a dead hard drive (from last month), I also took that opportunity to adjust the DDL server config to hopefully fix the 403 problem. All done, and service fully back up.

 

[Maiku_Ando]

Site News: AnimeSuki Troll & Late Episodes

AnimeSuki decided to be really anal and remove like 5 of the shows which were just about to end, citing "At request" as the only explaination. As we use their feed for episode informaiton we didnt recieve the last episodes to those shows. We also didnt notice straight away which is why there was a delay. We have now manually added in the rest of the episodes for those shows. We'd like to thank AnimeSuki and whoever made the "request" for being total assholes, what the hell is the point in sending take downs with 1 episode left????

Manually resolving the ending season is not too much of an issue, however we will be having a problem with the following new shows:

Sora no Otoshimono Forte
Hyakka Ryouran Samurai Girls

The latter of the two (Hyakka) is being released by 3 of the groups we host DDLs for, therefore the releases will atleast arrive on the server in a timely maner, but it will still require manual work to get them on the site, this will take 'slot 1'.

As a general rule, MATV will not host more than 1 'manual' show per season, simply because of the need to keep track of the releases etc, this season I will stretch the rule to 2 (for Forte), however we do not know if anything else is going to get stream licensed or not. Therefore, we'll just have to wait and see what gets the no 2 slot.

 

[Maiku_Ando]

Oct 24th
Well as you can see the site is having issues again. And again its a drive problem. We are now running the site from one of our backup locations (Natsumi), this allows the main site to work, but no downloads or screenshots. The forum is also offline, however this can be brought up fairly easily. At present we are waiting to see what we can do / how long it will take etc before we do anymore.


Oct 25th
The site is back up and fully operational for the moment now on a replacement drive. The old drive is still present and will have to be removed at some point, which means there be about another 20 mins of downtime sometime within the next 48 hours, but other than that we should be all ok from now on. We hope.

<The above items were moved from the front page>

 

[Maiku_Ando]

Nov - Dec 2010: Misc Downtimes

There have been several brief outages during this time, one for removal of the second failed hard drive, 1 as a precaution due to a security/malware issue and 2 others due to a bug in the operating system.

The bug issue has now been tracked down to its root cause and we should be able to work around it for the moment, the other 2 issues have been resolved for the moment, however we are looking into ways to make the system more reliable/fault tollerant for the future.

Update: '403 Forbidden' Errors originally reported Sept 4th

The original fix for this issue did not fully resolve the issue, therefore we split off the images host from the ddls back in early November and this has been confirmed as fully fixing the issue. Any further issues should be reported to us as usual in the forum and we'll investigate.

<The above items were moved from the front page>

 

[Maiku_Ando]

New Features Dec 2010
We have some new features which you may have noticed appearing around the site the last couple of weeks, they are now officialy 'live'.

Anime News Network Feed
We now have a feed from ANN which is shown on the front page to give you the latest Anime news, we may integrate this further at a later time.

YouTube Related Videos
We have added a 'Related Videos' section to the series page powered by YouTube, this will automatically show the latest Youtube videos related to the series your viewing. This will be expanded later with more features, do let us know what you think of it.

Edit - The related videos feature has been pulled for the moment as it has a bug. Were working on fixing it, sorry. Fixed.

 

[Maiku_Ando]

Upgraded to DDL-V6 Code (& Broken Links on V5)

One of the servers we use for hosting some of the 'Ecchi & Hentai' titles has become unavailable, unfortunatly we didnt know and the current multi-server support dosnt check server availability before giving the links. This meant 2 titles had broken links. We currently do not know if we can use that server again, however we have activated links using another server.

The V5 code dosn't support disabled / unavailable file links therefore we quickly released V6 of the DDL code to get that working. This means we can leave the links in the database, and if we get the other server back up it can load balance between them.

All files/links should be working again.

 

[Maiku_Ando]

Lots of news on what we are planning and working on right now...

Upgrade Plans
We need to replace our current main DDL server (Naru) due to some long standing issues, mainly down to not having RAID and we dont want to loose all our data for a third time. Our data centre had been offering a nice deal for bigger drives and RAID for a lower price than our current machine, with more ram and same CPU. We were considering this as one option however they have now removed that offer, the other was to use colo and buy our own server, which would be much cheaper monthly, but we'd have the hardware cost. After about 6 months we'd be better off with that option.

Another idea was to simply replace the one big-ish server with 2 smaller ones - less CPU/RAM, about the same HD not in raid and because its 2 servers we'd actually double the bandwidth. Lack of raid wouldn't matter becuase we'd duplicate the data over both servers. We were also considering other ways of getting more bw, either upgrading to gig on Naru or adding in another 100mbit line.

Unfortunatly our ad revenue has dropped so much that the above plans may take the cheaper option, or maybe go for a smaller server or less bandwidth. Details here.

Haruka

I'd also like to announce the addition of a brand new server called Haruka, named after the character from Namco Bandi's The iDOLM@STER games on Xbox 360 and PSP. Haruka is a virtual server which is Cloud hosted at gigabit speed in London, and is currently in testing, the idea is to use this as a more reliable place to store the DBs and run the site from (basically the web pages). It may also be used to host the screen caps. Haruka is also running our NMS (Network Monitoring System) and I'm currently working on integrating this into MATV XI (MATV version 11) which is what Haruka is running a beta of at present.
As a sub note as we talked about finances above, Haruka was commissioned before we had the large drop in ad revenue and is actually only costing us 12 to operate, unfortunatly Haruka cant run our DDLs due to her small size and bw limits, she is insanely fast and small, perfect for the DB and site, but not for ddls.

MATV XI
The current site code (MATV X) has been running very well over the past 10 months, with quite a few additions and changes bolted onto it in various ways along the way. Version x (10) was built out of an urgent need to move away from the Andromeda platform which we were using. Andromeda was a great solution, but it wasnt designed for our uses, and so I decided to write my own. After running it for almost a year, I've found a lot of faults and areas where it could be better, and its those areas which have brought about XI (version 11).

Some planned features:

• Redesigned front page.
  With the old listings moving aside (still there but more like a menu now) and a much better and cleaner looking display of the latest episode updates. No longer will you have to dig around to find out if episode X is out yet.
• Youtube Related videos
  Every anime (or other item) will now have a selection of regularly updated Youtube videos.
  This is already present in a reduced form, and will be better in the final XI release.
• NMS Integration
  The site will talk to our network monitoring system to detect which servers are up before sending you to a download location. If a server goes down, the site will no longer send you to that url.
• Multi Server Support
  This is part of the NMS above, and means we can keep the site running even if we loose a server. Haruka is a big part of this, running the NMS as well as being a cloud hosted server itself to hold all the DBs and site code.
• Code Upgrades
  Removing and tuning the code to be more reliable and perform better. Also some security updates.
• Performance Modifications
  Changes to the code and backend of the site as well as server changes to make it run faster.
• Re-design of many core functions
  Some features require a re-write of some of the backend code to make them work.
• Re-design of the DDL system and the 'MATV For Fansubs Groups' site.
  This is the system used by Fansubbers to seed and release the files on the site as well as provide DDLs on their sites. This hasnt seen much done to it since the original version when we were still running Andromeda. Its currently struggeling to keep up now we have so many files, and needs to be updated. Therefore I'm going to not only update it but extend it to add in a lot of the features the groups have been asking for.

This is a big project with a full code review, therefore expect a lot more changes to be added later. I'm also very interested in any suggestions, feel free to send me a PM, find us on IRC - irc.rizon.net #matv or you can post in the other thread.

 

[Maiku_Ando]

MATV XI & Haruka Go Live!

I'd like to announce the launch of our NEW Site for 2011 on a brand new server called Haruka, named after the character from Namco Bandi's The iDOLM@STER games on Xbox 360 and PSP. Haruka is a virtual server which is Cloud hosted at gigabit speed in London, the idea is to use this as a more reliable place to store the DBs and run the site from (basically the web pages). It is also used to host the screen caps. Haruka is also running our NMS (Network Monitoring System) and I'm currently working on integrating this into MATV XI (MATV version 11). Haruka is now running the live site and the main database, the forum and gallery are still on Naru. Haruka has a beta of our revised gallery, but due to space limitations it can't hold everything.

As a sub note Haruka cant run our DDLs due to her small size and bw limits, she is insanely fast and small, perfect for the DB and site, but not for ddls.

MATV XI
The previous site code (MATV X) has been running very well over the past 10 months, with quite a few additions and changes bolted onto it in various ways along the way. Version x (10) was built out of an urgent need to move away from the Andromeda platform which we were using. Andromeda was a great solution, but it wasnt designed for our uses, and so I decided to write my own. After running it for almost a year, I've found a lot of faults and areas where it could be better, and its those areas which have brought about XI (version 11).

Some new features:

• Redesigned front page.
  With the old listings moving aside (still there but more like a menu now) and a much better and cleaner looking display of the latest episode updates. No longer will you have to dig around to find out if episode X is out yet.
• Youtube Related videos
  Every anime (or other item) will now have a selection of regularly updated Youtube videos.
  This is already present in a reduced form, and will be better in the final XI release.
• NMS Integration*
  The site will talk to our network monitoring system to detect which servers are up before sending you to a download location. If a server goes down, the site will no longer send you to that url.
• Multi Server Support
  This is part of the NMS above, and means we can keep the site running even if we loose a server. Haruka is a big part of this, running the NMS as well as being a cloud hosted server itself to hold all the DBs and site code.
• Code Upgrades
  Removing and tuning the code to be more reliable and perform better. Also some security updates.
• Performance Modifications
  Changes to the code and backend of the site as well as server changes to make it run faster.
• Re-design of many core functions
  Some features require a re-write of some of the backend code to make them work.
• Re-design of the DDL system and the 'MATV For Fansubs Groups' site.*
  This is the system used by Fansubbers to seed and release the files on the site as well as provide DDLs on their sites. This hasnt seen much done to it since the original version when we were still running Andromeda. Its currently struggeling to keep up now we have so many files, and needs to be updated. Therefore I'm going to not only update it but extend it to add in a lot of the features the groups have been asking for.

*These items will be launced later.
This is a big project with a full code review, therefore expect a lot more changes to be added later. I'm also very interested in any suggestions, feel free to send me a PM, find us on IRC - irc.rizon.net #matv or you can post in the other thread.

 

[Maiku_Ando]

Naru Drive Issues
Over the past month we have had a number of drive related problems on Naru, so far we have replaced the sata cables 2 times, and next time we will have the main board replaced. The main boot drive is affected and seems to have i/o problems which then locks up the kernel module, and thus crashes the server. We now have a long term plan to replace this server entirely, however we need to sort out a few other things before we can arrange this, so we'll have to keep ploding along for a bit longer. We hope to get the new server next month (May) sometime.

Haruka Performance
Haruka has been performing magnificently since she went live, but we have noticed that at busy periods she tends to have some minor performance issues, this appears to be down to the way we are running PHP and cgi, we have made some changes to the webserver config and switched to another module. From our initial tests it appears to have made a rather dramatic effect on the site performance and it appears to be running much faster. We wont know for sure until it has been running for a while however.

Disk Space
We are once again running low on disk space, we have enough to complete this season, however we do not have enough for the next. As we are planning to replace Naru, we are working on a totally new solution for download storage. This should be in place before the end of the season.

Ad Revenue Vs Server Costs
Ad revenue is at an all time low, and has been since January. As we are now into the 4th month of this, we have decided to scale back our hosting platform. The cost of hosting the site is now so far from the ad revenue and donations ($0) that we can't keep the server we're on. As the site traffic has also dropped by a huge amount, we are barely using a quarter of the bandwidth we have. Therefore at the end of next month we will be shutting down Naru, and moving to a much smaller server. This will significantly reduce our hosting bill. Combined with the new DDL storage system (above) that will save us about 25% of disk space, we should be able to manage on the the smaller system. After that, we will monitor the situation, if ad revenue goes back up and we run out of disk space we will add on some smaller boxes just for the extra, this is still cheaper than the current solution. If ad revenue dosent go back up and we run out of space, then we will start thinning out the least downloaded animes and/or pick up less shows next season.

 

[Maiku_Ando]

Naru Brief Outage
Naru went offline without warning a couple of days ago, upon investigation the cause was our datacentre staff who decided for no apparent reason to move our server - without telling us, or even bothering to shut it down properly. Further still, when they pluged it back in, they put the network cable into the wrong port, and didnt bother to check. I complained, and they still can't seem to understand why I wasnt happy about it. As a result, I am now trying to speed up our departure from Naru. It is back again for the moment.

Forum DB Migration
Yesterday I migrated the forum database from Naru to Haruka. This means that the header code and the main site will still be able to check existing logins and sessions (as well as load up the pages if your logged in) even if Naru has been unplugged.

Full Forum Migration
After moving the database I decided to go all out and move the entire forum over to Haruka as well, including all the avatars and images etc. This means the entire main site, forum and the main site and forum databases are now moved away from Naru.

Naru - Whats left?
The backend MATV For Fansub groups service is still on Naru, and so are the TorrentFlux applications, the front end (the DDL redirector/landing page and the RSS feeds) are now all on Haruka, which means more will stay up even if Narua dosnt.
DDL Files - Kinda a big item this one, but basically all the actual downloadable episode files are still on Naru - its not easy to just chuck 750GB on a new server.
Gallery - The larger images for the gallery are still on Naru - again due to space limitations.

<edit>
Anime Subdomain Migration
I have also moved the subdomain anime.maikuando.tv to haruka, this was having some issues due to permissions on naru, these were fixed but I have moved it over anyway. DNS takes time to update so both will be in use until its all updated anyway. This is mainly used by fansub groups using the DDL service, in the early versions of the site the DDL landing page used a different domain in case the main site had issues, this is not really needed now the main site is on Haruka, but the domain is still in all the urls.
</edit>

Replacements
I'm currently looking at lots of places for potential replacement servers, so far about 6 places look quite good.

 

[Maiku_Ando]

ADVANCE NOTICE OF DOWNTIME
This is advance notice that the downloads on Naru, the Flux torrent clients TF1 and TF2 and the entire MATV For Fansub Groups will be migrated to a new platform and server in the next 1-2 weeks.

The exact date will be advised closer to the time when its been finalised.

Downtime is expected to be approx 2 days, with approx 10 days of reduced service.

 

[Maiku_Ando]

16:10 BST
Network Issues - Naru DDLs, Flux1 and Flux2 currently down.

This appears to be a datacentre network issue in Denver, we have no current eta on when it will be resolved.

Update 17:25 BST
Still no ETA from the datacentre.
Appears to be affecting several users as there are posts on their forum, however no idea how widespread it is yet.

Update 20:39 BST
Naru, DDLs and Flux (1,2) are now back up again, and everything should be working normally.

No explaination yet for the outage.

Update 21:09 [Final]
The outage was caused by cut fibre outside of the datacentre which caused a routing loop.

 

[Maiku_Ando]

Network Issues on Naru.

We are experiencing a large amount of packet loss to and from Naru.

The DC has offered to swap out the network card, therefore the DDLs and Flux services will be down for a few moments while this is done.

Update 4:00 PM BST
Network card replaced and all services are back up again.

 

[Maiku_Ando]

No Issues.

 

[Maiku_Ando]

Introducing Chihaya!

Finally I'd like to announce our new server, Chihaya!
Named after another Idol from the Namco Bandai iDOLM@STER series of games on the Xbox 360 (and soon to be an Anime!) , Chihaya is the replacement for the rather troubled Naru.
With a total of 1.5TB of storage we will also be getting a space upgrade in the process. Chihaya also marks our move out of Denver and back to Chicago. Our time at FDC in Denver was not a particually happy one, with frequent network issues, buggy hardware (we lost 2 hard drives, had numerious sata cables and even a network card replaced) and datacentre staff who can't understand why it would be a problem to ramdomly unplug and move our server without any notice, we will finally be getting out of that mess. We are still with FDC, but their Chicago operation is much better. I don't see us ever going back to Denver any time soon.

Chihaya is already taking up some of the downloads, this is mainly because of our new torrenting system. With the new torrenter we can run torrents on any of the servers and automatically release them to the site. The torrenter is still being worked on, and has some minor bugs, but it is already working really well and those following our Dog Days listings will have already seen the speedy results (we have caught up bar 1 episode already).

Chihaya dosn't have a screenshot engine at present, this is because we're using a different layout which the old screenshot engine isn't compatable with, and I'm still deciding if its better to write a new one into the Torrenter release system or just modify the old one, the latter is easier, but its not very efficent, so for the moment please excuse the lack of screen shots from Chihaya based videos.

The current estimate is approx 1 more month before we fully switch over to Chihaya, however Chihaya is about 90% synced with Naru, so if we have any major incidents with Naru, we can do an emergency redirect to Chihaya, but we'd prefere to keep using Naru a little longer, as this gives us time to work on fixing some of the little but annoying issues which we don't normally have the chance to work on.

 

[Maiku_Ando]

Welcome Hitode Fansubs!

We're pleased to welcome Hitode Fansubs to our DDL service, MATV will now be providing official direct downloads for current and past Hitode releases, they are also the first group to use the new version of the ddl service and the new torrenter. Please look out for their releases right here on MATV!

 

[Maiku_Ando]

CURRENT SITE ISSUES
Since sometime before 9pm GMT on Sunday we lost all connectivity to Haruka.
Haruka runs our DBs, the backhaul VPN and is the main www server, and runs the forums.
With those services down, all the release services (Flux, Torrenter etc) all went down and the main site also went down.

We were advised this was caused by a core router in the datacentre having issues. For some reason all of the virtual machines / cloud servers were also shutdown.

This morning, around 10am we got the server back up, but its running very slowly, its having real trouble running the website and the DBs.
We have shutdown all non-vital services, but its still having huge problems. In order to keep the other site functions working, the DB takes priority, therefore when things get too loaded, we are shutting down the web server (the part that runs the webpages, the user interface basically) this allows the other servers to keep running, and allows the sub groups to add releases - which we have several queued up right now.

If your using the site and it drops off / stops working, please just wait a few mins and it will be back.

Update 17:30 GMT
After a rather long conversation and demanding a manager come on the phone I finally found out what was going on. It appears that datacentre 5 in London had a centre wide cooling failure due to the intense heatwave we're having right now. The equipment heated up so much it got to unsafe levels and had to be shutdown, pritty much all of the servers, switches, routers and the SAN storage units, basically several entire 'clouds' and most of the datacentre. They then brought in mobile cooling units to cool down the powered off kit back to safe levels, and eventaully started some of it back up again. They now claim everything is ok.

However, the VPS cant take any load at all, and is having serious lag issues even just running the DB and the website. Given the cooling issue, and them running on mobile units, heres a rather solid bet they are running with the bare minimum machines up, literally maxing out the servers to keep as many of them powered off as they can, that would explain the huge loss of processing power, and why the VPS is having such a hard time doing anything. The heat in that datacentre must be intense right now, even with all the mobile cooling.

Heavy rain is expected tonight, hopefully it will bring the temperature down to something reasonable, and maybe enough to bring a couple more servers online, then hopefully I can actually check my mail :/

Update 23:30 GMT
Decided to try rebooting Haruka to see if it clears things up at all. Seems to have helped, we now have everything running again, albeit a little slow still.

Update Wednesday/29th
We get Haruka through itsFido, who in turn get the cloud service from UK2, UK2 however has been rather difficult and extreamly unhelpful over the problems we have had since Sunday, their management clearly dosn't give the slightest care about customers with small VPS accounts, and even dosn't seem to care about re-sellers or bigger clients either, from what we have heard from our upper provider, they havnt even bothered replying to them or even doing anything about the complaints. We could get another provider and keep our deal with itsfido (they offered to move us), but we'd still have to rebuild the server, and to be honest the work involved is not something I have time for right now. So if your doing hosting, we say steer clear of UK2 if you want replys to emails or tickets.