Converting DVDs for viewing on a tablet, while inlining captions

Previously, I  described how to convert HDTV videos for my EEE Pad Transformer.  Now, I'll go over something a bit more difficult.

My wife and I have some DVDs of Bollywood films that we enjoy watching.  Aaja Nachle, Om Shanti Om, 3 Idiots, Billu, among others.  These films are mostly in Hindi, but there are English subtitles available.  As we don't understand Hindi, we watch the movies with the subtitles.  The Android media viewer that comes with the tablet doesn't have a way to select subtitles from an alternate video stream.

Now, I wanted to make files of these movies that I could watch on the Android tablet.  As noted in the previous article, the resulting files have to be H.264 Baseline profile, and under 2GB in size.

Here's how I did this.  Note that this procedure required no less than 70 GB of free disk space to hold a large intermediate file, as I wanted to avoid artefacts introduced by running through multiple codecs, so I used a lossless intermediate state.

First of all, I used the MythTV option to rip a perfect copy of the DVD.  That gave me a file, say 3IDIOTS.vob.

Next, I used mencoder to inline the captions directly into the video stream:

mencoder -ovc lavc -lavcopts vcodec=ljpeg:aspect=16/9 \
    -vobsubid 0 -oac lavc -lavcopts acodec=flac \
    -o 3idiots 3IDIOTS.vob

The output file, 3idiots, was, as noted, huge.  It consisted of a lossless jpeg video stream, with the subtitle 0 track overlaid on the video stream itself.

Next, the file had to be converted to H.264 Baseline.  In this case, I decided, rather than setting a qmax, that I would set a bitrate.  That way I could be certain ahead of time what the final size of the file would be, though at the cost of increased trancoding time.  To get a fixed bitrate, it is necessary to run ffmpeg in two passes, once to collect statistics, and the second time to generate the file itself.  Here's how this is run:

ffmpeg -pass 1 -i 3idiots -vcodec libx264 -vpre fast \
    -vpre baseline -b 1400 -acodec libfaac -ab 64k \
    -ac 2 -ar 44100 -threads 3 \
    -deinterlace -y junkfile.mp4

ffmpeg -pass 2 -i 3idiots -vcodec libx264 -vpre fast \
    -vpre baseline -b 1400k -acodec libfaac -ab 64k \
    -ac 2 -ar 44100 -threads 3 \
    -deinterlace 3idiots.mp4 

The "junkfile.mp4" file can be deleted.  The H.264 file, 3idiots.mp4, came in at 1.8 GB, and was of quite acceptable quality to view on the tablet.

Converting HDTV videos for viewing on a tablet

I have an Android-based tablet computer, the EEE Pad Transformer.  My MythTV computer can record digital over-the-air broadcasts in high definition now that I have put an HDHomerun on my network.  So, it would be nice to be able to transfer some HDTV programs to the Android computer to watch them there while traveling.  The HDTV shows are 1080i, encoded as mpeg2 video, at a bitrate of close to 16000 kbits/sec.

So, what are our constraints?  The Android computer is not powerful enough to play videos without hardware assist, and that hardware assist is only available when viewing H.264 videos encoded with the baseline profile.  It doesn't work on main profile H.264 videos.  Also, the Micro-SD card that I plug into the tablet must be formatted as VFAT, it isn't recognized when I reformat it to any more modern Linux filesystems, so our files are going to have to be under 2GB in size.  Also, the Android screen is only 1280x800, so there's no point copying a 2560x1080 file there, the machine will have to reduce the resolution, we might as well do it before we copy it to the card.

So, a 1 hour show, recorded on the MythTV box, is about 8 GB and in the wrong format.  We convert it in two steps.  First, cut out any commercials and transcode it at high quality.  For network broadcast television that chops off about 25% of the file size, and you probably didn't want to watch the commercials while sitting on the train/airplane anyway.

Next, it has to be transcoded to H.264 Basline.  This can be done with ffmpeg:

ffmpeg -i PROGRAM.mpg -vcodec libx264 -vpre fast \
     -vpre baseline -s hd720 -qmax 30 -acodec libfaac \
     -ab 128k -ac 2 -threads 4 -ar 44100 -deinterlace \
     PROGRAM.mp4

This takes the HDTV .mpg file from mythtv, "PROGRAM.mpg", and converts it.  We use the libx264 video codec, fast settings, baseline profile, formatted for a high definition 720 line screen.  "qmax" sets a limit on quality loss, I usually use a value between 25 and 30.  We use the FAAC audio codec at 128kbits/sec, deinterlace the result, and write it to "PROGRAM.mp4".

The resulting file, about 45 minutes of air time, is about 600 MB in size.

When your on-the-road ISP blocks your outbound mail

Now, we talked about allowing your computer to relay mail through the home machine when the ISP through which you're connecting has made it onto a block list. What do you do when the ISP simply blocks all outgoing connections on port 25? Now you can't even connect to your home computer to relay the mail.

The ISP does this to force you to pass email through their servers. The hope is that infected Windows computers will just try to open connections directly, and not forward the mail through the ISP servers. As noted in this story, that is not necessarily true.

So, now you find yourself unable to open connections on port 25, but you still want to send email. You could set up your computer to relay mail through the ISP's servers, as described in this earlier article, but that may not be convenient if, for instance, you're accessing the Internet at a relative's home, since they would have to give you their passwords for you to do that.

So, the first thing to do is to check that you can connect to your home computer on the ESMTP port number 587. Telnet onto that port number on your home computer, and if you get a response, then this technique will work for you.

First of all, you should already have set up relaying as described here. If you set it up a while ago, verify that your keys are still valid and haven't expired.

As we're discussing this in the spirit of a temporary work-around, we'll be editing the sendmail.cf file directly. First, of course, make a backup copy of your current sendmail.cf file, because you'll want to reset it to its former behaviour after you stop using this particular ISP.

Now, go into your sendmail.cf file and find the smart relay line. It will look something like this:

# "Smart" relay host (may be null)
DS

Change that line to indicate that you're sending ESMTP to your home machine. It will look a bit like this:

# "Smart" relay host (may be null)
DSesmtp:mail-host.example.com

Next, we have to tell sendmail that it is to use port 587 for outbound mail to esmtp smart relays. Locate the block in the sendmail.cf file that looks like this:

Mesmtp,         P=[IPC], F=mDFMuXa, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, L=990,
                T=DNS/RFC822/SMTP,
                A=TCP $h

and change the last line to read:

                A=TCP $h 587

That's it. Restart the sendmail program, and you should be able to relay all mail through your home machine using authenticated relaying on port 587.

Web browsing behind the great firewall of China

I sometimes spend time in China, and while there, I work remotely to my office and to my home computer. I do somewhat technical work that sometimes requires online research, and it's annoying that a significant fraction of non-Chinese sites are unreachable from China.

The thing to remember is that the firewall isn't there to keep me from working. I'm a Canadian passport holder, and they really don't care what I read while in China. That explains certain curious omissions, such as the fact that TCP port 22 (ssh) is not blocked.

So, here I am, in China, with a Linux laptop, and I'd like to browse the web. Rather than take my chances with the firewall, I proxy the connection through my home computer's apache daemon.

So, first I set up the proxy service on my apache. Make sure you've built the httpd with these configuration options:

--enable-mods-shared="proxy proxy-http proxy-connect"

These settings turn on the proxy service and set it to proxy HTTP traffic. The "proxy-connect" flag allows the httpd to be used as a reflector for SSL connections. If you want to visit a banking website, the data still travels as SSL between your laptop and the home machine, but the home machine just reflects the traffic to the bank without knowing what's in the data stream (the home machine cannot decode that data, if it could, it would count as a man-in-the-middle compromise of the SSL stream).

Next, add some lines to the httpd configuration file. Mine's in /etc/apache/httpd.conf.

LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so

<IfModule mod_proxy.c>
ProxyRequests On

<Proxy *>
 Order deny,allow
 Deny from all
 Allow from 127.0.0.1
</Proxy>
</IfModule>

What this does is to enable proxying, but only on connections from localhost. I don't want my httpd to be a proxy for any random person in the outside world.

Next, I set up my ssh on connections to my home computer. You can either add a switch like this to the invocation:

-L 8080:127.0.0.1:80

or you can add a line to your ~/.ssh/config entry for the connection to the home computer:

LocalForward 8080 127.0.0.1:80

Now, you ssh into your home computer.

Finally, you start up firefox, and select the menu item:
Edit->Preferences->Advanced->Network->Settings
Select "Manual proxy configuration", and point your HTTP and SSL proxies at "localhost" with the port number 8080.

That's it, now when you browse websites, the HTTP-related data stream appears simply as a pile of encrypted bits over your ssh connection. The firewall cannot know what websites you're visiting, it can't even tell that you're visiting a website at all.

Important note: this system proxies the HTTP data. That means web pages, frames, images in the page, RSS feeds, and so on. It does not proxy UDP or post-connection traffic, like youtube videos. If your web browser has a plugin that downloads data from an external site, that plugin may not be using your proxy.

If you want to know what data is not passing through your proxy, you can run tcpdump in another window. Something like this:

tcpdump 'host <IPNUM> and not port 22'

where is the IP number of your external interface (not 127.0.0.1). You may have to add a "-i" switch if your laptop has more than one network interface. This command will show you all traffic that is not going over the ssh connection.