My leisure interests

Martial arts Heritage railways Garden railways Motorcycles DIY

Downloads

Linux downloads Windows downloads Java downloads Perl downloads Home automation downloads

About me

Home & family My CV

Site info

Contact the author Download policy Keyword index

Using the transcode utility on Linux to rip DVDs for the Archos AV500 (and similar devices)

Last modified: Fri Jan 26 10:40:21 2007

[Updated January '07 to deal with 16:9 content presented anamorphically in a 4:3 frame]

This article explains how to use Thomas Oestreich's `transcode' utility, which is available for most flavours of Linux, to rip DVDs to play on the Archos AV500. While the AV500 will play a variety of video frame sizes, I'm assuming here that you want to optimise the video file so it fits exactly -- or as exactly as possible -- on the AV's 480x272 screen. As we shall see, it won't always be possible to get a perfect fit, and compromises will have to be made.
      I assume in this article that you want to rip DVDs to watch on the internal LCD screen, not on an external video display. The AV500 is quite tolerant of a wide range of video frame sizes, so if you ripped a DVD into an MPEG4 file with a fairly large frame size (e.g, 640x480) it would probably play OK on both an external display and the LCD. However, if you just want to play on the internal screen, you can do a lot better -- you can size it to fit the LCD, and get the best compromise between file size and image quality.
      Because the AV500 has such an amazing screen, it's worth spending a bit of time and effort getting things just right. This is where transcode comes in -- because it allows for incredibly fine control over all aspects of the format conversion, if you have the patience you can achieve far better results than are obtainable by point-and-click converters. In addition, because there's no graphical interface, all the CPU power goes into crunching the data, so the job gets done quicker (but it's still time-consuming, even on fast hardware).
      The disadvantage is that transcode is fiddly to use, and takes some getting used to. However, once you've figured out which settings work for you, you've done all the hard work, and after that it's easy.
      This article also provides some background on how DVDs are encoded, in the hope that it will help the reader deal with situations that aren't specifically covered here.
      Please note that I only have much experience with PAL DVDs. The image sizes and frame rates are slightly different with NTSC, so you'll need to calculate accordingly. I'm assuming also that the reader is tolerably familiar with Linux and the use of the command line -- if you've never used a sophisticated command-line utility before, I venture to suggest that getting stuck into transcode is not the way best way to learn. If the idea of using a command line brings you out if a cold sweat, read no further -- there are no shortage of point-and-click converters out there.
      What follows applies to devices other than the AV500, provided they have the same, or at least similar, screens. That includes the more recent Archos 504 and 604, as well as the Sony PSP. This article doesn't really apply to the PMA430 or to the newer Archos 404, because these units have 4:3 screens.

Background

To sum up... if we have a 16:9 output device (like the AV500 screen), and we have a DVD encoded as 16:9, and a 16:9 programme, we don't have to change the aspect ratio at all. We just scale the width and height equally to give the right pixel size for the screen. Optionally we crop off the black bars at the top and bottom of the physical frame, because there's no point filling the ripped file with black bits we aren't going to watch.
      If we have a 4:3 programme on a DVD encoded as 4:3, we can do essentially the same thing -- we scale the DVD physical frame uniformly so that the height of the resulting image is the height of the display device. We can't match the width as well, because the aspect ratio of the display device (16:9) is not the same as the aspect ratio of the programme (4:3). We can watch the ripped 4:3 prorgramme either with black bars at the left and right, or we can ask the display device to scale up so that the image fills the frame, and accept that we'll lose some picture at the top and bottom.
      If you want to be really sophisticated, you can compromise between black bars and picture loss at the top and bottom. So, for example, rather than scaling so that the entire display screen is full, you could scale to reduce, not remove, the black bars, and lose a bit less of the picture at the top and bottom. This is a matter of taste. Most display devices aren't clever enough to do this `variable' scaling operation -- the AV500 can't -- so if you want to try this, you'll need to do it using the appropriate transcode settings.
      In any event, the important thing to realize is that in none of the operations described above had we had to worry about aspect ratio conversion -- the ripped image has the same aspect ratio as the physical frame in all cases.
      I will point out in passing -- completely off-topic, and only because it's a hobby-horse of mine -- that whatever artistic arguments might be offered in favour of wide-screen presentation, wide-screen DVD formats do not offer higher-quality images. Or, at least in so far as quality is measured by the amount of pixel data in the image (and it usually is -- look at digital camera reviews). All 16:9 encoding formats contain 30% fewer pixels than 4:3 formats. The `quality' is therefore worse. But that, as I said, is beside the point.

So now we move on to the real trouble-spot -- `anamorphic' DVD presentations. These are presentations that are intended to be played at a different aspect ratio from that of the physical frame. Remember that the aspect ratio of the physical frame is always 4:3, and even a 16:9 programme retains this aspect ratio, by the simple expedient of what amounts to padding it out at the top and bottom to make it fit the physical frame.
      Anamorphic formats are hard to explain in abstract terms, but easier -- I hope -- by example. Consider a typical cinematic movie presentation, with an aspect ratio of 2.35:1. People often think that 2.35:1 is the same as 16:9, for the simple reason that you often see the two things written side-by-side on the back of a DVD sleeve. But, in fact, 16:9 is 1.78:1. So 2.35:1 is much wider than 16:9. Now, we already know that the physical frame of a DVD cannot change, and we already know that only two encodings are supported -- 4:3 and 16:9. So how do we accomodate 2.35:1? Well, one method would be to proceed in the same way that 16:9 is accomodated. We'll just put the programme image into the middle of the physical frame, fitting it as best we can. If we do this, we find that we need to make the programme image 306 pixels high (720/306 = 2.35 or thereabouts). We can encode a DVD like this, and if we have one it is very easy to display (and to rip, for that matter) because, again, we aren't worried about aspect ratio.
      But remember what I said earlier about the image quality of widescreen formats? If we encode a 16:9 presentation in a 4:3 physical frame, we lose about 30% of the image pixels. But if we put a 2.35:1 presentation into a 4:3 physical frame, we lose 46% of the image pixels. The best-fit image size will be 306x720 pixels, and the quality to will be about the same as that of a VHS tape! Admittedly, it's going to be the same as a pristine VHS tape direct from the studio, not a dodgy one that's been copied about five times and has blank patches in the tape oxide; but all the same it's not going to be `DVD quality' in spirit.
      To get around this problem, the studio will usually distort the aspect ratio of a wide-screen (wider than 16:9) presentation so that it fills the same part of the physical frame as a 16:9 presentation would. That is, instead of encoding the programme as 306x720 pixels, -- required to maintain the 2.35:1 aspect ratio -- it is encoded as 406x720, as a 16:9 presentation would be. We get an extra hundred vertical pixels in each frame -- and therefore a much better image quality -- but at the expense of a distorted aspect ratio.
      You might be wondering why, if we're prepared to distort the aspect ratio at all, why not go the whole hog an put the 2.35:1 movie into the whole physical frame? Since we've got to correct the aspect ratio on display anyway, why does it matter how much we distort it? We'd certainly get better-quality images if we did this. The answer lies in a consideration of what DVD was originally intended for -- it was designed for showing movies on bog-standard television sets. If your television set cannot do aspect ratio correction, then if you watch a 2.35:1 movie which has been stretched vertically to be 16:9, it won't look too bad. You'll find that people look a bit skinnier than usual, but the picture will be watchable. However if you try to watch a 2.35 movie that has been stretched to 4:3, and you can't correct the aspect ratio, you're going to be in trouble -- the picture will be ghastly. In short, we stretch a 2.35:1 movies to 16:9, not 4:3, because we can't be sure that the display device will be able to correct the aspect ratio. There might be some DVDs out there which are encoded in this `extreme anamorophic' manner, but I've never come across one. In reality, nearly all DVD movies are in `anamorphic 2.35:1' format. That is, they are stretched vertically to fill an image 406x720. When you rip the movie to a video file, you will have to compress the image vertically more than you compress it horizontally, so that it ends up back at the correct aspect ratio.
      Recently (that is, since the first version of this article) studios have started releasing DVDs with a 16:9 presentation encoded anamorphically as 4:3 (rather than encoding as 16:9 with black bars). While you can't really get away with encoding 2.35:1 as 4:3 (because the vertical stretch is too much for a television set), we can just about get by encoding 16:9 as 4:3. OK, so everybody looks seven feet tall, but it's watchable.
      There are a few DVD movies that are even wider than 2.35:1. This does not present a problem when ripping, because the DVD will be intended to be played on equipment that can handle 2.35:1 anamorphic, so if you proceed on the basis that it is 2.35:1 anamorphic you'll get good results. If you want to be sophisticated you can shave a few megabytes off the rip by cropping off the thin black bars (as described below).
      The other movie presentation format you need to know about is Academy Standard (called `Flat' in the trade), which has an aspect ratio of 1.85:1. This is almost 16:9, but not quite -- it's a little wider. This will probably be put on a DVD as if it were 16:9, so you should be able to proceed as for a true 16:9 programme. However, some studies encode this material anamorphically as 4:3. In practice, if you proceed on the basis that it is a 16:9 presentation, either encoded 16:9 or anamorphically as 4:3, you'll be OK.

To sum up... most PAL DVD movies are presented in one of four ways:

• 4:3 programme content in 4:3 encoding -- when you rip this for the AV500 (or any other 16:9 display) all you'll need to do is to scale the physical frame uniformly to fit the display's height. The AV500 will either black-bar the sides, or zoom to fill the display, depending on how you want to watch
• 16:9 or Academy Standard programme content in 16:9 DVD encoding -- this will need to be cropped at the top and bottom, then scaled uniformly to fit the height of the display (it will automatically fit the width if you fit the height, because the aspect ratio of the AV500 is the same as that of the programme)
• 2.35:1 programme content in 16:9 DVD encoding (therefore anamorphic). This will need to be cropped, then scaled anamorphically to fit the screen and correct the aspect ratio
• 16:9 or Academy Standard program content in 4:3 encoding -- this must be scaled anamorphically to fit the screen and correct the aspect ratio. No cropping is necessary, because the material fits the 4:3 physical frame

A problem that you will encounter when ripping any DVD is that the studios don't normally put any technical information on the packaging. It's very rare (to say the least) to find a DVD sleeve that says ``This movie is presented in an aspect ratio of 2.45:1, encoded as 16:9, with 402 vertical programme scan lines and 87 blank scan lines above and below the programme content. The image is compressed vertically by 30.1% over the horizontal''. If they did, we could write a piece of software that would use that information to rip a DVD with no trial-and-error at all. Alas, what you'll see on the sleeve is something like ``Wide-screen presentation'' or ``Enhanced 16:9'' -- which are meaningless terms. You might need to try a few different ripping protocols until you get the one that produces the best results. After all, the best rip is the one that looks best when you watch it. It is nearly always necessary, in my experience, to rip part of the DVD first without any kind of transformation at all, to get an idea how it is presented and encoded.

An overview of transcode

transcode -a 0 -T NN,-1 -x dvd -i /dev/dvd -w NNN -b N,N,N \
  -V -f N -j NN,NN,NN,NN -B NN,NN,NN -y divx4 -I 1 \
  -o outputfile.avi

• -a 0 Rip audio track 0. Your DVD might have multiple audio tracks (different languages, or commentaries, for example), and there's no obvious way to tell which is which. You'll need to try a few until you get the right one. Note that transcode will automatically extract a stereo soundtrack from a mutli-channel one, so you don't need to do anything special here
• T NN,-1 Rip title number NN. Usually title numbers start at 1, and increase in steps of 1. But don't bet on it. You might need to try a few different title numbers until you get the right one. The `-1' part tells transcode to read all the chapters in the title and put them in the same output file. Movies are usually put into a single title -- even that Behemoth The Two Towers has all three hours in a single title. DVDs of television series usually put each episode in a single title
• -x dvd The input source is a DVD
• -i /dev/dvd The input source is at /dev/dvd
• -w NNN Target bandwidth in kilobits per second. For the internal LCD display, 500 works reasonably well. You can get away with much lower values if what you want to do is squeeze as many movies onto the hard disk as possible and you aren't too fussy about quality
• -b N,N,N LAME MP3 encoder parameters: bitrate, VBR flag, quality, and mode. See the transcode documentation for full details, but for movies I usually specify -b 128,0,0,0. This gives a constant-bitrate audio encode, which the Archos units seem to prefer. Although they have improved in recent versions, there have been problems in the past with variable bitrate audio tracks. Archos units prior to the 604 only support MP3 audio, so you have to use the LAME encoder in transcode
• -V Do colourspace arithmetic in YUV format. This is much faster -- always specify this switch unless it breaks something [Note that this is the default for the latest releases of transcode]
• -f NN Output frame rate. Only specify this if you're actually changing it
• -j NN,NN,NN,NN Clip the source image by NN pixels from the top, left, bottom, and right respectively. Much more on this switch below
• -B NN,NN,NN,NN Indicates how much to reduce the image size. More below.
• -y divx All the Archos units support DIVX, and the more modern ones support Xvid as well. The results are similar, but some people prefer one or the other. Using divx gives better compatibility [Update: probably not true any more, and the divx codec is no longer maintained. Use `-y xvid' instead]
• -I 1 De-interlacing mode. PAL DVDs are usually encoded in such a way as to be played on an interlaced display (the bog-standard television set). LCD displays are not usually interlaced -- the AV500's is not. The picture will look horrible unless you de-interlace. Look in the documentation for the different de-interlacing modes -- I normally use mode 1 for the internal LCD
• -o outputfile.avi The AVI file you want to write

-j tells transcode to crop a certain number of lines and columns from the top, left, bottom, and right of the source frame (the source frame, in this case, is always 720x576 pixels). In practice, you won't normally need to crop the left or right of a DVD, and the amount to crop from the top and bottom, if any, is usually the same. If you calculate that, for example, you need to crop 84 lines from the top and bottom of the source image, you'll say

-j 84,0,84.0

-B blocks_to_remove_vertically,blocks_to_remove_horizontally,block_size

Number of vertical lines to remove is (576-272) = 304
Number of vertical pixel groups is (304/8) = 38

Number of horizonal columns to remove is (720-344) = 376
Number of horizontal pixel groups is (376/8) = 47 

And so to business

4:3 programme material

transcode -a 0 -T NN,-1 -x dvd -i /dev/dvd -w 500 -b 128,0,0 \
   -V -B 38,47,8 -y divx4 -I 1 -o outputfile.avi

16:9 programme material, not anamorphic

transcode -a 0 -T NN,-1 -x dvd -i /dev/dvd -w 500 \
  -b 128,0,0 -V -j 84,0,84,0 -B 17,30,8 \
  -y divx4 -I 1 -o outputfile.avi

2.35:1 programme material, encoded anamorphically in 16:9

transcode -a 0 -T NN,-1 -x dvd -i /dev/dvd -w 500 \
  -b 128,0,0 -V -j 84,0,84,0 -B 25,30,8 \
  -y divx4 -I 1 -o outputfile.avi

16:9 programme material, encoded anamorphically in 4:3

transcode -a 0 -T NN,-1 -x dvd -i /dev/dvd -w 500 \
  -b 128,0,0 -V -B 19,15,16 \
  -y divx4 -I 1 -o outputfile.avi

Two-pass encoding

New Archos models

Closing observations

Make usage decisions carefully Ripping DVDs is not a quick process, even on the fastest computer hardware. It's not quick enough, at any rate, that you'll want to do it twice for the same programme. If you plan on watching your programmes on a larger device than the AV500 screen, it's probably best to take that into account when ripping. After all, the AV500 will play larger frame sizes than its own screen. For best results, consider ripping to a size which is of the same aspect ratio as the LCD screen, e.g., 720x406. This is a relatively fast thing do to with transcode, because you haven't got to scale the image at all -- just crop out the 406 lines from the middle of the physical frame. Similarly, if you always prefer to crop off a part of the image rather than having black bars at the left and right, consider doing the crop in transcode rather than having the AV500 zoom the image to remove the black bars. Why? Well, if you scale up the image on the AV500, it will tend to exacerbate any defects in the encoding -- you'll never get better quality by making pixels stretch further. But if you crop the image in transcode and end up with something that exactly fits the AV500 screen the way you like it, the unit won't have to scale anything. Of course, this is not a sensible thing to do if you plan to watch on other screens -- you're throwing away part of the source material.

File size considerations The settings I describe above will generate AVI files whose size is about 260 Mb for each hour of programme. At a bitrate of 500 kbits/sec (-w 500), and two-pass encoding, you're effectively watching a DVD on a small screen. You'll see some encoding artefacts if you put your face up to the screen, but for normal viewing the quality is as good as it needs to be, in my opinion. However, if minimizing file size is your goal, you'll find that with the divx codec, and two-pass encoding, you can turn the bitrate down to about 200 kbits/sec, and still end up with something watchable. You wouldn't want to put your face up to the screen, and you certainly wouldn't want to play it on a large screen, but for watching on the train, for example, it should be fine. At this bitrate setting, your file sizes will be about 160 Mb/hour.

Shameless plug

By the author of this site. Buy on-line from Amazon USA | UK

Editorial

So you want to be a university lecturer? Read this first! Speak like your boss: new developments in managerese

Computing features

File handling in the Linux kernel: an in-depth look at how Linux handles files, filesystems, and file I/O All sorts of Linux stuff Confused about CLASSPATH? answers are here First steps in EJB using jBoss (recently revised for jBoss 3.2)