I've moved my posts to a new site, a self-hosted WordPress site.
Any further updates will appear there, at http://www.cneufeld.ca/genie/blog/author/christopher/
Thursday, September 5, 2013
Saturday, February 4, 2012
Hamlet, insanity, and English essays
This is going to be a bit off-topic, at least in the context of my previous posts.
I was watching a production of Ambroise Thomas' opera, Hamlet, recently. It called to mind the first essay I was assigned to write after starting CEGEP. I was taking a class on Shakespearean tragedies, and we were asked to take and defend a position on the question, "was Hamlet insane?"
I recall being a bit perplexed by the question itself, but I did write something. I don't recall now what position I took, that isn't really the point of this posting. However, when I ponder the question itself, now, many years later, I really don't think I'd be able to write on this topic without a lot of research. If I were posed this question now, it would be a much more involved assignment, and would take a lot longer to answer. I might, in fact, have to answer with the substance of this posting, rather than taking a position on the question of Hamlet's sanity.
Before continuing, I'll say that phrasing a question so broadly is a bit awkward. What, precisely, does one mean by "insane"? Are we talking about a specific psychiatric diagnosis, or something less well-defined? In what follows I will assume that we have settled on a particular meaning for the word as used in the question posed by my professor.
So, why is this question difficult for me to answer now, when I was able, as a teenager, to produce several pages of arguments? No doubt my outlook on the world has changed, it's been a long time, and people change. I'll describe the way I look at this problem now, as a no-longer-teenager.
First of all, I do enjoy works of fiction. Books, movies, television shows, they keep my attention, and I focus on them while watching/reading. However, I do always know that such things are fiction. One or more writers sat in a room and created the story out of whole cloth. I am more likely to be affected by the sight of somebody's cut finger in the real world than a traumatic injury in a fictional television show, because I know that the latter is not a real event, an actor is portraying the injury with the help of a special effects team, and at the end of the day that actor will return home having suffered no harm. In that respect, I somewhat sympathize with the position of the character Chandler on the show "Friends", who said, somewhat indignantly, that Bambi's mother didn't die, they simply stopped drawing her.
So, with that introduction, we come to the issue that I think was perplexing my younger self, but which I couldn't really pin down in words at the time. The character of Hamlet is a fictional creation, whose actions and behaviours were imposed by William Shakespeare. The actions of the characters in the play are not those of sentient creatures, but are driven by the dictates of the plot and the historical setting that Shakespeare is trying to present in the play.
Every year or two there will be a newspaper story about some group of researchers who declare that a certain historical figure, dead a thousand years or more, suffered from this or that undiagnosed malady, and they offer as evidence some small set of collected descriptions or writings that they assert proves their claim. While one may find issue with specific instances of these announcements, the researchers are writing about real people, who interacted with other people in the world, and left traces. There are objective facts about these people, from which one might be able to draw conclusions.
Fictional characters, such as those in the play "Hamlet", don't have those interactions with reality. When they are not on stage, they don't exist, except as required by the logical continuity of the plot in which they find themselves. If nothing in the story hinges on what Polonius had for breakfast a week before the opening scene, then one cannot reasonably expect to "discover" what he had for breakfast, because the character is in a sort of limbo, existing only as dictated by the logic of the story. We know that, years earlier, he had a daughter, because she exists in the play, but we don't know about things that are not explicitly derived from the text of the play.
So, we come to the question of Hamlet's sanity. There are three possible positions one can take. He was sane, he was not sane, or his sanity is unknowable. The answer lies, first and last, in the mind of William Shakespeare. When he was writing the story, he knew the answer to that question. He could then choose to write the story to communicate the idea that Hamlet was sane or insane, or he could choose to make the story deliberately ambiguous on that point, allowing the audience to ponder the question but not supplying enough information to allow a definite determination. Think of the end of the Arnold Schwarzenegger version of "Total Recall". There was chatter on the SF Usenet groups discussing the question of whether or not the entire story was an implanted memory, where the character in the story never went to Mars, and in fact never got out of the fancy chair. My position on this question is that the filmmakers deliberately left it an open question. There's no way to determine which is the "actual" answer, because there is nothing outside of the screenplay, and the writers want the answer to be "I don't know".
To my thinking, then, the question of "was Hamlet insane?" comes down to asking to know the mind of the author. We have to read the text, and take our cues not from the actions of the characters but in the way the author chose to depict them. Don't think "why did Ophelia say that?", she didn't. Think, instead, "why did Shakespeare choose to have Ophelia say that?" The play underwent several revisions, we can assume that the rewrites would at least have allowed Shakespeare to communicate the most important points of the story to his satisfaction, and the question of the sanity of Hamlet certainly would be one of those important points, as it is raised several times by characters within the play. If characters are wandering around asking each other if Hamlet is insane, one hopes that the playwright pondered the issue himself, and that the text of the play communicated, to his satisfaction, the answer that he wanted to convey to the audience.
Fine, so how does one go about knowing the mind of the author, as he is currently unavailable for interviews? This is where this stops being a CEGEP homework assignment and becomes something more like an English major's university project.
We have a playwright, and we have his contemporary audience. Shakespeare was not writing for the modern audience, of course, so the way he wrote was to communicate his thoughts to those living in the same time. Shakespeare's depiction has to be examined in the culture of the time. To begin, the play is based on a story, already hundreds of years old by Shakespeare's time, of "Amleth". How much did the Elizabethan theatre-going audience know about that story? Was the title character in there famously known to be insane? If so, Shakespeare need do little more than demonstrate questionable behaviour by Hamlet, and the audience would fill in that yes, the character of Hamlet was insane. But if that was not the popular understanding of the time, then it does little to illuminate the question. There may be other cultural cues that I haven't thought of, but I'm a physicist and not an English major, and I haven't done this research, I merely point out where I think the research has to go in order properly to address the question.
After looking at the cultural context, we go on to the play itself. The characters were written around the end of the 16th century, but one imagines they might have behaved anachronistically, saying things or behaving in ways that were not appropriate for Denmark of hundreds of years earlier. The story was written for the Elizabethan audience, the members of which had their own cultural cues defining eccentricity and insanity as they would have experienced it and talked of it in England of the time. Certain phrases, gestures, or reactions of the character might have communicated sanity or insanity quite unambiguously to the audience of the time, but a modern reader, not steeped in the culture, might miss those cues entirely, introducing ambiguity where none was intended. To address this question, one would have to perform a quite lengthy examination of the popular perception of insanity at the time, both in reality, the pub gossip of the time, and in popular fiction of the era, to look for any literary shorthand that might have commonly been used to convey the concept of insanity. After this research was performed (I hesitate to say "completed" because this sounds like a project that one could never truly decide was finished), the play would be re-examined in the context of this new understanding, to see if there were any unambiguous markers in the play.
Having examined the play, our journey into the mind of William Shakespeare would now go to his other works. We would examine characters throughout his plays, ones in which insanity is unambiguously manifest, starting with Ophelia but not restricting ourselves just to the one play. We would learn how Shakespeare developed his insane characters in his works, and how he depicted them as progressing. We could then look at the character of Hamlet once again, and say that Shakespeare's writing style strongly suggests that he was sane, or insane, or maybe progressing toward insanity, or, once again, we might decide that it was ambiguous, that Shakespeare has not given us a clear answer to the question.
There's at least one more avenue of research that I believe ought to be followed. One would have to research the reviews and discussions of the play Hamlet as it was being performed, while it was still susceptible to being edited by Shakespeare. What did the reviewers say about the character of Hamlet? What did people say about it at the pub? Were there letters to the local newspapers that touched on the topic of Hamlet's sanity? Basically, what can we learn about what the contemporary audiences got out of the story? If it was ambiguous to them, at a time when William Shakespeare would have known their reactions and been able to edit the play to better communicate his position, then I think we can say with some confidence that he deliberately wanted the character's mental health to be ambiguous, and anything we might say, as modern teenagers in a CEGEP English class, has very little weight compared to that.
So, there we are. If a professor asked me that question today, I don't think I could submit any response I'd be happy to hand in without doing a lot of tedious research with dusty books in old libraries. It's not something I would throw together on a typewriter in the week before it was due.
I was watching a production of Ambroise Thomas' opera, Hamlet, recently. It called to mind the first essay I was assigned to write after starting CEGEP. I was taking a class on Shakespearean tragedies, and we were asked to take and defend a position on the question, "was Hamlet insane?"
I recall being a bit perplexed by the question itself, but I did write something. I don't recall now what position I took, that isn't really the point of this posting. However, when I ponder the question itself, now, many years later, I really don't think I'd be able to write on this topic without a lot of research. If I were posed this question now, it would be a much more involved assignment, and would take a lot longer to answer. I might, in fact, have to answer with the substance of this posting, rather than taking a position on the question of Hamlet's sanity.
Before continuing, I'll say that phrasing a question so broadly is a bit awkward. What, precisely, does one mean by "insane"? Are we talking about a specific psychiatric diagnosis, or something less well-defined? In what follows I will assume that we have settled on a particular meaning for the word as used in the question posed by my professor.
So, why is this question difficult for me to answer now, when I was able, as a teenager, to produce several pages of arguments? No doubt my outlook on the world has changed, it's been a long time, and people change. I'll describe the way I look at this problem now, as a no-longer-teenager.
First of all, I do enjoy works of fiction. Books, movies, television shows, they keep my attention, and I focus on them while watching/reading. However, I do always know that such things are fiction. One or more writers sat in a room and created the story out of whole cloth. I am more likely to be affected by the sight of somebody's cut finger in the real world than a traumatic injury in a fictional television show, because I know that the latter is not a real event, an actor is portraying the injury with the help of a special effects team, and at the end of the day that actor will return home having suffered no harm. In that respect, I somewhat sympathize with the position of the character Chandler on the show "Friends", who said, somewhat indignantly, that Bambi's mother didn't die, they simply stopped drawing her.
So, with that introduction, we come to the issue that I think was perplexing my younger self, but which I couldn't really pin down in words at the time. The character of Hamlet is a fictional creation, whose actions and behaviours were imposed by William Shakespeare. The actions of the characters in the play are not those of sentient creatures, but are driven by the dictates of the plot and the historical setting that Shakespeare is trying to present in the play.
Every year or two there will be a newspaper story about some group of researchers who declare that a certain historical figure, dead a thousand years or more, suffered from this or that undiagnosed malady, and they offer as evidence some small set of collected descriptions or writings that they assert proves their claim. While one may find issue with specific instances of these announcements, the researchers are writing about real people, who interacted with other people in the world, and left traces. There are objective facts about these people, from which one might be able to draw conclusions.
Fictional characters, such as those in the play "Hamlet", don't have those interactions with reality. When they are not on stage, they don't exist, except as required by the logical continuity of the plot in which they find themselves. If nothing in the story hinges on what Polonius had for breakfast a week before the opening scene, then one cannot reasonably expect to "discover" what he had for breakfast, because the character is in a sort of limbo, existing only as dictated by the logic of the story. We know that, years earlier, he had a daughter, because she exists in the play, but we don't know about things that are not explicitly derived from the text of the play.
So, we come to the question of Hamlet's sanity. There are three possible positions one can take. He was sane, he was not sane, or his sanity is unknowable. The answer lies, first and last, in the mind of William Shakespeare. When he was writing the story, he knew the answer to that question. He could then choose to write the story to communicate the idea that Hamlet was sane or insane, or he could choose to make the story deliberately ambiguous on that point, allowing the audience to ponder the question but not supplying enough information to allow a definite determination. Think of the end of the Arnold Schwarzenegger version of "Total Recall". There was chatter on the SF Usenet groups discussing the question of whether or not the entire story was an implanted memory, where the character in the story never went to Mars, and in fact never got out of the fancy chair. My position on this question is that the filmmakers deliberately left it an open question. There's no way to determine which is the "actual" answer, because there is nothing outside of the screenplay, and the writers want the answer to be "I don't know".
To my thinking, then, the question of "was Hamlet insane?" comes down to asking to know the mind of the author. We have to read the text, and take our cues not from the actions of the characters but in the way the author chose to depict them. Don't think "why did Ophelia say that?", she didn't. Think, instead, "why did Shakespeare choose to have Ophelia say that?" The play underwent several revisions, we can assume that the rewrites would at least have allowed Shakespeare to communicate the most important points of the story to his satisfaction, and the question of the sanity of Hamlet certainly would be one of those important points, as it is raised several times by characters within the play. If characters are wandering around asking each other if Hamlet is insane, one hopes that the playwright pondered the issue himself, and that the text of the play communicated, to his satisfaction, the answer that he wanted to convey to the audience.
Fine, so how does one go about knowing the mind of the author, as he is currently unavailable for interviews? This is where this stops being a CEGEP homework assignment and becomes something more like an English major's university project.
We have a playwright, and we have his contemporary audience. Shakespeare was not writing for the modern audience, of course, so the way he wrote was to communicate his thoughts to those living in the same time. Shakespeare's depiction has to be examined in the culture of the time. To begin, the play is based on a story, already hundreds of years old by Shakespeare's time, of "Amleth". How much did the Elizabethan theatre-going audience know about that story? Was the title character in there famously known to be insane? If so, Shakespeare need do little more than demonstrate questionable behaviour by Hamlet, and the audience would fill in that yes, the character of Hamlet was insane. But if that was not the popular understanding of the time, then it does little to illuminate the question. There may be other cultural cues that I haven't thought of, but I'm a physicist and not an English major, and I haven't done this research, I merely point out where I think the research has to go in order properly to address the question.
After looking at the cultural context, we go on to the play itself. The characters were written around the end of the 16th century, but one imagines they might have behaved anachronistically, saying things or behaving in ways that were not appropriate for Denmark of hundreds of years earlier. The story was written for the Elizabethan audience, the members of which had their own cultural cues defining eccentricity and insanity as they would have experienced it and talked of it in England of the time. Certain phrases, gestures, or reactions of the character might have communicated sanity or insanity quite unambiguously to the audience of the time, but a modern reader, not steeped in the culture, might miss those cues entirely, introducing ambiguity where none was intended. To address this question, one would have to perform a quite lengthy examination of the popular perception of insanity at the time, both in reality, the pub gossip of the time, and in popular fiction of the era, to look for any literary shorthand that might have commonly been used to convey the concept of insanity. After this research was performed (I hesitate to say "completed" because this sounds like a project that one could never truly decide was finished), the play would be re-examined in the context of this new understanding, to see if there were any unambiguous markers in the play.
Having examined the play, our journey into the mind of William Shakespeare would now go to his other works. We would examine characters throughout his plays, ones in which insanity is unambiguously manifest, starting with Ophelia but not restricting ourselves just to the one play. We would learn how Shakespeare developed his insane characters in his works, and how he depicted them as progressing. We could then look at the character of Hamlet once again, and say that Shakespeare's writing style strongly suggests that he was sane, or insane, or maybe progressing toward insanity, or, once again, we might decide that it was ambiguous, that Shakespeare has not given us a clear answer to the question.
There's at least one more avenue of research that I believe ought to be followed. One would have to research the reviews and discussions of the play Hamlet as it was being performed, while it was still susceptible to being edited by Shakespeare. What did the reviewers say about the character of Hamlet? What did people say about it at the pub? Were there letters to the local newspapers that touched on the topic of Hamlet's sanity? Basically, what can we learn about what the contemporary audiences got out of the story? If it was ambiguous to them, at a time when William Shakespeare would have known their reactions and been able to edit the play to better communicate his position, then I think we can say with some confidence that he deliberately wanted the character's mental health to be ambiguous, and anything we might say, as modern teenagers in a CEGEP English class, has very little weight compared to that.
So, there we are. If a professor asked me that question today, I don't think I could submit any response I'd be happy to hand in without doing a lot of tedious research with dusty books in old libraries. It's not something I would throw together on a typewriter in the week before it was due.
Friday, October 28, 2011
Converting DVDs for viewing on a tablet, while inlining captions
Previously, I described how to convert HDTV videos for my EEE Pad Transformer. Now, I'll go over something a bit more difficult.
My wife and I have some DVDs of Bollywood films that we enjoy watching. Aaja Nachle, Om Shanti Om, 3 Idiots, Billu, among others. These films are mostly in Hindi, but there are English subtitles available. As we don't understand Hindi, we watch the movies with the subtitles. The Android media viewer that comes with the tablet doesn't have a way to select subtitles from an alternate video stream.
Now, I wanted to make files of these movies that I could watch on the Android tablet. As noted in the previous article, the resulting files have to be H.264 Baseline profile, and under 2GB in size.
Here's how I did this. Note that this procedure required no less than 70 GB of free disk space to hold a large intermediate file, as I wanted to avoid artefacts introduced by running through multiple codecs, so I used a lossless intermediate state.
First of all, I used the MythTV option to rip a perfect copy of the DVD. That gave me a file, say 3IDIOTS.vob.
Next, I used mencoder to inline the captions directly into the video stream:
The output file, 3idiots, was, as noted, huge. It consisted of a lossless jpeg video stream, with the subtitle 0 track overlaid on the video stream itself.
Next, the file had to be converted to H.264 Baseline. In this case, I decided, rather than setting a qmax, that I would set a bitrate. That way I could be certain ahead of time what the final size of the file would be, though at the cost of increased trancoding time. To get a fixed bitrate, it is necessary to run ffmpeg in two passes, once to collect statistics, and the second time to generate the file itself. Here's how this is run:
The "junkfile.mp4" file can be deleted. The H.264 file, 3idiots.mp4, came in at 1.8 GB, and was of quite acceptable quality to view on the tablet.
My wife and I have some DVDs of Bollywood films that we enjoy watching. Aaja Nachle, Om Shanti Om, 3 Idiots, Billu, among others. These films are mostly in Hindi, but there are English subtitles available. As we don't understand Hindi, we watch the movies with the subtitles. The Android media viewer that comes with the tablet doesn't have a way to select subtitles from an alternate video stream.
Now, I wanted to make files of these movies that I could watch on the Android tablet. As noted in the previous article, the resulting files have to be H.264 Baseline profile, and under 2GB in size.
Here's how I did this. Note that this procedure required no less than 70 GB of free disk space to hold a large intermediate file, as I wanted to avoid artefacts introduced by running through multiple codecs, so I used a lossless intermediate state.
First of all, I used the MythTV option to rip a perfect copy of the DVD. That gave me a file, say 3IDIOTS.vob.
Next, I used mencoder to inline the captions directly into the video stream:
mencoder -ovc lavc -lavcopts vcodec=ljpeg:aspect=16/9 \
-vobsubid 0 -oac lavc -lavcopts acodec=flac \
-o 3idiots 3IDIOTS.vob
The output file, 3idiots, was, as noted, huge. It consisted of a lossless jpeg video stream, with the subtitle 0 track overlaid on the video stream itself.
Next, the file had to be converted to H.264 Baseline. In this case, I decided, rather than setting a qmax, that I would set a bitrate. That way I could be certain ahead of time what the final size of the file would be, though at the cost of increased trancoding time. To get a fixed bitrate, it is necessary to run ffmpeg in two passes, once to collect statistics, and the second time to generate the file itself. Here's how this is run:
ffmpeg -pass 1 -i 3idiots -vcodec libx264 -vpre fast \
-vpre baseline -b 1400 -acodec libfaac -ab 64k \
-ac 2 -ar 44100 -threads 3 \
-deinterlace -y junkfile.mp4
ffmpeg -pass 2 -i 3idiots -vcodec libx264 -vpre fast \
-vpre baseline -b 1400k -acodec libfaac -ab 64k \
-ac 2 -ar 44100 -threads 3 \
-deinterlace 3idiots.mp4 The "junkfile.mp4" file can be deleted. The H.264 file, 3idiots.mp4, came in at 1.8 GB, and was of quite acceptable quality to view on the tablet.
Converting HDTV videos for viewing on a tablet
I have an Android-based tablet computer, the EEE Pad Transformer. My MythTV computer can record digital over-the-air broadcasts in high definition now that I have put an HDHomerun on my network. So, it would be nice to be able to transfer some HDTV programs to the Android computer to watch them there while traveling. The HDTV shows are 1080i, encoded as mpeg2 video, at a bitrate of close to 16000 kbits/sec.
So, what are our constraints? The Android computer is not powerful enough to play videos without hardware assist, and that hardware assist is only available when viewing H.264 videos encoded with the baseline profile. It doesn't work on main profile H.264 videos. Also, the Micro-SD card that I plug into the tablet must be formatted as VFAT, it isn't recognized when I reformat it to any more modern Linux filesystems, so our files are going to have to be under 2GB in size. Also, the Android screen is only 1280x800, so there's no point copying a 2560x1080 file there, the machine will have to reduce the resolution, we might as well do it before we copy it to the card.
So, a 1 hour show, recorded on the MythTV box, is about 8 GB and in the wrong format. We convert it in two steps. First, cut out any commercials and transcode it at high quality. For network broadcast television that chops off about 25% of the file size, and you probably didn't want to watch the commercials while sitting on the train/airplane anyway.
Next, it has to be transcoded to H.264 Basline. This can be done with ffmpeg:
This takes the HDTV .mpg file from mythtv, "PROGRAM.mpg", and converts it. We use the libx264 video codec, fast settings, baseline profile, formatted for a high definition 720 line screen. "qmax" sets a limit on quality loss, I usually use a value between 25 and 30. We use the FAAC audio codec at 128kbits/sec, deinterlace the result, and write it to "PROGRAM.mp4".
The resulting file, about 45 minutes of air time, is about 600 MB in size.
So, what are our constraints? The Android computer is not powerful enough to play videos without hardware assist, and that hardware assist is only available when viewing H.264 videos encoded with the baseline profile. It doesn't work on main profile H.264 videos. Also, the Micro-SD card that I plug into the tablet must be formatted as VFAT, it isn't recognized when I reformat it to any more modern Linux filesystems, so our files are going to have to be under 2GB in size. Also, the Android screen is only 1280x800, so there's no point copying a 2560x1080 file there, the machine will have to reduce the resolution, we might as well do it before we copy it to the card.
So, a 1 hour show, recorded on the MythTV box, is about 8 GB and in the wrong format. We convert it in two steps. First, cut out any commercials and transcode it at high quality. For network broadcast television that chops off about 25% of the file size, and you probably didn't want to watch the commercials while sitting on the train/airplane anyway.
Next, it has to be transcoded to H.264 Basline. This can be done with ffmpeg:
ffmpeg -i PROGRAM.mpg -vcodec libx264 -vpre fast \
-vpre baseline -s hd720 -qmax 30 -acodec libfaac \
-ab 128k -ac 2 -threads 4 -ar 44100 -deinterlace \
PROGRAM.mp4
This takes the HDTV .mpg file from mythtv, "PROGRAM.mpg", and converts it. We use the libx264 video codec, fast settings, baseline profile, formatted for a high definition 720 line screen. "qmax" sets a limit on quality loss, I usually use a value between 25 and 30. We use the FAAC audio codec at 128kbits/sec, deinterlace the result, and write it to "PROGRAM.mp4".
The resulting file, about 45 minutes of air time, is about 600 MB in size.
Friday, September 18, 2009
Getting GSM + Internet while visiting Taiwan
I have a Palm Centro phone, not SIM-locked, so I can obtain local phone numbers when I travel. Of course, it doesn't make sense to try to sign up for a contract if you're only in the country for two weeks, so a pre-paid phone card is really the best choice.
I recently made one of my frequent trips to Taiwan to visit family there. While there, I wanted a local phone number so people could reach me, and I also wanted to be able to use my phone's web browser and Google Earth, meaning I needed Internet access.
This may not be the only way to do that, and may not be the cheapest, it just happens to be the way I did it. I went to a few cell-phone company shop fronts. FarEasTone had not yet opened for the day, so I went to Aurora. The Aurora staff told me they could give me a pre-paid account, but not with Internet access. So the third stop in my quest was Chunghwa Telecom (中華電信). It's one of the larger companies. Here, I was able to get everything that I was looking for.
You will need:
I signed up for a pre-paid G3 card. While my phone is GSM, the G3 cards are backward compatible to the older format. I paid NT$300 (about C$10) up front, and got the full amount in credit on my account, there are no setting-up fees, all the money goes toward pre-paid airtime minutes. The transaction took only a few minutes, and I was able to make test calls from my phone before getting up from my chair. The SIM card comes with a 4-digit unlock code, you have to enter this every time you turn on the cell phone. You can set up voice mail, but I didn't bother.
I was actually a bit surprised that I could set up a new phone number for only C$10. I would have expected that the administration costs would make it impractical to offer such a low-price entry. In the end, I used only about C$23 in air time charges in the almost three weeks I was there.
Calls are billed by the second. After each outgoing call, a text message is sent to your phone telling you the number you called, the time spent on the call, the amount charged against your account for the call, and the expiry date of your account. The account expiration timer is 180 days, reset every time you add funds to your account. To add funds to your account, you simply walk into any 7-11 and tell them you want to buy a recharge for your Chunghwa Telecom (zhong1 hua2 dian4 xin4) phone. Recharges cost NT$300, and again, all of the money you pay goes into pre-paid airtime minutes, without anything held back for "access fees" etc. To use the recharge, follow the directions printed on the card. Basically, you call 928, go through a couple of menu options, then type in the PIN revealed by scratching the back of the recharge.
The employee who set up my account warned me that Internet use was expensive, but I didn't find it so. Of course, I wasn't watching television shows on my cell phone, just visiting a few websites in the morning to read the news from home. The price quoted for GPRS is NT$0.005 per "packet". A GPRS packet is about 1 kB, so that would make the price about NT$5 per megabyte.
You should probably ask that your service be set for English. If you don't do it at the counter, you can change your language preference at any time by calling the number 928. While I can read Chinese, my phone, bought in North America, doesn't have Chinese fonts, so the text messages sent after every outgoing call are unreadable unless you've set your language preference to English.
EDIT: returning to Taiwan this year, the same SIM card in the same cell phone couldn't log onto the Internet. After a bit of discussion between their customer service staff and their IT support, it was determined that we had to edit the network settings on my cell phone, specifically assigning an APN string of "emome".
I recently made one of my frequent trips to Taiwan to visit family there. While there, I wanted a local phone number so people could reach me, and I also wanted to be able to use my phone's web browser and Google Earth, meaning I needed Internet access.
This may not be the only way to do that, and may not be the cheapest, it just happens to be the way I did it. I went to a few cell-phone company shop fronts. FarEasTone had not yet opened for the day, so I went to Aurora. The Aurora staff told me they could give me a pre-paid account, but not with Internet access. So the third stop in my quest was Chunghwa Telecom (中華電信). It's one of the larger companies. Here, I was able to get everything that I was looking for.
You will need:
- An unlocked GSM (or G3) cell phone.
- Two pieces of photo ID. I used my Canadian passport and my Ontario driver's license.
- A local address in Taiwan. A hotel address is fine.
- The ability to communicate with the staff.
I signed up for a pre-paid G3 card. While my phone is GSM, the G3 cards are backward compatible to the older format. I paid NT$300 (about C$10) up front, and got the full amount in credit on my account, there are no setting-up fees, all the money goes toward pre-paid airtime minutes. The transaction took only a few minutes, and I was able to make test calls from my phone before getting up from my chair. The SIM card comes with a 4-digit unlock code, you have to enter this every time you turn on the cell phone. You can set up voice mail, but I didn't bother.
I was actually a bit surprised that I could set up a new phone number for only C$10. I would have expected that the administration costs would make it impractical to offer such a low-price entry. In the end, I used only about C$23 in air time charges in the almost three weeks I was there.
Calls are billed by the second. After each outgoing call, a text message is sent to your phone telling you the number you called, the time spent on the call, the amount charged against your account for the call, and the expiry date of your account. The account expiration timer is 180 days, reset every time you add funds to your account. To add funds to your account, you simply walk into any 7-11 and tell them you want to buy a recharge for your Chunghwa Telecom (zhong1 hua2 dian4 xin4) phone. Recharges cost NT$300, and again, all of the money you pay goes into pre-paid airtime minutes, without anything held back for "access fees" etc. To use the recharge, follow the directions printed on the card. Basically, you call 928, go through a couple of menu options, then type in the PIN revealed by scratching the back of the recharge.
The employee who set up my account warned me that Internet use was expensive, but I didn't find it so. Of course, I wasn't watching television shows on my cell phone, just visiting a few websites in the morning to read the news from home. The price quoted for GPRS is NT$0.005 per "packet". A GPRS packet is about 1 kB, so that would make the price about NT$5 per megabyte.
You should probably ask that your service be set for English. If you don't do it at the counter, you can change your language preference at any time by calling the number 928. While I can read Chinese, my phone, bought in North America, doesn't have Chinese fonts, so the text messages sent after every outgoing call are unreadable unless you've set your language preference to English.
EDIT: returning to Taiwan this year, the same SIM card in the same cell phone couldn't log onto the Internet. After a bit of discussion between their customer service staff and their IT support, it was determined that we had to edit the network settings on my cell phone, specifically assigning an APN string of "emome".
Saturday, July 4, 2009
Musings on the "Impact" miniseries
I watched that 4-hour television miniseries, "Impact", yesterday. I'm now going to set down some of my observations, from a physics and astronomy viewpoint. Even a broadcast like that can teach you something, if it is used as a starting point to explain the things the writers got wrong. And there is a lot of teaching available here, even in the first ten minutes.
OK, we start off with people observing this "biggest meteor shower in 50000 years". It is seen starting up, so we know the observations were simultaneous. There were groups in New Mexico, the East coast of the US, and in Germany, all watching the meteor shower begin. Sunset times between those locations are as much as 9 hours apart, and in the summer time (when this movie appears to have been set) there aren't 9 hours of full darkness. This is a common mistake, movies and television shows will often show two participants in a phone call sitting half a world apart, both in full daylight.
Then, that meteor shower was a disappointment. There are recent records of much more intense meteor showers. The Leonid showers of 1833 and 1966 were, from their descriptions, much more spectacular than the shower shown in this movie.
Two astronomers are observing the meteor swarm through telescopes, before it reaches the Earth. We see a field of rocks large enough to be seen through telescopes, and so densely packed as to block sight lines so that astronomers couldn't see another object at the back of the swarm. This isn't a meteor swarm, a meteor swarm is rocks smaller than pebbles, separated from one another by kilometres of empty space. This is an avalanche in space.
Next, we find out that an object, visible while it's still moving in space, was traveling with the cloud of meteors and is going to strike the moon. Seen from the ground, this object had a visibly different track across the sky, which doesn't make sense if the objects were all traveling together. But if it were traveling in the same direction as the visible meteors, it wouldn't stand out and so would be less desirable from a dramatic standpoint, so we'll let that one pass.
So, this mystery object. Let's forget about the "brown dwarf" babble, and just describe it as a super-dense, magnetized object with, as they say in the movie, a mass twice that of the Earth's. It hit the moon, and bad things happened.
Now, the science used to explain the effects on Earth is all nonsense, of course. The "levitating frog" experiment did not produce anti-gravity. It exerted a force on a frog. A string tied to the frog's leg would also exert a force. This was just like that, but it used a magnetic field to apply the force. Gravity was still affecting the frog, but the frog was being supported against the force of gravity by a force of magnetic origin, one related to the gradient of the magnetic field (how much the field changes over a short distance). So, starting from a misunderstanding of an old news release, the writers created weird fantasy effects where objects that are not too small and not too large levitate in spooky ways in random places on the Earth, then crash to the ground. Whatever, we're not going to talk about that anymore.
OK, back to "small, very heavy object hits the moon". Our astronomers mention that the moon has 1/6 the mass of the Earth. No, it doesn't. It has about 1/6 the surface gravity, but only about 1/80 of the mass of the Earth. This is a common mistake, believing that gravity is a function solely of the mass of the object, and ignoring the different sizes. To take a dramatic example, Saturn has almost 100 times the mass of the Earth, but the force of gravity exerted at the cloud tops is not much higher than the force of gravity at the surface of the Earth, because the cloud tops are over 9 times as far from the centre of Saturn as the surface of the Earth is from its centre.
The moon gets hit by something very small that weighs two Earth masses, and is traveling very fast. And they stick together. 160 times the mass of the moon smacks into it with a speed of, let's say, several kilometres per second. This object wouldn't stop. It would barely even notice the moon. If the entire moon got in its way, it would sweep it up and continue on its path practically unaware that it was now carrying a moon with it. Since the thing is small, only a bit of the moon gets in its way. It's a very small and extremely fast bullet striking a very large soap bubble. You don't expect the soap bubble to be carried away by the bullet, you expect to find a punctured bubble. You certainly don't expect the bullet to stop dead in the bubble.
160 times the mass. Imagine you're driving down a highway, and a raccoon is crossing the road. Just as your car is about to hit it, the raccoon jumps straight up and hits the front of your car. Your car stops dead as if it had struck a concrete wall, and the mid-air raccoon is barely pushed at all. Even cartoons don't try to get you to believe that.
That much mass, stopping all at once within the moon. Just the kinetic energy released is about the same as the total output of the sun over the space of 48 hours. Not the light hitting the Earth, the light leaving the entire solar sphere. The moon would vanish in a puff of gas. The Earth would vanish in a larger puff of gas.
OK, so suddenly the moon weighs twice what the Earth does. This would have some fairly obvious effects. For one thing, the tides on the surface of the Earth would go from a few metres to a few hundred metres in amplitude. That would have a serious effect on the coastal regions (and with those tides, Missouri is a coastal region).
Increase the mass of the Earth-moon system, and the rotational period will decrease. A month would go from about 30 days to about 10 days. But in the movie, the moon was making complete orbits around the Earth on plot-driven timescales. Sometimes the orbital period was a few days, and toward the end of the movie the orbital period seems to have become about 90 days, because they had deduced that the moon would hit the Earth on this orbit, but they still had 40 days left to try to find a solution. And these weird, sudden "orbital shifts" don't make sense. Yes, an uneven mass distribution can result in smooth and gradual changes to orbits, but the moon didn't have an uneven mass distribution. It was a big mass travelling in orbit, with a light, insignificant, moon stuck to it like a bug on a windshield. The pre-impact mass of the moon isn't even an important perturbation on the mass distribution.
Good news, everybody! We just happen to have a lunar expedition fueled up and ready to go, prepared before the impact. Our heroes can fly to the moon and use some special technology to push the big new mass out of the moon. Well, the mass weighs 160 times what the moon does, so Newton's laws tell us that you're not going to push the mass out of the moon, you're going to push the moon away. The mass won't be appreciably disturbed. The plan is to push the mass out of the moon so it flies toward the sun, but really all you'd do is send the moon away at high speed while the big dense mass stays firmly in its orbit around the Earth.
Now, about this lunar mission. The good news is that you don't need as much fuel to cross over, because of the changes to the shape of the gravitational potential fields in the vicinity. The bad news is that the moon's surface gravity is at least 25 times that of the Earth. That assumes that the colliding mass is at the centre of the moon. In the movie, it's actually only partway down, and our heroes have to land near it, so they'll feel a gravitational force much higher than that. OK, they've been working out, they can walk and work in 25 gravities of force. But their lander was designed to land on rockets in 1/6 normal gravity. It would be like designing a parachute to land you safely, and then you decide that you'll change the parameters, the parachute will be used to land a bit more weight. You plus 159 of your friends, all hanging on the one parachute. You might reasonably conclude that the parachute was not designed for that kind of treatment. A similar argument can be made for the lunar lander and that little rocket jumper vehicle, whose engines certainly cannot supply the thrust to land under the new conditions. The lander would probably crumple under its own weight just trying to sit still on the moon, and taking off from the surface would be similarly difficult because of the changed conditions.
OK. Gravitation, tides, astronomy, orbital motion. If you learned something new from this movie, it's almost certainly wrong.
OK, we start off with people observing this "biggest meteor shower in 50000 years". It is seen starting up, so we know the observations were simultaneous. There were groups in New Mexico, the East coast of the US, and in Germany, all watching the meteor shower begin. Sunset times between those locations are as much as 9 hours apart, and in the summer time (when this movie appears to have been set) there aren't 9 hours of full darkness. This is a common mistake, movies and television shows will often show two participants in a phone call sitting half a world apart, both in full daylight.
Then, that meteor shower was a disappointment. There are recent records of much more intense meteor showers. The Leonid showers of 1833 and 1966 were, from their descriptions, much more spectacular than the shower shown in this movie.
Two astronomers are observing the meteor swarm through telescopes, before it reaches the Earth. We see a field of rocks large enough to be seen through telescopes, and so densely packed as to block sight lines so that astronomers couldn't see another object at the back of the swarm. This isn't a meteor swarm, a meteor swarm is rocks smaller than pebbles, separated from one another by kilometres of empty space. This is an avalanche in space.
Next, we find out that an object, visible while it's still moving in space, was traveling with the cloud of meteors and is going to strike the moon. Seen from the ground, this object had a visibly different track across the sky, which doesn't make sense if the objects were all traveling together. But if it were traveling in the same direction as the visible meteors, it wouldn't stand out and so would be less desirable from a dramatic standpoint, so we'll let that one pass.
So, this mystery object. Let's forget about the "brown dwarf" babble, and just describe it as a super-dense, magnetized object with, as they say in the movie, a mass twice that of the Earth's. It hit the moon, and bad things happened.
Now, the science used to explain the effects on Earth is all nonsense, of course. The "levitating frog" experiment did not produce anti-gravity. It exerted a force on a frog. A string tied to the frog's leg would also exert a force. This was just like that, but it used a magnetic field to apply the force. Gravity was still affecting the frog, but the frog was being supported against the force of gravity by a force of magnetic origin, one related to the gradient of the magnetic field (how much the field changes over a short distance). So, starting from a misunderstanding of an old news release, the writers created weird fantasy effects where objects that are not too small and not too large levitate in spooky ways in random places on the Earth, then crash to the ground. Whatever, we're not going to talk about that anymore.
OK, back to "small, very heavy object hits the moon". Our astronomers mention that the moon has 1/6 the mass of the Earth. No, it doesn't. It has about 1/6 the surface gravity, but only about 1/80 of the mass of the Earth. This is a common mistake, believing that gravity is a function solely of the mass of the object, and ignoring the different sizes. To take a dramatic example, Saturn has almost 100 times the mass of the Earth, but the force of gravity exerted at the cloud tops is not much higher than the force of gravity at the surface of the Earth, because the cloud tops are over 9 times as far from the centre of Saturn as the surface of the Earth is from its centre.
The moon gets hit by something very small that weighs two Earth masses, and is traveling very fast. And they stick together. 160 times the mass of the moon smacks into it with a speed of, let's say, several kilometres per second. This object wouldn't stop. It would barely even notice the moon. If the entire moon got in its way, it would sweep it up and continue on its path practically unaware that it was now carrying a moon with it. Since the thing is small, only a bit of the moon gets in its way. It's a very small and extremely fast bullet striking a very large soap bubble. You don't expect the soap bubble to be carried away by the bullet, you expect to find a punctured bubble. You certainly don't expect the bullet to stop dead in the bubble.
160 times the mass. Imagine you're driving down a highway, and a raccoon is crossing the road. Just as your car is about to hit it, the raccoon jumps straight up and hits the front of your car. Your car stops dead as if it had struck a concrete wall, and the mid-air raccoon is barely pushed at all. Even cartoons don't try to get you to believe that.
That much mass, stopping all at once within the moon. Just the kinetic energy released is about the same as the total output of the sun over the space of 48 hours. Not the light hitting the Earth, the light leaving the entire solar sphere. The moon would vanish in a puff of gas. The Earth would vanish in a larger puff of gas.
OK, so suddenly the moon weighs twice what the Earth does. This would have some fairly obvious effects. For one thing, the tides on the surface of the Earth would go from a few metres to a few hundred metres in amplitude. That would have a serious effect on the coastal regions (and with those tides, Missouri is a coastal region).
Increase the mass of the Earth-moon system, and the rotational period will decrease. A month would go from about 30 days to about 10 days. But in the movie, the moon was making complete orbits around the Earth on plot-driven timescales. Sometimes the orbital period was a few days, and toward the end of the movie the orbital period seems to have become about 90 days, because they had deduced that the moon would hit the Earth on this orbit, but they still had 40 days left to try to find a solution. And these weird, sudden "orbital shifts" don't make sense. Yes, an uneven mass distribution can result in smooth and gradual changes to orbits, but the moon didn't have an uneven mass distribution. It was a big mass travelling in orbit, with a light, insignificant, moon stuck to it like a bug on a windshield. The pre-impact mass of the moon isn't even an important perturbation on the mass distribution.
Good news, everybody! We just happen to have a lunar expedition fueled up and ready to go, prepared before the impact. Our heroes can fly to the moon and use some special technology to push the big new mass out of the moon. Well, the mass weighs 160 times what the moon does, so Newton's laws tell us that you're not going to push the mass out of the moon, you're going to push the moon away. The mass won't be appreciably disturbed. The plan is to push the mass out of the moon so it flies toward the sun, but really all you'd do is send the moon away at high speed while the big dense mass stays firmly in its orbit around the Earth.
Now, about this lunar mission. The good news is that you don't need as much fuel to cross over, because of the changes to the shape of the gravitational potential fields in the vicinity. The bad news is that the moon's surface gravity is at least 25 times that of the Earth. That assumes that the colliding mass is at the centre of the moon. In the movie, it's actually only partway down, and our heroes have to land near it, so they'll feel a gravitational force much higher than that. OK, they've been working out, they can walk and work in 25 gravities of force. But their lander was designed to land on rockets in 1/6 normal gravity. It would be like designing a parachute to land you safely, and then you decide that you'll change the parameters, the parachute will be used to land a bit more weight. You plus 159 of your friends, all hanging on the one parachute. You might reasonably conclude that the parachute was not designed for that kind of treatment. A similar argument can be made for the lunar lander and that little rocket jumper vehicle, whose engines certainly cannot supply the thrust to land under the new conditions. The lander would probably crumple under its own weight just trying to sit still on the moon, and taking off from the surface would be similarly difficult because of the changed conditions.
OK. Gravitation, tides, astronomy, orbital motion. If you learned something new from this movie, it's almost certainly wrong.
Friday, January 16, 2009
When your on-the-road ISP blocks your outbound mail
Now, we talked about allowing your computer to relay mail through the home machine when the ISP through which you're connecting has made it onto a block list. What do you do when the ISP simply blocks all outgoing connections on port 25? Now you can't even connect to your home computer to relay the mail.
The ISP does this to force you to pass email through their servers. The hope is that infected Windows computers will just try to open connections directly, and not forward the mail through the ISP servers. As noted in this story, that is not necessarily true.
So, now you find yourself unable to open connections on port 25, but you still want to send email. You could set up your computer to relay mail through the ISP's servers, as described in this earlier article, but that may not be convenient if, for instance, you're accessing the Internet at a relative's home, since they would have to give you their passwords for you to do that.
So, the first thing to do is to check that you can connect to your home computer on the ESMTP port number 587. Telnet onto that port number on your home computer, and if you get a response, then this technique will work for you.
First of all, you should already have set up relaying as described here. If you set it up a while ago, verify that your keys are still valid and haven't expired.
As we're discussing this in the spirit of a temporary work-around, we'll be editing the sendmail.cf file directly. First, of course, make a backup copy of your current sendmail.cf file, because you'll want to reset it to its former behaviour after you stop using this particular ISP.
Now, go into your sendmail.cf file and find the smart relay line. It will look something like this:
Change that line to indicate that you're sending ESMTP to your home machine. It will look a bit like this:
Next, we have to tell sendmail that it is to use port 587 for outbound mail to esmtp smart relays. Locate the block in the sendmail.cf file that looks like this:
and change the last line to read:
That's it. Restart the sendmail program, and you should be able to relay all mail through your home machine using authenticated relaying on port 587.
The ISP does this to force you to pass email through their servers. The hope is that infected Windows computers will just try to open connections directly, and not forward the mail through the ISP servers. As noted in this story, that is not necessarily true.
So, now you find yourself unable to open connections on port 25, but you still want to send email. You could set up your computer to relay mail through the ISP's servers, as described in this earlier article, but that may not be convenient if, for instance, you're accessing the Internet at a relative's home, since they would have to give you their passwords for you to do that.
So, the first thing to do is to check that you can connect to your home computer on the ESMTP port number 587. Telnet onto that port number on your home computer, and if you get a response, then this technique will work for you.
First of all, you should already have set up relaying as described here. If you set it up a while ago, verify that your keys are still valid and haven't expired.
As we're discussing this in the spirit of a temporary work-around, we'll be editing the sendmail.cf file directly. First, of course, make a backup copy of your current sendmail.cf file, because you'll want to reset it to its former behaviour after you stop using this particular ISP.
Now, go into your sendmail.cf file and find the smart relay line. It will look something like this:
# "Smart" relay host (may be null)
DS
Change that line to indicate that you're sending ESMTP to your home machine. It will look a bit like this:
# "Smart" relay host (may be null)
DSesmtp:mail-host.example.com
Next, we have to tell sendmail that it is to use port 587 for outbound mail to esmtp smart relays. Locate the block in the sendmail.cf file that looks like this:
Mesmtp, P=[IPC], F=mDFMuXa, S=EnvFromSMTP/HdrFromSMTP, R=EnvToSMTP, E=\r\n, L=990,
T=DNS/RFC822/SMTP,
A=TCP $h
and change the last line to read:
A=TCP $h 587
That's it. Restart the sendmail program, and you should be able to relay all mail through your home machine using authenticated relaying on port 587.
Monday, December 22, 2008
When other sites discard or refuse your email
We've covered setting up your sendmail to act as a relay for certain computers. Now, we look at another relaying problem.
You like running your own sendmail, you're using it to manage your own email accounts. You could use your ISP's mail server for all outbound messages, but let's say you're not doing that. Now, some third party, maybe another ISP, let's call them "Dogers", decides to silently discard all email coming from IP blocks owned by your ISP unless the sending IP number is one of the mail servers of your ISP. Even if you're running a responsible sendmail on a static IP number, messages sent to "Dogers" just vanish.
The solution is to arrange your sendmail so that, when sending to certain domains, it relays the messages through your ISP's servers. We'll need two more features for this. First, the mailertable function will allow you to use a different mailer for certain addresses. Second, depending on your ISP, you may have to authenticate yourself with the ISP's server before it will relay your messages. This configuration will show how to perform that authentication.
Make sure your sendmail.mc contains the following two lines before the first "MAILER" line:
Also, add the following line anywhere in the file:
You will need to have cyrus-sasl installed, and configured for logins. Here is a sample cyrus-sasl configuration invocation:
OK, now the mailertable entry. Add a line for the dogers domain, telling your sendmail to forward mail for those addresses through your ISP's server:
Now, to authenticate with the ISP. We told sendmail that our credentials would be stored in /etc/mail/auth/client-info, so we create a file there:
Then, we just have to rehash the mailertable and authentication files with a command like this:
Now, assuming you've rebuilt your sendmail.cf after the changes we made to the .mc file above, you can just send a SIGHUP to the sendmail processes, and you should be able to send email to anybody at the dogers.com domain by relaying those messages through your ISP's mail server.
You like running your own sendmail, you're using it to manage your own email accounts. You could use your ISP's mail server for all outbound messages, but let's say you're not doing that. Now, some third party, maybe another ISP, let's call them "Dogers", decides to silently discard all email coming from IP blocks owned by your ISP unless the sending IP number is one of the mail servers of your ISP. Even if you're running a responsible sendmail on a static IP number, messages sent to "Dogers" just vanish.
The solution is to arrange your sendmail so that, when sending to certain domains, it relays the messages through your ISP's servers. We'll need two more features for this. First, the mailertable function will allow you to use a different mailer for certain addresses. Second, depending on your ISP, you may have to authenticate yourself with the ISP's server before it will relay your messages. This configuration will show how to perform that authentication.
Make sure your sendmail.mc contains the following two lines before the first "MAILER" line:
FEATURE(`authinfo',`hash /etc/mail/auth/client-info')dnl
FEATURE(`mailertable')
Also, add the following line anywhere in the file:
define(`confAUTH_MECHANISMS', `EXTERNAL GSSAPI DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
You will need to have cyrus-sasl installed, and configured for logins. Here is a sample cyrus-sasl configuration invocation:
./configure --prefix=/usr/local --enable-anon --enable-plain \
--enable-login --disable-krb4 --with-mysql \
--with-saslauthd=/var/state/saslauthd --with-openssl=/usr/local/ssl \
--with-plugindir=/usr/local/lib/sasl2/ --enable-cram \
--enable-digest --enable-otp --without-des
OK, now the mailertable entry. Add a line for the dogers domain, telling your sendmail to forward mail for those addresses through your ISP's server:
dogers.com smtp:smtp.<MY>.<ISP>
Now, to authenticate with the ISP. We told sendmail that our credentials would be stored in /etc/mail/auth/client-info, so we create a file there:
AuthInfo:smtp.<MY>.<ISP> "U:root" "I:wintertoad@<MY>.<ISP>" "P:<password>" "M:LOGIN"
Then, we just have to rehash the mailertable and authentication files with a command like this:
# makemap hash file.db < file
Now, assuming you've rebuilt your sendmail.cf after the changes we made to the .mc file above, you can just send a SIGHUP to the sendmail processes, and you should be able to send email to anybody at the dogers.com domain by relaying those messages through your ISP's mail server.
Sunday, July 13, 2008
A Curious Permissions Problem with ALSA
While I'm waiting for kde4 to reach everyday usability (defined for my purposes as supporting panel auto-hiding), I periodically check out an updated subversion tree of kde4, compile it, and try it out.
There's some pain with switching between kde3 and kde4 and back again, so I try kde4 with a different login, specially created for testing kde4.
There are still bugs in kde4, particularly when you compile subversion trees, and not specific tagged releases. In my latest foray into kde4, several applications did crash, including amarok, the audio media player.
After spending some time in kde4, I logged out and switched back to kde3 with my regular username. Once there, I found that there was no audio from ALSA applications. So, the usual course when this happens is to examine the permissions on the appropriate audio devices. In this case, however, all of the permissions looked fine.
So, what was preventing ALSA applications from running? When I ran strace on an ALSA application, I found that I was getting permissions problems (reported as EPERM on the return from a syscall) on semctl() syscalls. So, the next step is to run ipcs. This shows the SYSV IPC resources currently in use. There, I found two shared memory segments and two semaphores that were owned by the "kde4" username. Since all kde4-owned processes had exited, this indicated that some process had experienced an abnormal exit without releasing some SYSV IPC resources.
ALSA uses such resources when applications want to generate sound, and it is not possible for an unprivileged user to obtain or release the resources of another user. This produced the permission problems and prevented the applications from working correctly with ALSA.
The solution was to become the root user, and use the ipcrm command to release all resources owned by the kde4 user. Once that was done, ALSA applications run as my regular username could, once again, produce sound.
There's some pain with switching between kde3 and kde4 and back again, so I try kde4 with a different login, specially created for testing kde4.
There are still bugs in kde4, particularly when you compile subversion trees, and not specific tagged releases. In my latest foray into kde4, several applications did crash, including amarok, the audio media player.
After spending some time in kde4, I logged out and switched back to kde3 with my regular username. Once there, I found that there was no audio from ALSA applications. So, the usual course when this happens is to examine the permissions on the appropriate audio devices. In this case, however, all of the permissions looked fine.
So, what was preventing ALSA applications from running? When I ran strace on an ALSA application, I found that I was getting permissions problems (reported as EPERM on the return from a syscall) on semctl() syscalls. So, the next step is to run ipcs. This shows the SYSV IPC resources currently in use. There, I found two shared memory segments and two semaphores that were owned by the "kde4" username. Since all kde4-owned processes had exited, this indicated that some process had experienced an abnormal exit without releasing some SYSV IPC resources.
ALSA uses such resources when applications want to generate sound, and it is not possible for an unprivileged user to obtain or release the resources of another user. This produced the permission problems and prevented the applications from working correctly with ALSA.
The solution was to become the root user, and use the ipcrm command to release all resources owned by the kde4 user. Once that was done, ALSA applications run as my regular username could, once again, produce sound.
Thursday, May 29, 2008
When distributions patch wrongly
Events of recent weeks have provided another reason one might be inclined to avoid the use of distributions. Let's call this the "debian SSL bug". A patch applied by a well-meaning Debian coder made cryptographic keys generated by numerous applications on that distribution entirely useless. Details can be found here.
The Debian patch affected derived distributions as well, such as Ubuntu. For almost two years, many cryptographic transactions were severely compromised. The biggest problem was that the patch was not correctly passed back to development team of the OpenSSL project. Had it been, they would have pointed out its fatal security implications, and this entire headache would have been avoided.
I always feel uncomfortable when I see distributions applying patches against the original sources. There can be several reasons for these patches.
The Debian patch affected derived distributions as well, such as Ubuntu. For almost two years, many cryptographic transactions were severely compromised. The biggest problem was that the patch was not correctly passed back to development team of the OpenSSL project. Had it been, they would have pointed out its fatal security implications, and this entire headache would have been avoided.
I always feel uncomfortable when I see distributions applying patches against the original sources. There can be several reasons for these patches.
- They may be back-porting selected bugfixes to an earlier version of a library rather than including the latest version of the library with all of its new, and possibly untested features.
- They may be modifying a logo or informational string to include something specific to the distribution.
- They may be changing some default pathnames or other resources to mesh more well with the idiosyncracies of their own distribution.
- They may be changing the appearance of the interface to make it more consistent with other applications.
- They may be applying changes that the original maintainers of the package do not consider necessary, but which the distribution maintainers find desirable.
- Other...
Sunday, April 27, 2008
Web browsing behind the great firewall of China
I sometimes spend time in China, and while there, I work remotely to my office and to my home computer. I do somewhat technical work that sometimes requires online research, and it's annoying that a significant fraction of non-Chinese sites are unreachable from China.
The thing to remember is that the firewall isn't there to keep me from working. I'm a Canadian passport holder, and they really don't care what I read while in China. That explains certain curious omissions, such as the fact that TCP port 22 (ssh) is not blocked.
So, here I am, in China, with a Linux laptop, and I'd like to browse the web. Rather than take my chances with the firewall, I proxy the connection through my home computer's apache daemon.
So, first I set up the proxy service on my apache. Make sure you've built the httpd with these configuration options:
These settings turn on the proxy service and set it to proxy HTTP traffic. The "proxy-connect" flag allows the httpd to be used as a reflector for SSL connections. If you want to visit a banking website, the data still travels as SSL between your laptop and the home machine, but the home machine just reflects the traffic to the bank without knowing what's in the data stream (the home machine cannot decode that data, if it could, it would count as a man-in-the-middle compromise of the SSL stream).
Next, add some lines to the httpd configuration file. Mine's in /etc/apache/httpd.conf.
What this does is to enable proxying, but only on connections from localhost. I don't want my httpd to be a proxy for any random person in the outside world.
Next, I set up my ssh on connections to my home computer. You can either add a switch like this to the invocation:
or you can add a line to your ~/.ssh/config entry for the connection to the home computer:
Now, you ssh into your home computer.
Finally, you start up firefox, and select the menu item:
Edit->Preferences->Advanced->Network->Settings
Select "Manual proxy configuration", and point your HTTP and SSL proxies at "localhost" with the port number 8080.
That's it, now when you browse websites, the HTTP-related data stream appears simply as a pile of encrypted bits over your ssh connection. The firewall cannot know what websites you're visiting, it can't even tell that you're visiting a website at all.
Important note: this system proxies the HTTP data. That means web pages, frames, images in the page, RSS feeds, and so on. It does not proxy UDP or post-connection traffic, like youtube videos. If your web browser has a plugin that downloads data from an external site, that plugin may not be using your proxy.
If you want to know what data is not passing through your proxy, you can run tcpdump in another window. Something like this:
where is the IP number of your external interface (not 127.0.0.1). You may have to add a "-i" switch if your laptop has more than one network interface. This command will show you all traffic that is not going over the ssh connection.
The thing to remember is that the firewall isn't there to keep me from working. I'm a Canadian passport holder, and they really don't care what I read while in China. That explains certain curious omissions, such as the fact that TCP port 22 (ssh) is not blocked.
So, here I am, in China, with a Linux laptop, and I'd like to browse the web. Rather than take my chances with the firewall, I proxy the connection through my home computer's apache daemon.
So, first I set up the proxy service on my apache. Make sure you've built the httpd with these configuration options:
--enable-mods-shared="proxy proxy-http proxy-connect"
These settings turn on the proxy service and set it to proxy HTTP traffic. The "proxy-connect" flag allows the httpd to be used as a reflector for SSL connections. If you want to visit a banking website, the data still travels as SSL between your laptop and the home machine, but the home machine just reflects the traffic to the bank without knowing what's in the data stream (the home machine cannot decode that data, if it could, it would count as a man-in-the-middle compromise of the SSL stream).
Next, add some lines to the httpd configuration file. Mine's in /etc/apache/httpd.conf.
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
<IfModule mod_proxy.c>
ProxyRequests On
<Proxy *>
Order deny,allow
Deny from all
Allow from 127.0.0.1
</Proxy>
</IfModule>
What this does is to enable proxying, but only on connections from localhost. I don't want my httpd to be a proxy for any random person in the outside world.
Next, I set up my ssh on connections to my home computer. You can either add a switch like this to the invocation:
-L 8080:127.0.0.1:80
or you can add a line to your ~/.ssh/config entry for the connection to the home computer:
LocalForward 8080 127.0.0.1:80
Now, you ssh into your home computer.
Finally, you start up firefox, and select the menu item:
Edit->Preferences->Advanced->Network->Settings
Select "Manual proxy configuration", and point your HTTP and SSL proxies at "localhost" with the port number 8080.
That's it, now when you browse websites, the HTTP-related data stream appears simply as a pile of encrypted bits over your ssh connection. The firewall cannot know what websites you're visiting, it can't even tell that you're visiting a website at all.
Important note: this system proxies the HTTP data. That means web pages, frames, images in the page, RSS feeds, and so on. It does not proxy UDP or post-connection traffic, like youtube videos. If your web browser has a plugin that downloads data from an external site, that plugin may not be using your proxy.
If you want to know what data is not passing through your proxy, you can run tcpdump in another window. Something like this:
tcpdump 'host <IPNUM> and not port 22'
where
Wednesday, April 2, 2008
Fixing sound in Linux Civ:CTP
When my old 64-bit motherboard died, and I replaced it with the DP35DP, one of the surprises I ran into was that the sound was badly broken on "Civilization:Call To Power". All other applications that I tried worked well, any programs using the ALSA interface, as well as a few 32-bit binaries on the OSS interface, like Quake 2 and Heroes 3. However, with Civ:CTP, the sound stuttered and looped horribly. I couldn't use the aoss wrapper because Civ:CTP is statically linked. After a lot of tinkering, I finally came to the conclusion that, for this one application, I had to load the sound module with different parameters.
For every application except Civ:CTP, my snd-hda-intel module is loaded with the parameters
However, in order to run Civ:CTP, I have to exit all sound applications, remove the snd-hda-intel module, and re-load it with the parameters:
With this change, the sound in Civ:CTP sounds fine. However, all other applications have poor sound, scratchy and unpleasant to the ear, so I only make this module change just prior to running the game, and re-load the module with the usual parameters immediately afterwards.
For every application except Civ:CTP, my snd-hda-intel module is loaded with the parameters
position_fix=1 model=5stack
However, in order to run Civ:CTP, I have to exit all sound applications, remove the snd-hda-intel module, and re-load it with the parameters:
position_fix=3 model=5stack
With this change, the sound in Civ:CTP sounds fine. However, all other applications have poor sound, scratchy and unpleasant to the ear, so I only make this module change just prior to running the game, and re-load the module with the usual parameters immediately afterwards.
Wednesday, March 19, 2008
Experiences compiling X11R7
Compiling and installing X11R7, x.org 7.3, was a bit more rough than the X11 compiles I used to perform. I used the build script supplied with the source packages. When it finished, apparently successfully, there were two problems whose solutions were not obvious.
First of all, OpenGL worked on my NVidia box, but not my ATI laptop.
Second, my Chinese fonts for traditional characters in Emacs looked different, much worse. The simplified Chinese characters still looked fine.
OK, what were the problems, and how did I fix them. First of all, the OpenGL issue. I compiled X11 the same way on both computers, why did OpenGL not work on the ATI laptop? Well, both NVidia and ATI ship closed-source binary blobs with support libraries. The difference is that NVidia supplies its own libGL.so, while ATI uses the one from x.org. So, somehow I was failing to compile and install the OpenGL stuff. This didn't matter for the NVidia case, because it supplied all of the libraries required, but ATI doesn't do that. I had compiled and installed libMesa, so OpenGL should have worked. The OpenGL component is compiled as part of the xorg-server-1.4 package, and its configure script is executed by the build script that came with x.org. Aha, but in order to compile OpenGL, you have to provide the configure script with the path of the libMesa source tree. The x.org build script doesn't do that, so OpenGL is not built. The solution is to interrupt the build at the point where the xorg-server-1.4 is about to be built (you can edit the script and put in an 'exit 0' there, for instance), then configure, build, and install the xorg-server-1.4 archive by hand, remembering to tell it where the mesa source tree is located. Once that completes, you can continue the build with the xorg script (I just commented out all entries above the server compile and resumed).
Now, the font problem. My TTF fonts are in /usr/share/fonts, and I verified that the files there were being read when I asked Emacs to display Chinese characters. So, it appeared as if the Chinese TTF fonts were the ones that were looking bad. A bit of research showed that Emacs does not, as of version 22.1.1, use scalable fonts. So I decided that it probably wasn't supposed to be using those TTF fonts. Now, I had kept my old X11R6 tree around in case of issues like this, so a quick comparison of directories showed that there were some Chinese PCF fonts in the old install that I had forgotten to copy to the new location. So, I copied these files into their location in the X11R7 tree, and Emacs was restored to its former behaviour with respect to the displaying of Chinese fonts. The fonts, by the way, are taipei15.pcf, taipei16.pcf, taipei24.pcf, taipei24k.pcf, and taipeil24.pcf.
First of all, OpenGL worked on my NVidia box, but not my ATI laptop.
Second, my Chinese fonts for traditional characters in Emacs looked different, much worse. The simplified Chinese characters still looked fine.
OK, what were the problems, and how did I fix them. First of all, the OpenGL issue. I compiled X11 the same way on both computers, why did OpenGL not work on the ATI laptop? Well, both NVidia and ATI ship closed-source binary blobs with support libraries. The difference is that NVidia supplies its own libGL.so, while ATI uses the one from x.org. So, somehow I was failing to compile and install the OpenGL stuff. This didn't matter for the NVidia case, because it supplied all of the libraries required, but ATI doesn't do that. I had compiled and installed libMesa, so OpenGL should have worked. The OpenGL component is compiled as part of the xorg-server-1.4 package, and its configure script is executed by the build script that came with x.org. Aha, but in order to compile OpenGL, you have to provide the configure script with the path of the libMesa source tree. The x.org build script doesn't do that, so OpenGL is not built. The solution is to interrupt the build at the point where the xorg-server-1.4 is about to be built (you can edit the script and put in an 'exit 0' there, for instance), then configure, build, and install the xorg-server-1.4 archive by hand, remembering to tell it where the mesa source tree is located. Once that completes, you can continue the build with the xorg script (I just commented out all entries above the server compile and resumed).
Now, the font problem. My TTF fonts are in /usr/share/fonts, and I verified that the files there were being read when I asked Emacs to display Chinese characters. So, it appeared as if the Chinese TTF fonts were the ones that were looking bad. A bit of research showed that Emacs does not, as of version 22.1.1, use scalable fonts. So I decided that it probably wasn't supposed to be using those TTF fonts. Now, I had kept my old X11R6 tree around in case of issues like this, so a quick comparison of directories showed that there were some Chinese PCF fonts in the old install that I had forgotten to copy to the new location. So, I copied these files into their location in the X11R7 tree, and Emacs was restored to its former behaviour with respect to the displaying of Chinese fonts. The fonts, by the way, are taipei15.pcf, taipei16.pcf, taipei24.pcf, taipei24k.pcf, and taipeil24.pcf.
Tuesday, March 11, 2008
Selective sendmail relaying based on self-signed keys
Back in the early days of the Internet, people trusted one another not to abuse email. Sure, there were accidents. A badly configured mailing list could fill up with traffic as two vacation programs talked to one another, each informing the other that his latest message would not be read until some later date, because the recipient was out of the office.
In those days, you set up your sendmail to relay messages for others. Many people had email addresses that weren't on a full-time connection to the network, they might be on a BBS that did a nightly download of email, or down some Bitnet rabbit hole. Email was relayed from one intermediate post to another, rather than being simply sent directly from the sender to the receiver. A sendmail daemon that relayed messages for others was helpful to the community, everybody pitched in to get everyone's email where it was ultimately intended.
Then came new developments. Canter & Siegel, the September that never ended, and the presence of people who would buy things they saw in an unsolicited email message. Spam started to appear in mailboxes. Suddenly, being a helpful person and relaying messages was no longer beneficial to the community, as commercial email senders used relays to hide the origins of their messages. People started turning off open relays on their boxes as a defensive move.
So, now you've got a domain set up with a sendmail daemon at home, and you're traveling with a laptop. To make this a bit more complicated, let's say your laptop is a work computer, and you send email from its sendmail, but with a different domain than your home computer. Everything's working fine, until you find that the coffee shop in Beijing where you're using your laptop has made it onto a list of spamming IP numbers. Some recipients of your messages may not receive them because their sendmail is set up to refuse messages from computers on these bad IP numbers. You know that your home computer is not on a banned IP number, so it would be nice if you could forward your laptop-generated work-related messages through your home computer. It would be even nicer if people selling generic pharmaceuticals could not do the same thing, otherwise your home computer's IP number will very quickly find itself on one of those banned lists. So, you want to allow relaying from your laptop, but only from your laptop, and do it easily even if you move to another coffee shop.
What you want, then, is a way for your home computer to recognize your laptop, and permit only that computer to relay messages through the home sendmail. This will be done with sendmail's TLS facility. You will create a private certificate authority, one you don't have to pay to sign your keys. You'll then use a signed certificate to verify the identity of the laptop. The following procedure will be performed on the home computer, only at the end of this process will the laptop be involved.
We'll start by creating two directories on your home computer, one for the certificate authority, and the other for the signed certificates. I'll use the directory locations that are found in the default OpenSSL configuration file, so that you don't have to edit too many files.
Copy the OpenSSL openssl.cnf file into /etc/mail/CA.
Next, we will create the signing certificate.
When prompted, you will have to enter a pass phrase twice. Remember this phrase, you will need it if you ever want to sign certificates with this signing certificate.
This command creates new files: /etc/mail/CA/demoCA/cacert.pem and /etc/mail/CA/demoCA/private/cakey.pem. The file contains encoded information related to a certificate signing authority that will be valid for 1000 days.
Next, you must create the certificate that you will use to validate your laptop. You enter the commands:
Again, you will have to answer some questions. Here is a sample dialogue:
Now, you have a certificate for your laptop, but it hasn't yet been signed. You use the signing certificate to vouch for the laptop certificate. First, we have to set up a bit more information for the signing process:
Now, we issue two commands to sign the laptop certificate:
Now, it's time to tell the home machine's sendmail that it should relay messages received from this key. Add a line to the /etc/mail/access.src file that looks like this:
You'll have to make that file readable by sendmail:
And now we have to make sure that the home machine's sendmail knows where to find its certificates and access file. Build a new sendmail.cf using a sendmail.mc something like this:
Now, we move some things around a bit. We copy the signing certificate and laptop signed certificate like this:
The three files, demoCA/cacert.pem, laptopcert.pem and signed-laptopcert.pem get copied onto the laptop, in its /etc/mail/certs directory. Now, you must tell the laptop's sendmail that these are its certificates. This is done by building (on the laptop) the sendmail.cf file from a sendmail.mc file that looks roughly like this:
Finally, you'll have to decide when you want to relay through the home computer. You really have two choices. You could set it up so that all messages are always relayed through the home computer, by setting a smart relay in your sendmail.cf, or you could relay them explicitly. There are other places that identify the technique for setting up a smart relay, so I'll just describe the second, on-demand version.
If you are trying to send email from your laptop to the user somebody@example.net, but want to relay it through your home computer at example.com, you would send the message to this email address:
And there you go, on-demand secure relaying of messages through your home computer.
In those days, you set up your sendmail to relay messages for others. Many people had email addresses that weren't on a full-time connection to the network, they might be on a BBS that did a nightly download of email, or down some Bitnet rabbit hole. Email was relayed from one intermediate post to another, rather than being simply sent directly from the sender to the receiver. A sendmail daemon that relayed messages for others was helpful to the community, everybody pitched in to get everyone's email where it was ultimately intended.
Then came new developments. Canter & Siegel, the September that never ended, and the presence of people who would buy things they saw in an unsolicited email message. Spam started to appear in mailboxes. Suddenly, being a helpful person and relaying messages was no longer beneficial to the community, as commercial email senders used relays to hide the origins of their messages. People started turning off open relays on their boxes as a defensive move.
So, now you've got a domain set up with a sendmail daemon at home, and you're traveling with a laptop. To make this a bit more complicated, let's say your laptop is a work computer, and you send email from its sendmail, but with a different domain than your home computer. Everything's working fine, until you find that the coffee shop in Beijing where you're using your laptop has made it onto a list of spamming IP numbers. Some recipients of your messages may not receive them because their sendmail is set up to refuse messages from computers on these bad IP numbers. You know that your home computer is not on a banned IP number, so it would be nice if you could forward your laptop-generated work-related messages through your home computer. It would be even nicer if people selling generic pharmaceuticals could not do the same thing, otherwise your home computer's IP number will very quickly find itself on one of those banned lists. So, you want to allow relaying from your laptop, but only from your laptop, and do it easily even if you move to another coffee shop.
What you want, then, is a way for your home computer to recognize your laptop, and permit only that computer to relay messages through the home sendmail. This will be done with sendmail's TLS facility. You will create a private certificate authority, one you don't have to pay to sign your keys. You'll then use a signed certificate to verify the identity of the laptop. The following procedure will be performed on the home computer, only at the end of this process will the laptop be involved.
We'll start by creating two directories on your home computer, one for the certificate authority, and the other for the signed certificates. I'll use the directory locations that are found in the default OpenSSL configuration file, so that you don't have to edit too many files.
mkdir /etc/mail/CA /etc/mail/certs /etc/mail/CA/demoCA /etc/mail/CA/demoCA/private
Copy the OpenSSL openssl.cnf file into /etc/mail/CA.
Next, we will create the signing certificate.
$ cd /etc/mail/CA
$ openssl req -new -x509 -keyout demoCA/private/cakey.pem -out demoCA/cacert.pem -days 1000 -config openssl.cnf
$ openssl req -new -x509 -keyout demoCA/private/cakey.pem -out demoCA/cacert.pem -days 1000 -config openssl.cnf
Generating a 1024 bit RSA private key
.............++++++
.........++++++
writing new private key to 'demoCA/private/cakey.pem'
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Ontario
Locality Name (eg, city) []:Toronto
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:Bert Ificate
Email Address []:bertificate@example.com
When prompted, you will have to enter a pass phrase twice. Remember this phrase, you will need it if you ever want to sign certificates with this signing certificate.
This command creates new files: /etc/mail/CA/demoCA/cacert.pem and /etc/mail/CA/demoCA/private/cakey.pem. The file contains encoded information related to a certificate signing authority that will be valid for 1000 days.
Next, you must create the certificate that you will use to validate your laptop. You enter the commands:
$ cd /etc/mail/CA
$ openssl req -nodes -new -x509 -keyout laptopcert.pem -out laptopcert.pem -days 365 -config openssl.cnf
Again, you will have to answer some questions. Here is a sample dialogue:
$ openssl req -nodes -new -x509 -keyout laptopcert.pem -out laptopcert.pem -days 365 -config openssl.cnf
Generating a 1024 bit RSA private key
....++++++
............................................++++++
writing new private key to 'laptopcert.pem'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:CA
State or Province Name (full name) [Some-State]:Alberta
Locality Name (eg, city) []:Calgary
Organization Name (eg, company) [Internet Widgits Pty Ltd]:Example
Organizational Unit Name (eg, section) []:
Common Name (eg, YOUR name) []:Rhoda Warrior
Email Address []:rhoda-warrior@example.com
Now, you have a certificate for your laptop, but it hasn't yet been signed. You use the signing certificate to vouch for the laptop certificate. First, we have to set up a bit more information for the signing process:
$ mkdir /etc/mail/CA/demoCA/newcerts
$ touch /etc/mail/CA/demoCA/index.txt
$ echo 01 > /etc/mail/CA/demoCA/serial
Now, we issue two commands to sign the laptop certificate:
$ openssl x509 -x509toreq -in laptopcert.pem -signkey laptopcert.pem -out tmp.pem
$ /usr/local/ssl/bin/openssl ca -config openssl.cnf -policy policy_anything -out signed-laptopcert.pem -infiles tmp.pem
$ openssl ca -config openssl.cnf -policy policy_anything -out signed-laptopcert.pem -infiles tmp.pem
Using configuration from openssl.cnf
Enter pass phrase for ./demoCA/private/cakey.pem:
Check that the request matches the signature
Signature ok
Certificate Details:
Serial Number: 1 (0x1)
Validity
Not Before: Mar 12 00:46:43 2008 GMT
Not After : Mar 12 00:46:43 2009 GMT
Subject:
countryName = CA
stateOrProvinceName = Alberta
localityName = Calgary
organizationName = Example
commonName = Rhoda Warrior
emailAddress = rhoda-warrior@example.com
X509v3 extensions:
X509v3 Basic Constraints:
CA:FALSE
Netscape Comment:
OpenSSL Generated Certificate
X509v3 Subject Key Identifier:
67:11:5A:25:6F:5C:70:36:03:14:3B:04:4A:8C:30:C9:CF:60:51:AE
X509v3 Authority Key Identifier:
keyid:F4:CE:58:BD:82:8A:E3:EC:0F:89:C6:60:E2:45:58:A4:CA:79:C8:89
Certificate is to be certified until Mar 12 00:46:43 2009 GMT (365 days)
Sign the certificate? [y/n]:y
1 out of 1 certificate requests certified, commit? [y/n]y
Write out database with 1 new entries
Data Base Updated
Now, it's time to tell the home machine's sendmail that it should relay messages received from this key. Add a line to the /etc/mail/access.src file that looks like this:
CertIssuer:/C=CA/ST=Ontario/L=Toronto/O=Example/CN=Bert+20Ificate/emailAd
dress=bertificate@example.com RELAY
You'll have to make that file readable by sendmail:
makemap hash access.db < access.src
And now we have to make sure that the home machine's sendmail knows where to find its certificates and access file. Build a new sendmail.cf using a sendmail.mc something like this:
divert(0)dnl
VERSIONID(`sendmail.mc for example.com version 01')
OSTYPE(linux)dnl
DOMAIN(example.com)dnl
FEATURE(`nouucp', `reject')
FEATURE(`virtusertable', `hash /etc/sendmail/virtusertable')dnl
FEATURE(`genericstable', `hash /etc/sendmail/genericstable')dnl
FEATURE(`local_procmail', `/usr/local/bin/procmail')
FEATURE(`access_db', `hash -T<TMPF> /etc/mail/access')
FEATURE(`mailertable')
MAILER(local)
MAILER(smtp)
define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/CAcert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')dnl
Now, we move some things around a bit. We copy the signing certificate and laptop signed certificate like this:
$ cd /etc/mail/CA
$ /bin/cp signed-laptopcert.pem /etc/mail/certs
$ /bin/cp demoCA/cacert.pem /etc/mail/certs/CAcert.pem
$ cd /etc/mail/certs
$ ln -s signed-laptopcert.pem `openssl x509 -noout -hash < signed-laptopcert.pem`.0
The three files, demoCA/cacert.pem, laptopcert.pem and signed-laptopcert.pem get copied onto the laptop, in its /etc/mail/certs directory. Now, you must tell the laptop's sendmail that these are its certificates. This is done by building (on the laptop) the sendmail.cf file from a sendmail.mc file that looks roughly like this:
divert(0)dnl
VERSIONID(`$Id: generic-linux.mc,v 8.1 1999/09/24 22:48:05 gshapiro Exp $')
OSTYPE(linux)dnl
DOMAIN(example.net)dnl
define(`confCACERT_PATH', `/etc/mail/certs/')
define(`confCACERT', `/etc/mail/certs/cacert.pem')
define(`confCLIENT_CERT', `/etc/mail/certs/laptopcert.pem')
define(`confCLIENT_KEY', `/etc/mail/certs/signed-laptopcert.pem')
define(`confSERVER_CERT', `/etc/mail/certs/laptopcert.pem')
define(`confSERVER_KEY', `/etc/mail/certs/signed-laptopcert.pem')
FEATURE(`genericstable')
FEATURE(`virtusertable')
FEATURE(`local_procmail', `/usr/local/bin/procmail')
MAILER(local)dnl
MAILER(smtp)dnl
Finally, you'll have to decide when you want to relay through the home computer. You really have two choices. You could set it up so that all messages are always relayed through the home computer, by setting a smart relay in your sendmail.cf, or you could relay them explicitly. There are other places that identify the technique for setting up a smart relay, so I'll just describe the second, on-demand version.
If you are trying to send email from your laptop to the user somebody@example.net, but want to relay it through your home computer at example.com, you would send the message to this email address:
somebody%example.net@example.com
And there you go, on-demand secure relaying of messages through your home computer.
Thursday, February 28, 2008
Why don't I get spam?
I have an anti-spam trick. It won't work for most people, but there might be some people out there who are inclined to take advantage of it. For the rest, this might be educational.
The trick that I use depends on the fact that I have my own domain. That means I can run sendmail on my computer, and I can create email addresses quickly and easily. I will use the domains example.com, example.net, and example.org for this document, as recommended in RFC 2606.
The basic idea is this: instead of having one email address, I have dozens. I create a new email address for every person with whom I exchange messages, as well as addresses for websites and companies when necessary. If an email address is accidentally revealed, or if one of the companies decides to start sending annoying amounts of unsolicited mail, I simply expire the email address and, if desired, contact the sending party to tell them about the new address. I don't have to contact all of my friends whenever I turn off one address, only the one person who uses that address to talk to me.
OK, how is this implemented? There are two things I have to do. First, I need my sendmail to accept the messages for the active addresses, and send them all to me. Second, I have to ensure that my outbound email has the correct Reply-To: address for the particular recipient of the message.
If you're familiar with sendmail, you can probably guess how I do the first thing. I set up a virtual user table. Here's the sendmail.mc file used to make this work:
Now, recall that sendmail doesn't read the virtusertable.src file, it reads another file called virtusertable.db. I've got a little Makefile in /etc/mail that I use to keep things up to date:
So, that's the receiving side. How about sending? There may be a way to configure sendmail to rewrite the outbound addresses according to a database of recipients, but I haven't figured one out. Instead, I have written a bit of code for my email client, which is rmail mode in Emacs. Here are the relevant bits of Emacs Lisp:
It also sets the sender address to bounces0000@example.com, which means that automated replies, such as sendmail daemon warnings and errors, will be delivered to that address. It should be redirected in the virtusertable to some appropriate address so that you can be notified of problems at the recipient's end (though many systems no longer generate bounce messages, because of spam abuse).
Anyway, with all this, I get really no spam. Every few months I may get one message on one of my email addresses, typically one that I used for a forum post or to send a bug report or patch to a mailing list. I retire the address, set up a new one, and never get spam at that address again.
Some time later I'll describe the cryptographic certificates in the mail configuration, and how they allow secure relaying.
The trick that I use depends on the fact that I have my own domain. That means I can run sendmail on my computer, and I can create email addresses quickly and easily. I will use the domains example.com, example.net, and example.org for this document, as recommended in RFC 2606.
The basic idea is this: instead of having one email address, I have dozens. I create a new email address for every person with whom I exchange messages, as well as addresses for websites and companies when necessary. If an email address is accidentally revealed, or if one of the companies decides to start sending annoying amounts of unsolicited mail, I simply expire the email address and, if desired, contact the sending party to tell them about the new address. I don't have to contact all of my friends whenever I turn off one address, only the one person who uses that address to talk to me.
OK, how is this implemented? There are two things I have to do. First, I need my sendmail to accept the messages for the active addresses, and send them all to me. Second, I have to ensure that my outbound email has the correct Reply-To: address for the particular recipient of the message.
If you're familiar with sendmail, you can probably guess how I do the first thing. I set up a virtual user table. Here's the sendmail.mc file used to make this work:
divert(0)dnl
VERSIONID(`sendmail.mc for example.com version 01')
OSTYPE(linux)dnl
DOMAIN(example.com)dnl
FEATURE(`nouucp', `reject')
FEATURE(`virtusertable', `hash /etc/sendmail/virtusertable')dnl
FEATURE(`genericstable', `hash /etc/sendmail/genericstable')dnl
FEATURE(`local_procmail', `/usr/local/bin/procmail')
FEATURE(`access_db', `hash -T<TMPF> /etc/mail/access')
FEATURE(`mailertable')
GENERICS_DOMAIN(mailhost.example.com)
MAILER(local)
MAILER(smtp)
define(`CERT_DIR', `MAIL_SETTINGS_DIR`'certs')dnl
define(`confCACERT_PATH', `CERT_DIR')dnl
define(`confCACERT', `CERT_DIR/CAcert.pem')dnl
define(`confSERVER_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confSERVER_KEY', `CERT_DIR/MYkey.pem')dnl
define(`confCLIENT_CERT', `CERT_DIR/MYcert.pem')dnl
define(`confCLIENT_KEY', `CERT_DIR/MYkey.pem')dnl
Cw mailhost.example.com
Cw example.com
hotels@example.com error:nouser Spammers found this address
disposable0000@example.com myself
ebay@example.com myself
electronics@example.com myself
slashdot@example.com myself
thinkgeek@example.com myself
mail-default-0000-r2@example.com myself
mail-0000-r7@example.com myself
mail-0001-q3@example.com myself
mail-0002-a4@example.com myself
mail-0003-c8@example.com error:nouser Spammers found this address
mail-0004-d7@example.com myself
Now, recall that sendmail doesn't read the virtusertable.src file, it reads another file called virtusertable.db. I've got a little Makefile in /etc/mail that I use to keep things up to date:
all : genericstable.db virtusertable.db mailertable.db aliases.db access.db
%.db : %.src
makemap hash $* < $<
aliases.db : aliases
newaliases
hup : all
killall -HUP sendmail
make -C /etc/mail hup
So, that's the receiving side. How about sending? There may be a way to configure sendmail to rewrite the outbound addresses according to a database of recipients, but I haven't figured one out. Instead, I have written a bit of code for my email client, which is rmail mode in Emacs. Here are the relevant bits of Emacs Lisp:
(setq user-mail-address "bounces0000@example.com")
(setq mail-specify-envelope-from t)
(setq outbound-address-alist
'(
("joe@example.org" "mail-0000-r7@example.com")
("frank@example.org" "mail-0001-q3@example.com")
("wilbur@example.net" "mail-0002-a4@example.com")
("euripedes@example.net" "mail-0004-d7@example.com")
(nil "mail-default-0000-r2@example.com")
)
)
(setq full-name "Winter Toad")
;; a function to parse out the header and send email as if from
;; different usernames. That way, I can obsolete a username if it
;; gets spam.
(add-hook 'mail-send-hook
'(lambda ()
(narrow-to-region 1 (mail-header-end))
(beginning-of-buffer)
(expand-mail-aliases 1 (mail-header-end))
(re-search-forward "^To: ")
;; parse out the recipient address
(let (recipient from-whom)
(cond
((looking-at "\\([^ \\t]*\\)$")
(setq recipient (match-string 1)))
((looking-at "[^<]*<\\([^>]*\\)>$")
(setq recipient (match-string 1))))
(setq from-whom (or (cadr (assoc recipient outbound-address-alist))
(cadr (assoc nil outbound-address-alist))))
(end-of-line)
(newline)
(insert "From: " full-name " <" from-whom ">")
(re-search-forward "^Reply-to: ")
(let ((namestart (point-marker)))
(end-of-line)
(kill-region namestart (point-marker))
(insert from-whom)))
(narrow-to-region 1 (1+ (buffer-size)))))
It also sets the sender address to bounces0000@example.com, which means that automated replies, such as sendmail daemon warnings and errors, will be delivered to that address. It should be redirected in the virtusertable to some appropriate address so that you can be notified of problems at the recipient's end (though many systems no longer generate bounce messages, because of spam abuse).
Anyway, with all this, I get really no spam. Every few months I may get one message on one of my email addresses, typically one that I used for a forum post or to send a bug report or patch to a mailing list. I retire the address, set up a new one, and never get spam at that address again.
Some time later I'll describe the cryptographic certificates in the mail configuration, and how they allow secure relaying.
Tuesday, February 26, 2008
Installing in non-standard places
I mentioned earlier the possibility of choosing an install prefix like /usr/local/samba, which installs the Samba libraries in a directory that may not commonly exist on distribution-managed machines. One possible effect of this is that you may turn up bugs in configuration and compilation scripts of other packages.
A configure script for another package may accept arguments related to the location of Samba libraries and header files, but compiling the package with these options set might not work. This isn't very surprising, it's a compilation option that is probably rarely used, so bit rot has a tendency to set in. A change somewhere that accidentally breaks the compilation when Samba is installed in an unusual place might not be noticed for some time. By putting Samba in its own directory, you are setting yourself up to test a valid, but rarely exercised option. You may find yourself submitting bug reports and patches to the package maintainers.
As I've said before, maintaining your box without a package manager and distribution is not easy. It's quite a bit more work, but it does force you to understand more about how the system is set up and what it's doing. For people who like the extra control and understanding this provides, this is a useful technique.
A configure script for another package may accept arguments related to the location of Samba libraries and header files, but compiling the package with these options set might not work. This isn't very surprising, it's a compilation option that is probably rarely used, so bit rot has a tendency to set in. A change somewhere that accidentally breaks the compilation when Samba is installed in an unusual place might not be noticed for some time. By putting Samba in its own directory, you are setting yourself up to test a valid, but rarely exercised option. You may find yourself submitting bug reports and patches to the package maintainers.
As I've said before, maintaining your box without a package manager and distribution is not easy. It's quite a bit more work, but it does force you to understand more about how the system is set up and what it's doing. For people who like the extra control and understanding this provides, this is a useful technique.
Sunday, February 24, 2008
Pharyngula readers in Ottawa
PZ over at Pharyngula reports that readers of his blog are meeting up in various places. Well, if there are any people in Ottawa who are interested in meeting, we can try to set it up here in the comments.
Any place I can get to by bus is fine with me, maybe a weekend lunch time? Possibilities would be
Any place I can get to by bus is fine with me, maybe a weekend lunch time? Possibilities would be
- Lone Star at Baseline and Fisher
- Sushi Kan at Baseline and Merivale
- Some place in Chinatown
Thursday, February 21, 2008
A Followup On Cryptographic Mounts, The Bad News
Previously, I discussed cryptographic mounts to hold sensitive data. It's worth pointing out an article that is making the rounds today by 9 authors from Princeton, in which the researchers describe an attack on cryptographic techniques, including the one I've described.
The technique relies on the fact that modern memory can retain its information for several minutes after the computer stops sending it refresh signals. What this means is that a person with physical access to the computer can pull the power connector from the computer and then remove the memory chips, insert them in another computer, and read the cryptographic keys out of the memory. I don't know of a good way to avoid this attack. If the cryptographic volumes are mounted when the computer falls into the hands of the attacker, the data will be, in theory, recoverable.
So, what can be done to prevent the key from being resident in the computer's memory at the instant that the attacker unplugs it? The key has to be available to the operating system so that it can read and write that data in normal operation. Sure, you could get specially modified hardware that deliberately overwrites the main memory from batteries when the power connector is removed, but maybe there's a way to store 128 bits somewhere other than in main memory?
A cache line on a modern CPU is 64 bytes, big enough to hold two 128-bit keys. Could the operating system subvert the hardware's L1 caching mechanism sufficiently to pin a value in the cache and remove it from L2 and main memory? This attack won't recover data from the L1 cache, so if that's the only place the key is kept, maybe that would be enough. You sacrifice a cache line, but maybe it's worth it?
How about the TLB? That's another part of the CPU that holds data, and that one is explicitly designed to interact with the operating system. Could we find a way to store 128 bits in parts of the TLB, and then deliberately avoid overwriting them? Can the operating system read those numbers back out of the TLB?
Are there any registers that could be used? Probably not on 32-bits, there aren't many registers there, and on 64-bits you'd probably have to use a special-purpose compiler to avoid these registers being touched by a context switch, and avoid them being saved to memory when an interrupt handler runs.
What if you have fifteen keys, all of 128 bits? Well, I believe we could handle that if we had 256 bits of volatile storage space. The first 128 bits of volatile space holds an XOR key, that decodes all of the fifteen keys. The second 128 bits of volatile space holds the decoded key in active use.
Those are my thoughts, anyway.
The technique relies on the fact that modern memory can retain its information for several minutes after the computer stops sending it refresh signals. What this means is that a person with physical access to the computer can pull the power connector from the computer and then remove the memory chips, insert them in another computer, and read the cryptographic keys out of the memory. I don't know of a good way to avoid this attack. If the cryptographic volumes are mounted when the computer falls into the hands of the attacker, the data will be, in theory, recoverable.
So, what can be done to prevent the key from being resident in the computer's memory at the instant that the attacker unplugs it? The key has to be available to the operating system so that it can read and write that data in normal operation. Sure, you could get specially modified hardware that deliberately overwrites the main memory from batteries when the power connector is removed, but maybe there's a way to store 128 bits somewhere other than in main memory?
A cache line on a modern CPU is 64 bytes, big enough to hold two 128-bit keys. Could the operating system subvert the hardware's L1 caching mechanism sufficiently to pin a value in the cache and remove it from L2 and main memory? This attack won't recover data from the L1 cache, so if that's the only place the key is kept, maybe that would be enough. You sacrifice a cache line, but maybe it's worth it?
How about the TLB? That's another part of the CPU that holds data, and that one is explicitly designed to interact with the operating system. Could we find a way to store 128 bits in parts of the TLB, and then deliberately avoid overwriting them? Can the operating system read those numbers back out of the TLB?
Are there any registers that could be used? Probably not on 32-bits, there aren't many registers there, and on 64-bits you'd probably have to use a special-purpose compiler to avoid these registers being touched by a context switch, and avoid them being saved to memory when an interrupt handler runs.
What if you have fifteen keys, all of 128 bits? Well, I believe we could handle that if we had 256 bits of volatile storage space. The first 128 bits of volatile space holds an XOR key, that decodes all of the fifteen keys. The second 128 bits of volatile space holds the decoded key in active use.
Those are my thoughts, anyway.
Wednesday, February 20, 2008
Choosing an install prefix
As noted in this posting, you generally will have to choose an install prefix for software that you are compiling yourself. Most packages you encounter will be configured to install under /usr/local, though some will be configured for /usr.
The first thing you'll want to do is to see if you already have an older version of the software installed anywhere. If the software was previously installed under /usr/local, and you install the new package under /usr, not only will you needlessly consume disk space, but the version that is run will depend on the setting of your PATH environment variable. A user may report that he can't use a certain feature in the new version, and it may take you a while to notice that his environment variable differs from yours, and that he's still running the old software. So, find the name of an executable that you expect will be installed. For example, if you're installing the binutils software, you will expect that the ld binary should be installed somewhere. Next, type the command:
Sometimes a package installs only one or a few binaries. You may decide to install this into its own directory. For example, I install firefox into the prefix /usr/local/firefox, SBCL into the prefix /usr/local/sbcl, and the apache httpd into /usr/local/apache2. These get their own directories because, while they may install a very small number of executables, they come with a large set of ancillary files. Rather than installing over top of the old directory, I move the old directory to a new location, say "/usr/local/sbcl.old", and then install and test the new version. If the new version doesn't work properly, I can revert to the old one by deleting the new install and renaming the ".old" directory. Alternatively, I can compare the two installations, the previously working one against the new one, and see if there are any obvious differences that could account for problems.
Of course, you probably won't be able to type the command firefox and expect it to run if it's installed in /usr/local/firefox/bin/. You will either want to add that directory to the PATH variable, or, more conveniently, put a symbolic link to the appropriate executable from a directory that is in your PATH. This command:
So, you really have a couple of choices. Put the program into a standard place, like /usr or /usr/local (and if upgrading try to install over top of the old version by using the same prefix that was used then), or installing the software in its own dedicated directory, like /usr/local/firefox or /usr/local/sbcl.
Now, when you set the prefix in an autoconf configure script, it also sets a number of derived values which can be separately overridden. Configuration files are, by default, put in <prefix>/etc, libraries in <prefix>/lib, headers in <prefix>/include, man pages in <prefix>/share/man (sometimes omitting the 'share' component), log files in <prefix>/var/log, and so on. The configure program lets you override these defaults separately, so that you can put configuration files into, say, /etc/http with the option "--sysconfdir=/etc/http", and so on. Think carefully about whether you want these additional directories to keep their defaults. You probably don't want your X-server log to be in /usr/X11/var/log, nobody will know where to look for it.
The first thing you'll want to do is to see if you already have an older version of the software installed anywhere. If the software was previously installed under /usr/local, and you install the new package under /usr, not only will you needlessly consume disk space, but the version that is run will depend on the setting of your PATH environment variable. A user may report that he can't use a certain feature in the new version, and it may take you a while to notice that his environment variable differs from yours, and that he's still running the old software. So, find the name of an executable that you expect will be installed. For example, if you're installing the binutils software, you will expect that the ld binary should be installed somewhere. Next, type the command:
which ld
Sometimes a package installs only one or a few binaries. You may decide to install this into its own directory. For example, I install firefox into the prefix /usr/local/firefox, SBCL into the prefix /usr/local/sbcl, and the apache httpd into /usr/local/apache2. These get their own directories because, while they may install a very small number of executables, they come with a large set of ancillary files. Rather than installing over top of the old directory, I move the old directory to a new location, say "/usr/local/sbcl.old", and then install and test the new version. If the new version doesn't work properly, I can revert to the old one by deleting the new install and renaming the ".old" directory. Alternatively, I can compare the two installations, the previously working one against the new one, and see if there are any obvious differences that could account for problems.
Of course, you probably won't be able to type the command firefox and expect it to run if it's installed in /usr/local/firefox/bin/. You will either want to add that directory to the PATH variable, or, more conveniently, put a symbolic link to the appropriate executable from a directory that is in your PATH. This command:
ln -s /usr/local/firefox/bin/firefox /usr/X11/bin/firefox
So, you really have a couple of choices. Put the program into a standard place, like /usr or /usr/local (and if upgrading try to install over top of the old version by using the same prefix that was used then), or installing the software in its own dedicated directory, like /usr/local/firefox or /usr/local/sbcl.
Now, when you set the prefix in an autoconf configure script, it also sets a number of derived values which can be separately overridden. Configuration files are, by default, put in <prefix>/etc, libraries in <prefix>/lib, headers in <prefix>/include, man pages in <prefix>/share/man (sometimes omitting the 'share' component), log files in <prefix>/var/log, and so on. The configure program lets you override these defaults separately, so that you can put configuration files into, say, /etc/http with the option "--sysconfdir=/etc/http", and so on. Think carefully about whether you want these additional directories to keep their defaults. You probably don't want your X-server log to be in /usr/X11/var/log, nobody will know where to look for it.
Compiling and installing by hand
If you're not using a package manager, or if you are, but there is no package available for a piece of software you'd like to install, you'll find yourself compiling the software by hand. Generally, you start by locating the official web page of the software, downloading an appropriate version of the source code, and extracting the tar file to a directory somewhere.
At this point in the process, you are not doing anything as the root user. You'll become root much later in this process.
The next thing you'll do is look in the top level of the extracted directory for promising looking files, like README, INSTALL, or Makefile. It is likely that you will see an executable script called "configure". It's always a good idea to start by looking at the README and INSTALL files, if present. They may be in the toplevel directory, or in a documentation directory, which will often have a name like "doc", "docs", or "documentation", possibly with different capitalizations.
If The Package Came With A Makefile
If there's a Makefile in the toplevel, that's usually because the software package is fairly small. You will want to look over the Makefile to ensure that it is correct for your intended installation. The most important things to look for are the installation directory and any optional features that might have to be turned on by editing the Makefile. If you can't find the installation directory, type the command:
This will ask "make" to print out the sequence of commands that it will be using to install the package. Since you haven't compiled anything yet, it will start with the sequence of commands required to compile your software, so look for the installation commands to occur near the end of the output generated by this command.
If your package came with a Makefile, you will now modify the Makefile if necessary, perhaps changing the installation directory of the product. You should do this before compiling it, because sometimes character strings holding the pathnames of configuration files are inserted into the compiled binary, so changing the installation target after compiling may result in an installation that doesn't work correctly. Editing the Makefile will usually not force a recompilation of the objects under its control, that is the Makefile is not, by default, considered a dependency for the targets in the file.
After this, you will, still as your non-root user, compile the package. This is usually done by simply entering the command make. If errors are encountered during the compile, you'll have to figure out what happened and how to fix it. The most common causes of errors are:
At this point, move down to the section of the text called "Installing The Software".
You Have A "configure.am" Script, But No "configure" Script
If you have a "configure.am" script, but no "configure" script, you'll have to generate the configure script. If there is an executable in this directory with a name like "autogen.sh", run it. This should be sufficient to set up the configure script. If you don't have an autogen script, you should run the commands automake then autoconf. This will often generate warnings, but unless the configure script you generate doesn't run, you can ignore those. So, now you have a configure script, you continue to the next section.
You Have A "configure" Script
If you generated the configure script yourself, you know that it's an autoconf configure script. Sometimes, though, software is produced that has a completely different script that happens to be called "configure". This can be confusing if it doesn't recognize the switch "--help". Start by typing:
There are a few things to look at in the options you get from "configure". One of them is the prefix location, and choosing that properly can require some care, which is discussed here. For now, let's assume that you've chosen a set of options that look suitable. You re-run the configure script with those options, and without the "--help" option. It will do some things, it may take a considerable amount of time to run. Eventually, the script should exit, sometimes generating a list of all options and whether or not they are active. Examine this list if present, there might be an option that you want to enable that has been turned off because the configure script failed to find a particular library, in which case you'll have figure out why that option was disabled and figure out how to get it working. When you're satisfied with the compilation options, type "make". If an error is encountered, see the possibilities mentioned in the earlier section referring to building from a Makefile. If you succeed in compiling the software package, go to next section, "Installing The Software".
Installing The Software
Now, you can become the root user. Change directory to the location where you compiled the binary, and run
Many packages these days produce a pkg-config file. This is usually a filename that ends in ".pc", and is installed in a directory like ".../lib/pkgconfig/". The pkg-config application often looks for these files when "configure" is being run, but it has a fairly definite idea of where to look. If your .pc file was installed into a directory where pkg-config doesn't normally look, you'll have to find some way to make this file visible to that program. There are three ways you can handle this:
Two more hints: "configure" writes its command line into a comment near the top of the file "config.log". If you need to remember how you last ran "configure", you will find the options you used there.
If you have a particularly detailed set of configure options, you might want to record them in a directory somewhere for future reference, both to see quickly what options you enabled when you compiled the software and to re-use the command the next time you recompile it after downloading a new version.
At this point in the process, you are not doing anything as the root user. You'll become root much later in this process.
The next thing you'll do is look in the top level of the extracted directory for promising looking files, like README, INSTALL, or Makefile. It is likely that you will see an executable script called "configure". It's always a good idea to start by looking at the README and INSTALL files, if present. They may be in the toplevel directory, or in a documentation directory, which will often have a name like "doc", "docs", or "documentation", possibly with different capitalizations.
If The Package Came With A Makefile
If there's a Makefile in the toplevel, that's usually because the software package is fairly small. You will want to look over the Makefile to ensure that it is correct for your intended installation. The most important things to look for are the installation directory and any optional features that might have to be turned on by editing the Makefile. If you can't find the installation directory, type the command:
make -n install
This will ask "make" to print out the sequence of commands that it will be using to install the package. Since you haven't compiled anything yet, it will start with the sequence of commands required to compile your software, so look for the installation commands to occur near the end of the output generated by this command.
If your package came with a Makefile, you will now modify the Makefile if necessary, perhaps changing the installation directory of the product. You should do this before compiling it, because sometimes character strings holding the pathnames of configuration files are inserted into the compiled binary, so changing the installation target after compiling may result in an installation that doesn't work correctly. Editing the Makefile will usually not force a recompilation of the objects under its control, that is the Makefile is not, by default, considered a dependency for the targets in the file.
After this, you will, still as your non-root user, compile the package. This is usually done by simply entering the command make. If errors are encountered during the compile, you'll have to figure out what happened and how to fix it. The most common causes of errors are:
- missing include files - you might have to add a "-I<directory>" to the CFLAGS, CXXFLAGS, or CPPFLAGS variables in your Makefile.
- missing libraries - you might have to add a "-L<directory>" to the LDFLAGS variable in your Makefile.
- bad version - the compilation may depend on a library you have on your machine, but the version you have may not be compatible with the software package. You might have to download a different version of that library and install it before you can continue with the software package.
- apparent code errors - the compiler may generate errors related to missing variables, bad function declarations, or syntax errors. Resist the urge to correct these immediately, and try to understand why you are seeing these errors. Remember, this package probably compiled for somebody before they released it, why doesn't it work for you? Is it that your compiler is a different version, and flags as errors things that used to be warnings? Is the Makefile configured for the wrong architecture or platform? Something else?
make -n install | less
At this point, move down to the section of the text called "Installing The Software".
You Have A "configure.am" Script, But No "configure" Script
If you have a "configure.am" script, but no "configure" script, you'll have to generate the configure script. If there is an executable in this directory with a name like "autogen.sh", run it. This should be sufficient to set up the configure script. If you don't have an autogen script, you should run the commands automake then autoconf. This will often generate warnings, but unless the configure script you generate doesn't run, you can ignore those. So, now you have a configure script, you continue to the next section.
You Have A "configure" Script
If you generated the configure script yourself, you know that it's an autoconf configure script. Sometimes, though, software is produced that has a completely different script that happens to be called "configure". This can be confusing if it doesn't recognize the switch "--help". Start by typing:
./configure --help | less
There are a few things to look at in the options you get from "configure". One of them is the prefix location, and choosing that properly can require some care, which is discussed here. For now, let's assume that you've chosen a set of options that look suitable. You re-run the configure script with those options, and without the "--help" option. It will do some things, it may take a considerable amount of time to run. Eventually, the script should exit, sometimes generating a list of all options and whether or not they are active. Examine this list if present, there might be an option that you want to enable that has been turned off because the configure script failed to find a particular library, in which case you'll have figure out why that option was disabled and figure out how to get it working. When you're satisfied with the compilation options, type "make". If an error is encountered, see the possibilities mentioned in the earlier section referring to building from a Makefile. If you succeed in compiling the software package, go to next section, "Installing The Software".
Installing The Software
Now, you can become the root user. Change directory to the location where you compiled the binary, and run
make install
ldconfig
Many packages these days produce a pkg-config file. This is usually a filename that ends in ".pc", and is installed in a directory like ".../lib/pkgconfig/". The pkg-config application often looks for these files when "configure" is being run, but it has a fairly definite idea of where to look. If your .pc file was installed into a directory where pkg-config doesn't normally look, you'll have to find some way to make this file visible to that program. There are three ways you can handle this:
- Add the appropriate directory to the system-wide environment variable PKG_CONFIG_PATH. Usually this means editing /etc/profile. You likely want it set at least to "/usr/lib/pkgconfig:/usr/local/lib/pkgconfig:/usr/X11/lib/pkgconfig", but you may want to add more search directories to it, if you expect many packages to be installed in the same prefix.
- Copy the .pc file into a directory that pkg-config searches. This is unwise, you may install another version of the software some time later, and unless you remember this step your .pc file will still be the old one, causing much aggravation as "configure" insists you still have a version of the package that you know you just replaced.
- Put a symbolic link to the file from a directory that is searched by pkg-config. Do this if you've got only one or two .pc files in this prefix, and don't expect to put in more.
Two more hints: "configure" writes its command line into a comment near the top of the file "config.log". If you need to remember how you last ran "configure", you will find the options you used there.
If you have a particularly detailed set of configure options, you might want to record them in a directory somewhere for future reference, both to see quickly what options you enabled when you compiled the software and to re-use the command the next time you recompile it after downloading a new version.
Subscribe to:
Posts (Atom)