December 30, 2005

WMF 0 Day exploit

WMF 0 Day exploits are surfacing already. Some of which look like worms. Link

December 22, 2005

bash on OS X

Funny thing happened... I had a directory with 2 dmp files. One was compressed using gzip, and one was un-compressed. When I typed gzip -d and pressed tab, it listed the only compressed file which had a name ending in gz. No matter how many times I pressed tab, it didn't pick the other file which I know exists in that directory. So obviously, me being me, typed gzip without -d on the shell and pressed tab... and voila, it listed the uncompressed file and didn't list the other file which was already compressed.

I've been working with bash for a long time, and haven't noticed such a feature before... I was pleasently surprised to find it.

December 12, 2005

Fun writing a search engine


Introduction


My interesting project for this quarter was writing a search engine to index blog entries. The experience to do something like this without knowing anything about resources required would probably be risky and stupid. But since this was just an educational project to undertand search technology and to learn java, capacity planning was last thing on my mind.

Based on the resource I had, it was pretty clear to me that I can't build another yahoo or google. Besides who needs another one of those anyway. Also when I started working on this project, google hadn't released thier blog search engine. Needless to say indexing blogs looked pretty interesting. How difficult could it be to build another Technorati anyway ? A few servers running a crawler and a few database servers is all one needs with a nice front end written in pretty php.


Crawling


If search engine was all just about searching a text from a database, then it would be called a database. To build a search engine one needs to solve a few other parts of the puzzle. One of them was the "crawler". If I had done my homework before I started off, I would have found out that there are quite a few crawlers already available for free.

A crawler's job is to crawl around on the internet looking for new URLs, which in this case needs to be RSS/Atom feeds. A unintelligent bot would probably go around in no particular order following new links. The crawler I wrote had a few optimizations which helped it collect a massive amount of URLs very quickly.

  • Supply with very rich seed urls

  • Allow feeds to be harnessed for new sources of feeds

  • Check sites like weblogs.com for new URLs periodically

  • Looking for words like 'rss' or 'atom' in URL and excluding those with 'gif' and 'jpg' would help identify the type of object without downloading it

  • One more thing which I should have done, but didn't was to use the HEAD command to get content type before downloading the complete URL.

  • When a URL is downloaded, it is parsed to look for more href links to other URLs.




Identifying feeds


Writing crawler is simple because there are only as many HTTP versions. RSS is another story. Everyone and their friends are coming out with thier own RSS extentions. Rome is how I got away with it. These guys are going a long way to build a generic RSS reader, and they make it a lot easier to add your own modules to it to read new add-ons.

Unfortunately one thing which I haven't investigated enough is the differences in various feed formats. Rome makes it so transparent that I didn't have to do too much of digging to accept the different formats. However, since it doesn't make sence to crawl multiple feeds(different formats) of the same site, I'll have to do this exercise at some point to figure out which feed format is better for indexing purposes.

As I said before, Rome, allows developers to add their own modules into Rome to support reading of non-standard (or newer) xml tags which are not supported by Rome by default. Form example I noticed some feeds had two versions of 'description', one was longer than the other. When I noticed that Rome was picking the wrong one, I wrote a module to extract the longer description. Similarly I also wrote an iTunes content extracter which helped Rome to understand iTunes content in Feeds.


Spam blogs


There have been studies done on the internet by various organizations and they have all concluded that we live in a very polluted blogging world , where people have been using blogs to skew search engine results.

Weblogs is a good place to see these spam logs in action. Some of the interesting patterns which I used to reduce the effect of spam were

  • Ignore blogs which publish large number of items at a time. Except some new websites most blogs are published one at a time over a long period of time

  • I noticed that spam bloggers are very uncreative when it comes to finding a name for the blog. Most of them have a - in the blog name or blog domain name. Skipping such domains helped a lot in reducing spam.

  • Free blog services are the largest source of spam blogs. Blogspot turned out to be the most annoying one when I analysed this problem

  • If I had some kind of language or pharse analyser, its possible that most of the items in the same feed would look very similar to each other. I didn't have time to do this, but I'm sure one of you guys would do it eventually




Other notes


This writeup is a work in progress.... here are some more notes which I still have to write about

  • You need to know databases very very very well to understand how to optimize searches

  • Mysql is good, but few critical features which it doesn't have could improve your search engine a lot

  • Java is expensive... c/c++ would be better. But Java is very easy to develop in. Google started by using python... so in short its possible

  • There is a lot of crap out there...

  • Its possible that 60 to 70% of the blogs pinging weblogs.com are spam

  • You better be ready to handle UTF8. I learnt it the hard way

  • And even if things are in latin script, not everything is in english

  • If you are planning to index blogs, make sure u understand the languages...

  • But even if you do understand the languages... you should look at all the new words these teens come up with. I swear I didn't recogonize half of them as english untill I really tried hard to understand them. I thought 3l33t was cool... but these are are way ahead

  • Distributed computing is the way to go. One computer itself can't do much

  • Make sure you have plans for scaling ready at hand. You have no idea how fast these things grow.

  • And when you start doing so many HTTP queries, weird things happen on the network. On mine, the router required a reset once every day. I'd never heard of this before... and when I called in to complain about this one year old product, even customer support hadn't heard about it


December 11, 2005

Interesting movie...

This is an interesting movie of the space shuttle launch.

RSS Hijacking on the rise...

RSS hijacking is probably unavoidable in short term. Here is an interesting discussion on this topic

Root servers

Some of you probably know what a Root Server is. But did you know that they had a site dedicated to it. Ofcourse internet won't exist without it, but I thought this information would be burried under one of the sites like ICANN or the US MIL site. You can download the latest named.root file from this location.

December 08, 2005

First post...

The pleasure of first post :)

August 26, 2005

Rsync for incremental backups


Introduction


Archiving files to tape is still considered one of the cheapest way of making backups. However with the prices of disk storage and solid-state storage decreasing rapidly, it won't be long before users make the switch to the faster disk storage for all thier backup needs. The problem, however, is that if you want to do anything more than mirroring data on a remote storage, there aren't too many good freeware tools to do it. This writeup explains one of the interesting ways to do incremental backups with snapshot cabability using a popular tool called rsync.


Traditional Backup


Traditional backup applications not only support backing up and restoring of files, directories, partitions and drives, but also allow incrementally backups to reduce time taken to backup large file systems. Since its not practical to restore the complete data from tape just to apply minor changes to it, most tape backup software store the differences in a seperate file or location on tape which can be used to patch the last full backup image on the tape. This feature of storing incremental updates also allows administrators to maintain multiple versions of data without keeping as many physical copies of data. Incremental updates to backup repository is extremly valuable feature in highly dynamic environments where maintaining multiple snapshot of data taken very frequently is important.


Rsync


Rsync was one of the first tools I used, which allows one to update copies of data by sending incremental updates. This dramatically cuts down the time to update a copy of data. The problem is that though rsync allows you to make a copy of data, and allows you to incrementally update it, it was not designed with tape in mind. It specifically doesn't allow you to keep the incremental updates in a different directory or file the way backup applications do. This limits the number of snapshots one can maintain using rsync.


The cp command


And that brings us to the last part of the puzzle which we need to know to make do incremental backups. The "cp" command on most unix operating systems allows copying of hard links (instead of the actual data). This feature allows you to maintain two physically different directories on the same partition (with different names) pointing to the same physical set of files. In linux this can be accomplised by the following command "cp -al $sourcedir $targetdir".


Rsync + cp


To demonstrate how these two can work together to provide us snapshot capability I did some tests on my linux box. The first step was to create a small directory structure which we would be using for this exercise. "ls -ila" on the directory shows the actual "inodes" (column 1) assigned to each of these files and directories within the test directory I created.


Original Directory Structure



    List of files
    ==============================
    /tmp/test/primary
    /tmp/test/primary/file1.txt
    /tmp/test/primary/file2.txt
    /tmp/test/primary/subdir1/file3.txt
    ==============================

    la:/tmp/test # ls -ila primary/*
    358315 -rw-r--r-- 1 root root 5 Aug 21 23:06 primary/file1.txt
    358316 -rw-r--r-- 1 root root 5 Aug 21 23:07 primary/file2.txt
    primary/subdir1:
    total 4
    358313 drwxr-xr-x 2 root root 80 Aug 21 23:07 .
    358312 drwxr-xr-x 3 root root 136 Aug 21 23:07 ..
    358314 -rw-r--r-- 1 root root 5 Aug 21 23:07 file3.txt



Next step is to do a traditional recursive copy from "primary" to "directory1". You can accomplish this by either "cp" or "rsync". In this following example I used "cp" command. Notice that when I list the inodes after the cp commands, it creates a new set of inodes for each of the files and directory in the new directory structure. This means that the traditional "cp" command did an actual recursive copy of file contents to new locations, and that there exists two identical copies of each of the objects.


After Copy using "cp -rp src target"



    la:/tmp/test # cp -rp primary directory1
    la:/tmp/test # ls -ila directory1/*
    358307 -rw-r--r-- 2 root root 5 Aug 21 23:06 directory1/file1.txt
    358308 -rw-r--r-- 2 root root 5 Aug 21 23:07 directory1/file2.txt
    directory1/subdir1:
    total 4
    358305 drwxr-xr-x 2 root root 80 Aug 21 23:07 .
    115666 drwxr-xr-x 3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r-- 2 root root 5 Aug 21 23:07 file3.txt



Now lets see how "cp" behaves when we ask it to preserve hardlinks. In this example we are copying "directory1" into a new directory "directory2". Notice how the inodes in the new directory are same as the ones from "directory1". This means that though there are two logical directories which look alike, the actual file and directories listed within each one of them are identical. Any modification done to one file within one directory (without modifying the inode) will affect the file in the other directory. This is almost same as symbolic linking, except that unlike symbolic links the file wont dissapear from "directory2" if I delete it from "directory1". In other words there is actually multiple owners of these inodes at this moment, which seems a little hard to digest.


"cp -la src target"



    la:/tmp/test # cp -al directory1 directory2
    la:/tmp/test # ls -ila directory2/*
    358307 -rw-r--r-- 2 root root 5 Aug 21 23:06 directory2/file1.txt
    358308 -rw-r--r-- 2 root root 5 Aug 21 23:07 directory2/file2.txt
    directory2/subdir1:
    total 4
    358310 drwxr-xr-x 2 root root 80 Aug 21 23:07 .
    358309 drwxr-xr-x 3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r-- 2 root root 5 Aug 21 23:07 file3.txt



So we know how interesting hard links are and we know how to create multiple directories look exactly the same without creating as many copies of the actual data. A little more research on your part would reviel that if you had modified "subdir1/file3.txt" the only two inodes which would change are "subdir1" and "subdir1/file3.txt". I didn't show inodes of "primary" directory in the dumps below, but what I did do is show you how the inodes look like after I rsync the changes from "primary" to "directory1".Notice that after rsync to "directory1" the inodes for "subdir1" and "subdir1/file3.txt" has changed (as expected). This is because rsync usually doesn't overwrite existing inodes. Instead it creates fresh copies of updated files and directories and deletes the old ones. Interestingly inodes of "directory2" still shows the old inodes for the files/directories which were modified.

"directory2" has now become a "snapshot" of "directory1" without actually having a duplicate copy of all the data in "directory1".


Modified file3.txt in primary copy"



    la:/tmp/test # rsync -rvgoutl primary/* directory1/
    building file list ... done
    subdir1/
    subdir1/file3.txt
    wrote 161 bytes read 40 bytes 402.00 bytes/sec
    total size is 21 speedup is 0.10

    la:/tmp/test # ls -ila directory1/*
    358307 -rw-r--r-- 2 root root 5 Aug 21 23:06 directory1/file1.txt
    358308 -rw-r--r-- 2 root root 5 Aug 21 23:07 directory1/file2.txt
    directory1/subdir1:
    total 4
    358305 drwxr-xr-x 2 root root 80 Aug 21 23:14 .
    115666 drwxr-xr-x 3 root root 136 Aug 21 23:07 ..
    358322 -rw-r--r-- 1 root root 11 Aug 21 23:14 file3.txt

    la:/tmp/test # ls -ila directory2/*
    358307 -rw-r--r-- 2 root root 5 Aug 21 23:06 directory2/file1.txt
    358308 -rw-r--r-- 2 root root 5 Aug 21 23:07 directory2/file2.txt
    directory2/subdir1:
    total 4
    358310 drwxr-xr-x 2 root root 80 Aug 21 23:07 .
    358309 drwxr-xr-x 3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r-- 1 root root 5 Aug 21 23:07 file3.txt




August 14, 2005

Bluetooth on the way back

When King Danish Harald Blåtand, united Norway and Denmark, little did he know that a technology named after him ( Blåtand translates to blue-tooth) will have a chance of becoming a corner stone of the telecommunication industry.

This industry is one of the fastest growing sectors in todays world, and whether you'd like it or not it is constantly changing the world around you.
If it were not for the cell phone industry, we would still be hooked to our wired phones, and had it not been for the internet E-mails would just have been a fantasy.
And in this fast changing world one protocol which is growing very rapidly is 'bluetooth' . And just like everything before 'bluetooth' wasn't created in a day. In fact it went through some rough times before its started catching on again.The telecom industry today is not very different from what it was 1000s of years ago. There still are many different ways to communicate and some are more popular than others. But human ingenuity over time and has lead to unification of communication protocols. Though it may look like its doing the same thing, a telephone is very different from a cellphone and a cellphone is different from a satellite phone. But they all manage to get along very well, and if I call your home phone line from a cellphone in US over a satellite connection, it will still reach you and we'd still be able to talk. Internet is another perfect example of this unification which brought together computers worldwide.

While people were still fascinated by internet and wired networks, in the early 90s Ericsson predicted that the day is not far away when computers inside your home will talk to other computers and even with other electronic devices like cell phones, digital cameras, keyboards and mouse wirelessly. In 1994 they started an effort to come up with a standard for devices to communicate with each other they way computers can over wired networks. This search for a new, inexpensive communication standard ( protocol ) which could allow one device to detect the presence of another and allow it to communicate with another it using low powered radio signals was soon joined by 5 companies. Unfortunately, in spite of some early success, the process of defining a standard slowed down significantly by 1999 when the consortium had over 1200 company participants. This is when blue-tooth's problems started.

While bluetooth was still in its infancy, a new protocol IEEE 802.11 started gaining momentum. This new communication protocol was specifically designed for high speed communication between computers and networking devices using radio frequency. This was probably the toughest moment in the history of bluetooth. Eclipsed by 802.11s success bluetooth standard was on the verge of extinction.

Interestingly, though 802.11 is faster, allowed greater distances and supported much more communication features, its complexity required the device to do more work and send stronger radio signals for it to be able to communicate with others. This inadvertently forced it to draw much more power. This was not a problem for devices which are hooked up to the power, or for laptops which are charged very frequently, but it definitely was a problem for devices like cellphones and digital camera's which have very small battery capacity and cant be connected to power outlet for extended periods of times. This together with the realization of low cost of manufacturing bluetooth devices marked the comeback of this unique protocol from the dead. IEEE 802.11 still has very strong market presence, but bluetooth has carved a niche for itself which has a very big fan base.

The most popular bluetooth device today which demonstrates the power and simplicity this popular protocol is the cellphone. Most new cellphones allow users to exchange phone numbers with a click of a button, some allow you to transfer files ands photographs between your computer, some even allow you to talk using handsfree bluetooth headset. Among the other devices which are very quickly catching on are bluetooth enabled keyboards and mouse which replace ps2 and usb wires giving the users the freedom of moving around without being tied to their computers.

Infact the day is not far when you would see bluetooth in remote controls of you televisions and VCRs and may be one day even be able to control your VCR from your cellphone. Bluetooth brings with it the freedom of communication with other devices which is unmatched by anything else in the communication industry today.

June 01, 2005

Mac OS X: The Unix admin perspective


Introduction




My desire to own a PowerBook grew out of few failed attempts at switching to laptops running linux. It's been a month since I ordered a 12inch PowerBook G4 and this seems to be a good time to write down my thoughts for those of you who are still on the fence. Being a unix systems administrator myself, I was more interested in the things inside the Operating System rather than the jazzy user interface outside. Though this write-up doesn't do justice to the feature-rich OS which we all know as Mac OS X, hopefully it would give some of you an interesting perspective of what I noticed when I switched to it for the first time.


Background


I've always surprised myself at how engrossed I get every-time I lay my hands on a new toy. I remember looking for things which others miss (sometimes by breaking it). As a Unix systems administrator by profession, I officially have a job of hacking OS, scripts and looking for performance tune-ups wherever possible. Interestingly one of the qualities a Unix admin develops over time is the ability to solve problems before a user gets the opportunity to ask themselves. For example ask any unix admin and they will tell you the number of times users ask "Why is my server slow ? ", or "Why is my process slow when it does XYZ operation". Then there are times when the users may not even ask you a question until the servers crashes for good. Thats where monitoring, log analysis and an inquisitive brain comes into play.

My first Unix


Even though I have never qualified myself as an early adopter, I've always tried to be more open to technology on the edge which has enough rough edges to cut you. After getting bored of writing TSRs (Terminate and Stay Resident Programs) and playing with Software Interrupts in MS-DOS and early version of windows back in 1995 I played with my first Unix OS when I deployed linux on my desktop using 5.25 inch disks which I think was an Intel-386 back then. It was a lot of fun breaking out of MS-Dos and Windows into something different and it felt even greater to have such a grip over the Operating system on your Desktop.

Compiling kernel for 30 minutes at a time for a few times every night was as fun loving as it was important to keep up with the fast changes the kernel was going through. But more than the kernel it was the OS with its vast ocean of free utilities which fascinated me. From that day till today I've used linux almost every single day of my life.


Saying good bye to Windows (unsuccessfully)


I've tried many times to say good bye to windows for good but never been able to for one of the many reasons which come back and haunt me.

  • Drivers : Finding drivers for all your hardware has always been a problem. Unless you build a desktop for Linux it was difficult to find all the drivers to run your hardware optimally

    • Sound Drivers: This had been a pain point until a few years ago.

    • Network Drivers: This was a problem initially, but it wasn't anymore if one knew the host OS was going to be Linux. In fact until recently Wireless networking drivers had the same problem

    • Video Drivers: Again it was a problem initially, but if you were using one of the recognized drivers you were fine



  • Software: It didn't take me long to figure out that the only way to use Linux for 100% of your work is to make sure you have all your tools for Linux. In fact back in the old days when piracy was rampant in India when no one every bought any software, I actually payed 100 dollars for Linux Office suite, which was the first software I ever paid for in my Life. That was long before OpenOffice.But the line of work I've been in since then almost always required me to own a full blown Windows OS. It was hard to refuse the same guys who were paying you

  • VMware: Around the time when Redhat 7.2 came out I made one more effort to switch to 100% linux (sortof) by switching my primary OS to Linux and by running Windows under emulation in VMware. I think this was one of my longest lasting attempts to run Linux as my primary OS. This was also the time when I bought my own personal laptop for the first time (Dell Inspiron 5100) and went ahead to do the same on my laptop too.For the most part this worked fine, except that I realized that for most of the times I was was doing both Unix and Windows tasks at the same time. Switching between Unix and Windows in VMware worked too... but not as elegantly as I'd liked it to be. Copying files back and forth was also not as clean as I'd wanted it to be. But it was progress in the right direction.But VMware was not free. Everytime I upgraded kernel/OS to a new distribution I had to either spend time to look on the net how to make old vmware drivers work with the new kernel, or wait for some time before VMware comes out with a new version of their OS which supports your new Linux Distribution/Kernel. Ofcourse you would have to pay for it all over again

    My love affair with VMware ended when I got frustrated that I couldn't get VMware to work after a Gentoo updates.






Choosing the right OS for the right job


Choosing an OS for job is not a game of absolute numbers. If anyone says otherwise they are just lying... or they don't know any better. At different stages of my live I've been flabbergasted many times for using Microsoft Windows in a Unix environment and giving my MCSE certification while I was employed as Unix Administrator. To have an absolute perception of anything without seeing the alternatives, however, is the worst thing you can do. Its no surprise that I use different OS at my work and home hardware depending on what its supposed to do.

Desktop at Work


Though I have always used Linux as my file server at home and for my website, I soon realized that using it at work may not be ideal in long run. My switch back to Windows was smooth and I was pleased with the speed gain. Since I we didn't actually use X11 on our Infrastructure there wasn't much I lost by the switch. Putty/WinSCP was good enough for most of my stuff, and Cygwin made up for the rest.

Time spent on hacking reduced

What shocked me was that I wasn't wasting much time upgrading the kernel and was focused on the real work at hand. I remember how many days I worked to get dual-head (two monitors) hooked up to my desktop running one of the older version of Gentoo. And even though I don't regret spending that time hacking X11 configuration files, I could have done something better if I had used Windows back then to do the same which had full support for dual head.


My hands were tied

I think the problem I was suffering from was that I couldn't control myself when I'm given too many options. Every new Linux Distribution out there and every release of of these distributions was an invitation to upgrade/migrate which I rarely refused. Funny enough, the very reason why I first switch to Linux for was the reason why I was frustrated of it and decided to switch back to Windows for Work.

The other problem I had was that unlike Microsoft Windows the life-cycle of Free Linux based Operating Systems were relatively shorter. After buying a Dell Laptop with Windows OS Professional on it, atleast I didn't have any desire to spend more money on Commercial Enterprise linux which had more support and longer shelf life. And unless I commit myself to one particular distribution I noticed that upgrade was always a challenge which kept on changing with the Distribution and version. Since my speciality was server security and Unix OS, I didn't have to worry much about Windows once I was part of my company NT domain. The patches were automatically served and I didn't have to worry about figuring out upgrades and driver incompatibility.



Linux Server for Home


At Home however, I preferred a little more control over the OS. After trying out Fedora for a while I quickly realized that it was not the same as RedHat which I was used to, and dumped it (and RedHat stock) in favor of Suse which has been running at home for a while now. Buy my choice of Linux didn't go uncontested. There was a brief period of few months when I was running both Windows and Linux at home. That was also the time I was experimenting with C# and .NET to see if I can use that instead of Linux/Perl for my web server.

Even though I was very impressed with .NET and C# (especially Visual Studio .NET), I soon realized that not only was it too bloated and expensive, it was also a one-way user lock-in technology, which wasn't the direction I wanted to go if I needed the flexibility of switching out of that architecture.

Features and applications like Squid, kernel-firewall support, ftp server, apache, php, perl, snort, bash, etc which most Unix admins take for granted in a Linux environment didn't come with windows. In fact for some of the stuff to really work well, you'd have to buy more Microsoft software which wasn't very cheap.


Windows for Laptop at Home


Laptop had a different problem. When I bought my 2.8Ghz Dell inspiron I quickly figured out that building kernels wasn't going to take as long as my older desktops. But I still had a very hard time getting drivers (sound/network/vpn/video) for all the devices on the laptop. Even if I were to use VMware on linux, it wouldn't be able to support all the devices which I need. Instead of spending time hacking and researching on the net, I made an executive decision to stay with windows+Cygwin ( I did try dual boot for short time without much success).




Thinking about Mac




Two years down the line I still have the same laptop and I still miss Linux a lot. When I got the opportunity to buy another laptop, it was not surprising that I was thinking about this all over again. Did I really want to spend more time to figure out if my laptop will work with linux ? Or should I pay premium and buy a pre-installed Linux laptop with some small no-name company ?

I have been using iPod for 6 months now, and have also been following interesting news about Apple's OS X, its Unix Internals, PowerBooks, 64Bit G5s and iPhoto. Its then that I realized that its not the Linux Operating system I'm seeking for, but the utilities provided by a Unix operating system to keep a Unix admin feel at home. I needed the drag and drop of windows and the power of kill -9 without having to install multiple 3rd party tools.

When I tried out Mac at the local Apple stores the only thing I could say was WOW. It seemed responsive, flexible, sexy and had very good driver support from 3rd party vendors. But coming from a Dell camp, the price was a concern. A 15inch laptop with just 512mb ram and 80GB drive costs significantly higher than a cheap laptop from Dell. The final decision to buy 12inch Powerbook however was made when I thought about all the troubles I had gone through in carrying around a 15inch 8 pound Dell Inspiron laptop. Trust me when I say that a 15inch laptop is the last thing you want to carry on an airplane.

First impressions


The powerbook which I bought "just to check it out", has changed my perception of Apple and OS X enough to say that "gosh, why I didn't do this before ?". Some of the impressions below were biased because I'd been stuck with a clunky 15 inch heavy dell laptop for 2 years. But again, these are just my personal impressions and you have the right to think otherwise.

Light

My PowerBook 12inch G4 with 512MB Ram and a SuperDrive (DVD/RW) came within 10 days of placing the order. After working with heavy 15inch laptops for more than 2 years, I had gained a lot of respect for thin, light laptops.


Low Screen Resolution or 12inch

But I wasn't prepared for the 1024x768 resolution 12inch offered me. Programming or doing multi tasking is very difficult with a small screen. And after being spoit by using a Dual-Head 19inch monitors at work, a tiny 1024x768 can take some time to get used to. I wish they had something with higher resolution.


Asthetics


I don't own a red car, and I don't wear Red Flashy cloths. But there is something different about a PowerBook with a fruity laptop lid which lights up when you are working on it, showing everyone around you that you own a mac. Apple has a reputation of designing excellent functional yet beautiful hardware.


Compact

This 12inch laptop came with everything I needed for my work/home (except iPod) including DVDrw Drive and 802.11g built in which meant I don't have to carry my heavy external 2 year old DVDrw drive anymore.


Drivers

The Powerbook came with Panther installed, for which I found all the drivers I wanted, including Checkpoint SecureClient for Mac which was important for me if I planned to use PowerBook as my only laptop at home. As luck would have it, though, Tiger broke the driver and I'm still waiting for Checkpoint to release the new one.

I also found drivers for my Printer, but I couldn't get it to work with my Canon Scanner. The Help page on Powerbook suggests that I should buy hardware which they recommend on their website to maintain compability. Had I known that 3 years ago, I probably would have bought a different scanner, but for now I'll have to use my other laptop to scan photographs.


Unix Back-end

I'll jump into this in detail a little later. But I had to mention that I was impressed with the suite of tools which a Mac OS X comes with by default. Based on my first impressions OS X had all the important tools which I take for granted in all Unix servers, including utilities like "top" which doesn't even come with Solaris. The SSH client, rsync client, apache server (with perl/php), nfs support and a pretty good Terminal window blows away Window+Cygwin from miles away.


Office Productivity Tools

The powerbook came with a 30 day trial edition of Microsoft Office which did everything which I could on a regular Microsoft Office 2003 on Wintel hardware. I know there are some OpenOffice ports to Mac, but I haven't got to the point of testing that yet. PGP and MSIE were other tools which I needed to use in my line of work to keep in touch with others in organization.. and both of them worked just fine.


Plethora of features

Inspite of extremely strong unix foundation, Mac OS X continues to be designed with the end user in mind. Features like "Expose" which allows users to select an active window to switch to using thumbnails of current open windows helps solve a problem which I always have on a system which has been running for a while. Doing multi-tasking is important for some of us, and Expose goes a long way in helping me speedup application switching.


Open APIs

Last but not the least, I have to mention that iTunes is rock solid on OS X. Which was no surprise. But iTunes is not where iLife ends. iLife is a suite of products which helps you create and manage your collection of digital memories. The stability of iTunes had helped me make up my mind to try out iPhoto. With a collection of neatly cataloged 7000 photographs (using perl scripts and mysql database) I was looking for a newer tool to manage the collection. iPhoto, though a little slow and bloated, looked stable enough to trust my photographs to. With a little bit of hacking using Mac::Glue and Applescript, importing metadata was simpler than what I originally expected. As long as apple keeps its APIs open, I have have no problem trusting it with my gems. I'll talk a little more about Apples APIs later in this article


One button mouse

As windows users switch to more and more buttons on their mouse, somehow Apple thinks that "one" is still the golden number. This probably does give a consistent user experience across all Mac OS deployments, but to me it was just annoying. The convoluted way of pressing "Ctrl" while pressing "mouse-click" required me to use two hands which is doesn't need to be as complicated. [ BTW, I've heard u can connect 2 button mouse... but seriously, who buys a 12" laptop and then uses a 5" mouse along with it ? ]


Keyboard shortcuts

For users who haven't used Mac, this has got to be one of the nastiest things to get used to. the Mac OS X keyboard shortcuts took me a couple of days to get used to, and it took me another few days before I discovered a few more things. This however is pretty much the complete list of documented Mac shortcuts.


Long lasting batteries

I haven't done any empirical tests, but overall I found the batteries to last much more than my 15" Dell Inspiron. Without wireless, it seems like it can last for upto 4 to 5 hours. I'll update this section when I find this out for sure.


No hibernation : Just Instant on

Powering off an Apple Powerbook is not as common as you would think. Unlike Windows OS, Powerbooks don't hibernate, they go into extremely deep standby sleep mode and wake up instantly at the press of a key. This gives an "Instant On" feel which is definitely better than waiting for ever for all your heavy applications to start up.


Application Binaries are actually Application Package Directories

Another interesting observation was the fact that unlike most Windows applications, the Mac OS X applications didn't have all the resource files embedded within the application binary itself. The application which is listed as "Mail.app" or "iCal.app" is not actually an application binary, but a directory on the file system. Finder application which launches applications hides the fact that its a directory which can be browsed. There are two ways you can see the actual contents of which one is click on "Show Package Contents" in the Finder menu. The other way to browse is to use the terminal to cd into the directory. Do a "find /Application/Main.app" on your Mac OS X to see the files inside.


    Royans-Tharakans-Computer:/Applications/Mail.app/Contents rkt$ ls -la
    total 32
    drwxrwxr-x 8 root admin 272 May 16 20:20 .
    drwxrwxr-x 3 root admin 102 May 16 20:20 ..
    -rw-rw-r-- 1 root admin 5014 May 3 20:16 Info.plist
    drwxrwxr-x 3 root admin 102 May 16 20:20 MacOS
    -rw-rw-r-- 1 root admin 8 Mar 22 02:22 PkgInfo
    drwxrwxr-x 3 root admin 102 Mar 22 02:26 PlugIns
    drwxrwxr-x 343 root admin 11662 May 9 21:11 Resources
    -rw-rw-r-- 1 root admin 460 May 3 20:19 version.plist



2. file system performance
3. packaging adding removing packages by hand


Spotlight

If you know google desktop search then you probably know what I'm talking about already. Tiger was perhaps the first OS ever released which has such a good indexing and search capability built into the OS itself. I found the feature very helpful, but it could have been better if it support "Microsoft Entourage" as well... but hey, nothing is perfect.



Unix Tools



Shells



Mac OS X comes with a number of shells for the end user including bash, tcsh, zsh, korn, Perl, PHP ,Ruby, Tcl/Tk and Python.


Tools


The complete list of tools available in default PATH in bash is attached as Appendix I. This list of more than 1000 commands is very impressive and powerful enough to entice a hardcore Solaris enthusiast. Not only does the PowerBook come with all the bells and whistles attached, unlike some operating systems its actually in Path so you don't have to do a "find / -name $tool" to look for the tool.


Directory Structure



Root Directory (/)



    drwxrwxr-t    2 root  admin       68 Jan  9  2004 cores
    d-wx-wx-wt 2 root admin 68 Jan 13 10:41 .Trashes
    -rw------- 1 root admin 589824 May 3 15:00 .hotfiles.btree
    -rw-r--r-- 1 root admin 0 May 3 17:55 User Guides And Information
    drwxr-xr-x 4 root admin 136 May 3 17:56 automount
    drwxr-xr-x 11 root wheel 374 May 5 23:45 usr
    lrwxr-xr-x 1 root admin 11 May 9 21:01 etc -> private/etc
    drwxr-xr-x 40 root wheel 1360 May 9 21:01 bin
    lrwxr-xr-x 1 root admin 11 May 9 21:04 tmp -> private/tmp
    lrwxr-xr-x 1 root admin 11 May 9 21:05 var -> private/var
    drwxr-xr-x 63 root wheel 2142 May 9 21:08 sbin
    drwxrwxr-t 6 root admin 204 May 9 21:08 Users
    drwxr-xr-x 4 root wheel 136 May 9 21:08 System
    drw------- 7 root admin 238 May 9 21:34 .Spotlight-V100
    -rw-r--r-- 1 root wheel 4313056 May 10 18:16 mach_kernel
    drwxrwxr-x 47 root admin 1598 May 12 19:53 Library
    drwxrwxr-x 2 root admin 68 May 13 12:02 opt
    drwxrwxr-x 15 root admin 510 May 14 07:41 Developer
    drwxrwxr-x 48 root admin 1632 May 27 19:53 Applications
    dr-xr-xr-x 2 root wheel 512 May 28 19:51 dev
    dr-xr-xr-x 2 root wheel 96 May 28 19:51 .vol
    drwxr-xr-x 6 root wheel 204 May 28 19:51 private
    -r--r--r-- 1 root admin 597984 May 28 19:51 mach.sym
    lrwxr-xr-x 1 root admin 9 May 28 19:51 mach -> /mach.sym
    drwxrwxr-t 31 root admin 1156 May 28 19:51 ..
    drwxrwxr-t 31 root admin 1156 May 28 19:51 .
    drwxr-xr-x 1 root wheel 512 May 29 23:10 Network
    -rw-r--r-- 1 root admin 321698 May 30 00:34 Desktop DF
    -rw-r--r-- 1 root admin 61952 May 30 12:41 Desktop DB
    -rw-rw-r-- 1 root admin 12292 May 30 15:22 .DS_Store
    drwxrwxrwt 5 root admin 170 May 30 15:37 Volumes





Home Directory (/Users/rkt)



    drwxr-xr-x   24 rkt   rkt      816 May 30 16:07 .
    drwxrwxr-t 6 root admin 204 May 9 21:08 ..
    -rw-r--r-- 1 rkt rkt 3 May 3 17:55 .CFUserTextEncoding
    -rw-r--r-- 1 rkt rkt 15364 May 30 16:13 .DS_Store
    drwx------ 14 rkt rkt 476 May 30 15:20 .Trash
    -rw------- 1 rkt rkt 11578 May 30 13:50 .bash_history
    drwx------ 5 rkt rkt 170 May 10 18:58 .gnupg
    drwxr-xr-x 15 rkt rkt 510 May 30 12:45 .jedit
    drwxr-xr-x 3 rkt rkt 102 May 20 21:17 .netbeans
    drwx------ 3 rkt rkt 102 May 5 22:45 .ssh
    -rw------- 1 rkt rkt 8109 May 30 16:07 .viminfo
    drwx------ 5 rkt rkt 170 May 30 00:14 Desktop
    drwx------ 13 rkt rkt 442 May 13 14:58 Documents
    drwx------ 39 rkt rkt 1326 May 22 16:14 Library
    drwxr-xr-x 6 rkt rkt 204 May 13 20:31 Magazines
    drwx------ 42 rkt rkt 1428 May 28 19:29 Movies
    drwx------ 6 rkt rkt 204 May 3 23:44 Music
    drwx------ 5 rkt rkt 170 May 5 21:44 Pictures
    drwxr-xr-x 5 rkt rkt 170 May 5 07:54 Public
    drwxr-xr-x 6 rkt rkt 204 May 5 07:54 Sites
    drwxrwxrwx 4 rkt rkt 136 May 7 00:16 System Volume Information





/etc/group



    ##
    # Group Database
    #
    # Note that this file is consulted when the system is running in single-user
    # mode. At other times this information is handled by lookupd. By default,
    # lookupd gets information from NetInfo, so this file will not be consulted
    # unless you have changed lookupd's configuration.
    ##
    nobody:*:-2:
    nogroup:*:-1:
    wheel:*:0:root
    daemon:*:1:root
    kmem:*:2:root
    sys:*:3:root
    tty:*:4:root
    operator:*:5:root
    mail:*:6:
    bin:*:7:
    staff:*:20:root
    lp:*:26:
    postfix:*:27:
    postdrop:*:28:
    certusers:*:29:root,jabber,postfix,cyrusimap
    utmp:*:45:
    uucp:*:66:
    dialer:*:68:
    network:*:69:
    www:*:70:
    mysql:*:74:
    sshd:*:75:
    qtss:*:76:
    mailman:*:78:
    appserverusr:*:79:
    admin:*:80:root
    appserveradm:*:81:
    clamav:*:82:
    amavisd:*:83:
    jabber:*:84:
    xgridcontroller:*:85:
    xgridagent:*:86:
    appowner:*:87:
    windowserver:*:88:
    accessibility:*:90:
    tokend:*:91:
    securityagent:*:92:
    unknown:*:99:





Files under /etc directory





FileSystem Domains



User Domain

Files under this domain are under full control of the user who is logged in. (For example in my case it was under /Users/rkt).
Every user account under Mac OS X comes with a standard set of directory under it which are "Applications", "Desktop", "Documents", "Library", "Movies", "Music", "Pictures", "Public", "Sites". Some of these directories are similar to the ones found under Microsoft Windows user directories.
Local Domain

Non-System Files shared among all users on the Operating System
Network Domain

Shared among all users on a Network File System
System Domain

Core OS files which the users can't modify. /etc directory is one example of the directories under this Domain.




Open Source, APIs and Documentation



Darwin, BSD and Open source


Apple's Mac OS X Unix page describes the rock solid Unix core on which it is build upon.
Here are a few more links you should have a look at to understand more about how much of Mac OS X comes from open source.





May 13, 2005

Switching to powerbook

Two weeks ago I did some thing which I didn't think I'll ever do. I'd always thought Apple/Mac were worse than Microsoft because they were not open enough. I thought, that if they were open, there would have been atleast 10 times more Apple users as there are today. Since I wasn't ready to drop $$$ dollars down the drain to buy a sleek little Mac laptop I never got around actually check it out.

Well, two weeks back after I lost a laptop in a disaster, I made a choice to switch based on few logical reasoning

  • I still believe in open source

  • Microsoft is worse than Linux

  • But Linux is not fully supported on laptops I like

  • I also have problem getting support because there are 2000 different distributions out there now

  • I've grown out of compiling/optimizing Linux kernel every day

  • I need access to realworld office application which my organization uses (MS Office, PGP, Checkpoint VPN

  • I know Mac can do most of these...

  • OS X has a BSD backend

  • Should be fun hacking something new for a change

  • powerbooks look sexy...

  • And I badly needed a sleek very mobile 12inch laptop which would cost me big bucks anyway... so might as well pay for a better designed one instead of going with cheaper dell alternative which seems to break everyone and then

Tape library yesterday and tommorow

BackupTape technology pretty much kept up with the Moore’s Law. Here's the history and future of current tape technology which I compiled from various sources













































































































































































































































































































TypeFormatVendorGB/HrGB/Hr compressedStorage CompressedStorageDate
TravanTraven 20Certance3.572010
TravanTraven 40Certance7144020
AITAIT-1HP/Sony143791351996
AITAIT-2HP/Sony1643130501998
AITAIT-3HP/Sony33862601002000
AITAIT-4HP/Sony200TBD *
AITAIT-5HP/Sony400TBD *
AITAIT-6HP/Sony8002008 *
DATDDS-4Certance/HP/Sony10204020
DATDAT72Certance/HP/Sony12.625.27236
DLTvs80/160DLT vs80HP/Quantum11228040
DLTvs80/160DLT vs160HP/Quantum295816080
DLT 1DLT 1HP/Quantum10208040
DLT 4000DLT 4000HP/Quantum5104020
DLT 7000DLT 7000HP/Quantum18267035
DLT 8000DLT 8000HP/Quantum22448040
Super DLTSDLT220HP/Quantum3978220110
Super DLTSDLT320HP/Quantum541083201602002
Super DLTSDLT600HP/Quantum1302606003002004
Super DLTSDLT1200HP/Quantum6002005 *
Super DLTSDLT2400HP/Quantum12002007 *
LTO/UltriumLTO-1HP/Certance541082001002000
LTO/UltriumLTO-2HP/Certance1082164002002003
LTO/UltriumLTO-3HP/Certance4002005 *
LTO/UltriumLTO-4HP/Certance800TBD *
Super AITS-AITSony10818013005002003
Super AITS-AIT2Sony10002005 *
Super AITS-AIT3Sony20002007 *
Super AITS-AIT4Sony40002009 *