RSS
 

Archive for the ‘unix’ Category

The Java+linux OS

27 Nov

This will be an interesting trend to follow. This linux+perl distribution is made up of just linux kernel and perl binaries. Rest of the tools are all written in perl shell scripts. Miguel de Icaza, the creator of mono is looking for folks to do the same with mono.

I think its a great experiment and will help validate mono as a practical alternative to other frameworks/languages on linux. But what will be even cooler (for me atleast) is if someone can create a true Object oriented shell experience like Microsoft’s powershell/monad. And incase you didn’t know, Powershell/Monad is the new shell by Microsoft using .net framework. It will probably replace cmd sometime in the future.
That being said, it doesn’t really have to be mono. Java is a perfect candidate for it as well. There was a java project related to a java based shell which I don’t think is active anymore… may be someone can revive it.

Can it be done ?

 
Comments Off

Posted in interesting, java, linux, unix

 

Linux initrd (initial RAM disk) overview

04 Aug

Initrd is one of those things in linux which most of us have taken for granted. Here is a very interesting writeup on how initrd really works. “The Linux® initial RAM disk (initrd) is a temporary root file system that is mounted during system boot to support the two-state boot process. The initrd contains various executables and drivers that permit the real root file system to be mounted, after which the initrd RAM disk is unmounted and its memory freed. In many embedded Linux systems, the initrd is the final root file system. This article explores the initial RAM disk for Linux 2.6, including its creation and use in the Linux kernel.

 
Comments Off

Posted in unix

 

Sysadmin Day

28 Jul

Pat yourselves on your back for fixing all those servers,
- doing backup,recovery and user creation.
Pat yourselves for saying no to root and yes to sudo,
- for writing ACLs and scripting voodoo…

Pat again for waking at 2am
- just to put your cellphone on charge.
..for dealing with people
- who wanted everything a day past

Pat again for reading 650 mails a day.
- for blocking SYNFIN floods on ur network
..for carrying those secure-ids
- even while you are not at work.

When you are done patting… please stop by a bar
- pick your pagers and throw away..
’cause you all need a break once in a while
- atleast on the feaking System Admin Day !!

 
Comments Off

Posted in blogging, unix

 

Could the Google and Sun rumor be about Java ?

18 Mar

If you have been following writings from Daniel M Harrison you would notice how strong his convictions are on this topic. Daniel strongly believes that Google is buying Sun. And any reader who doesn’t understand how Google and Sun operate can easily be swayed to believe this. But not me.

The fact that Google or Sun haven’t publicly denied these new rumors, means that something might be cooking. But Google buying Sun doesn’t sound very interesting.

  • Sun has a large pool of talent who know how to create fault tolerant, high performance parallel processing computing infrastructure.
  • Google has a large pool of talent who have perfected the art of distributed computing using cheap hardware clusters using free tools and operating systems
  • Sun is a hardware, software and services company
  • Google makes its revenue from advertisements
  • If Google buys Sun, it would be forced to use Sun technology. Microsoft had a hard time switching Hotmail.com from FreeBSD to Microsoft based solutions.
  • The change for Google to switch to Sun based hardware and software and the time spent to do it could be quite significant.
  • A lot of goodwill for Google stems from the fact that Google is Open source friendly. Even though Sun has made attempts to open its Operating System, the perception is not the same. Google might have to face some negative publicity if they don’t take immediate damage control initiatives after a buy out.

If there is any truth to these rumors, its more likely that its about Java than anything else.

  • Google already has an agreement with Sun over cross distribution of Java and Google desktop.
  • Based on what I know, its more likely that Google might buy out Sun’s Java technology than buying the whole company itself.
  • Java is one platform which is truely write-once-run-everywhere. Nothing else comes closer to this reality.
  • Google desktop has made significant inroads into desktop world running Microsoft OS. But lacks critical foothold in non-Microsoft world. This could change if it switches to Java as the application platform for all of its client side applications
  • If Google plans to quickly build applications like GDrive and integrate writely.com with other applications running on the local operating system, it would need a more universal platform. Java, though slow, is still faster than javascript and has more access to the operating system to do such tasks.
  • With better control over how Java develops, Google could use its strong technical background to speed it up and customize it for its own applications. The way Microsoft is trying to use .NET to spread its word.
  • This may or may not be a good thing for Java. But will definitely be a awesome add on for Google.
 
Comments Off

Posted in google, java, unix

 

Rsync for incremental backups

26 Aug

Introduction

Archiving files to tape is still considered one of the cheapest way of making backups. However with the prices of disk storage and solid-state storage decreasing rapidly, it won’t be long before users make the switch to the faster disk storage for all thier backup needs. The problem, however, is that if you want to do anything more than mirroring data on a remote storage, there aren’t too many good freeware tools to do it. This writeup explains one of the interesting ways to do incremental backups with snapshot cabability using a popular tool called rsync.

Traditional Backup

Traditional backup applications not only support backing up and restoring of files, directories, partitions and drives, but also allow incrementally backups to reduce time taken to backup large file systems. Since its not practical to restore the complete data from tape just to apply minor changes to it, most tape backup software store the differences in a seperate file or location on tape which can be used to patch the last full backup image on the tape. This feature of storing incremental updates also allows administrators to maintain multiple versions of data without keeping as many physical copies of data. Incremental updates to backup repository is extremly valuable feature in highly dynamic environments where maintaining multiple snapshot of data taken very frequently is important.

Rsync

Rsync was one of the first tools I used, which allows one to update copies of data by sending incremental updates. This dramatically cuts down the time to update a copy of data. The problem is that though rsync allows you to make a copy of data, and allows you to incrementally update it, it was not designed with tape in mind. It specifically doesn’t allow you to keep the incremental updates in a different directory or file the way backup applications do. This limits the number of snapshots one can maintain using rsync.

The cp command

And that brings us to the last part of the puzzle which we need to know to make do incremental backups. The “cp” command on most unix operating systems allows copying of hard links (instead of the actual data). This feature allows you to maintain two physically different directories on the same partition (with different names) pointing to the same physical set of files. In linux this can be accomplised by the following command “cp -al $sourcedir $targetdir”.

Rsync + cp

To demonstrate how these two can work together to provide us snapshot capability I did some tests on my linux box. The first step was to create a small directory structure which we would be using for this exercise. “ls -ila” on the directory shows the actual “inodes” (column 1) assigned to each of these files and directories within the test directory I created.

Original Directory Structure

    List of files
    ==============================
    /tmp/test/primary
    /tmp/test/primary/file1.txt
    /tmp/test/primary/file2.txt
    /tmp/test/primary/subdir1/file3.txt
    ==============================
    
    la:/tmp/test # ls -ila primary/*
    358315 -rw-r--r--  1 root root  5 Aug 21 23:06 primary/file1.txt
    358316 -rw-r--r--  1 root root  5 Aug 21 23:07 primary/file2.txt
    primary/subdir1:
    total 4
    358313 drwxr-xr-x  2 root root  80 Aug 21 23:07 .
    358312 drwxr-xr-x  3 root root 136 Aug 21 23:07 ..
    358314 -rw-r--r--  1 root root   5 Aug 21 23:07 file3.txt
Next step is to do a traditional recursive copy from “primary” to “directory1″. You can accomplish this by either “cp” or “rsync”. In this following example I used “cp” command. Notice that when I list the inodes after the cp commands, it creates a new set of inodes for each of the files and directory in the new directory structure. This means that the traditional “cp” command did an actual recursive copy of file contents to new locations, and that there exists two identical copies of each of the objects.

After Copy using “cp -rp src target”

    la:/tmp/test # cp -rp primary directory1
    la:/tmp/test # ls -ila directory1/*
    358307 -rw-r--r--  2 root root  5 Aug 21 23:06 directory1/file1.txt
    358308 -rw-r--r--  2 root root  5 Aug 21 23:07 directory1/file2.txt
    directory1/subdir1:
    total 4
    358305 drwxr-xr-x  2 root root  80 Aug 21 23:07 .
    115666 drwxr-xr-x  3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r--  2 root root   5 Aug 21 23:07 file3.txt
Now lets see how “cp” behaves when we ask it to preserve hardlinks. In this example we are copying “directory1″ into a new directory “directory2″. Notice how the inodes in the new directory are same as the ones from “directory1″. This means that though there are two logical directories which look alike, the actual file and directories listed within each one of them are identical. Any modification done to one file within one directory (without modifying the inode) will affect the file in the other directory. This is almost same as symbolic linking, except that unlike symbolic links the file wont dissapear from “directory2″ if I delete it from “directory1″. In other words there is actually multiple owners of these inodes at this moment, which seems a little hard to digest.

“cp -la src target”

    la:/tmp/test # cp -al directory1 directory2
    la:/tmp/test # ls -ila directory2/*
    358307 -rw-r--r--  2 root root  5 Aug 21 23:06 directory2/file1.txt
    358308 -rw-r--r--  2 root root  5 Aug 21 23:07 directory2/file2.txt
    directory2/subdir1:
    total 4
    358310 drwxr-xr-x  2 root root  80 Aug 21 23:07 .
    358309 drwxr-xr-x  3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r--  2 root root   5 Aug 21 23:07 file3.txt
So we know how interesting hard links are and we know how to create multiple directories look exactly the same without creating as many copies of the actual data. A little more research on your part would reviel that if you had modified “subdir1/file3.txt” the only two inodes which would change are “subdir1″ and “subdir1/file3.txt”. I didn’t show inodes of “primary” directory in the dumps below, but what I did do is show you how the inodes look like after I rsync the changes from “primary” to “directory1″.Notice that after rsync to “directory1″ the inodes for “subdir1″ and “subdir1/file3.txt” has changed (as expected). This is because rsync usually doesn’t overwrite existing inodes. Instead it creates fresh copies of updated files and directories and deletes the old ones. Interestingly inodes of “directory2″ still shows the old inodes for the files/directories which were modified.

“directory2″ has now become a “snapshot” of “directory1″ without actually having a duplicate copy of all the data in “directory1″.

Modified file3.txt in primary copy”

    la:/tmp/test # rsync -rvgoutl primary/* directory1/
    building file list ... done
    subdir1/
    subdir1/file3.txt
    wrote 161 bytes  read 40 bytes  402.00 bytes/sec
    total size is 21  speedup is 0.10
    
    la:/tmp/test # ls -ila directory1/*
    358307 -rw-r--r--  2 root root  5 Aug 21 23:06 directory1/file1.txt
    358308 -rw-r--r--  2 root root  5 Aug 21 23:07 directory1/file2.txt
    directory1/subdir1:
    total 4
    358305 drwxr-xr-x  2 root root  80 Aug 21 23:14 .
    115666 drwxr-xr-x  3 root root 136 Aug 21 23:07 ..
    358322 -rw-r--r--  1 root root  11 Aug 21 23:14 file3.txt
    
    la:/tmp/test # ls -ila directory2/*
    358307 -rw-r--r--  2 root root  5 Aug 21 23:06 directory2/file1.txt
    358308 -rw-r--r--  2 root root  5 Aug 21 23:07 directory2/file2.txt
    directory2/subdir1:
    total 4
    358310 drwxr-xr-x  2 root root  80 Aug 21 23:07 .
    358309 drwxr-xr-x  3 root root 136 Aug 21 23:07 ..
    358306 -rw-r--r--  1 root root   5 Aug 21 23:07 file3.txt
 
Comments Off

Posted in unix

 

Tape library yesterday and tommorow

13 May

BackupTape technology pretty much kept up with the Moore’s Law. Here’s the history and future of current tape technology which I compiled from various sources

Type Format Vendor GB/Hr GB/Hr compressed Storage Compressed Storage Date
Travan Traven 20 Certance 3.5 7 20 10
Travan Traven 40 Certance 7 14 40 20
AIT AIT-1 HP/Sony 14 37 91 35 1996
AIT AIT-2 HP/Sony 16 43 130 50 1998
AIT AIT-3 HP/Sony 33 86 260 100 2000
AIT AIT-4 HP/Sony 200 TBD *
AIT AIT-5 HP/Sony 400 TBD *
AIT AIT-6 HP/Sony 800 2008 *
DAT DDS-4 Certance/HP/Sony 10 20 40 20
DAT DAT72 Certance/HP/Sony 12.6 25.2 72 36
DLTvs80/160 DLT vs80 HP/Quantum 11 22 80 40
DLTvs80/160 DLT vs160 HP/Quantum 29 58 160 80
DLT 1 DLT 1 HP/Quantum 10 20 80 40
DLT 4000 DLT 4000 HP/Quantum 5 10 40 20
DLT 7000 DLT 7000 HP/Quantum 18 26 70 35
DLT 8000 DLT 8000 HP/Quantum 22 44 80 40
Super DLT SDLT220 HP/Quantum 39 78 220 110
Super DLT SDLT320 HP/Quantum 54 108 320 160 2002
Super DLT SDLT600 HP/Quantum 130 260 600 300 2004
Super DLT SDLT1200 HP/Quantum 600 2005 *
Super DLT SDLT2400 HP/Quantum 1200 2007 *
LTO/Ultrium LTO-1 HP/Certance 54 108 200 100 2000
LTO/Ultrium LTO-2 HP/Certance 108 216 400 200 2003
LTO/Ultrium LTO-3 HP/Certance 400 2005 *
LTO/Ultrium LTO-4 HP/Certance 800 TBD *
Super AIT S-AIT Sony 108 180 1300 500 2003
Super AIT S-AIT2 Sony 1000 2005 *
Super AIT S-AIT3 Sony 2000 2007 *
Super AIT S-AIT4 Sony 4000 2009 *
 
Comments Off

Posted in unix

 

Unix file system notes

29 Oct
UFS unlike what most think was not developed by Sun Microsystems. Instead it was the brainchild of UNIX at ATT. Oh and BTW UFS stands for “Unix File System”. Over the years its been used by many different Unix Distributions including Solaris, NetBSD, OpenBSD, and even MacOS. Linux supports it, but in a limited way.

References
Articles on General Performance
http://search.itworld.com/query.html?select1=&qt=ufs

File Systems
Why we should use UFS logging
VxFS vx UFS : VxFS is better
UFS and VxFS : VxFS is better
UFS vs VxFS UFS is comparable if not better
Benchmarking Filesystems It thinks JFS, ReiserFS or XFS is better than EXT3.
Journaling file system
Understanding Journaling File system Part I
Part II
Journaling File System in linux
Veritas for Linux
Compare VxFS with EXT3 and Reiser

Oracle related
Optimizing Oracle 10g on Linux using Automated Storage Management
Storage on Automatic

Volume Manager
VxVM vs DiskSuite

 
Comments Off

Posted in unix

 

A secure NFS environment?

29 Dec

A lot of organizations do not realise the danger of NFS untill they have been intruded by hostile crackers. This article would give a short description of most NFS realted problems and means to avoid it. Since I mostly use solaris, I’ll try to stick to Solaris examples in this paper.

Problems: Un-authenticated NFS mounts.

Many sys-admins including me, have setup uncontrolled NFS shares on solaris boxes. There might be many excuses for this. My popular excuse is that I was just testing it, or that I was asked to do that by someone else. No matter what the excuse is, its tough to recover from a hostile attack morally if its ever misused.

As a matter of policy shares should have restricted hosts, especially if it has read-write enabled. No NFS mounts should be allowed from hosts which are accessable from the Internet, and one should avoid critical write enabled NFS mounts in a non-secure zone.

Problems: home directories

Its is a popular to use NFS for home directories, especially for developer environment where no one likes to update profiles all over the network. Most of the environments I’ve worked with had NFS setup. In such a network, the NFS directories are only as secure as the weakest machine on the network. Its usually a good practice in such a senario to avoid “root” access to NFS.

Even if you think you can recover damage to the NFS directories using backups, you would have a difficult time if the cracker misused “r” commands and reaches other servers on the network. Even if a user has different password on each and every system on the network, the NFS home directories can effectively give a cracker access to the entire network if he sets up a .rhosts file. Ive noticed that by killing Inetd and setting up ssh make some Admins feel a little more secure. However, unfortunately ssh allows exactly the same set of accessability which a “r”command does. The only difference here is that the execution is secure to sniffing by corporate sniffers. Which in other words is more dangerous.

Problems: Trusted servers on NFS ?

Personally I think any machine on a NFS should be considered open to attack in the greatest degree. If you really want to build a secure trusted server for remote management, the first think you should do is shut down inetd and NFS completely. This is again for hte same reason as I explained above.

Problems: Suid on NFS ?

Well now that you know how NFS is insecure, its logical to conclude that if suid on NFS is changed on one machine, it will effect on all systems which run it. Hence, avoid suid if possible. Implement it on local drive. Run away from your manager and try to act as if you didn’t hear it if he proposes to enable SUID on nfs.

Problems: Dont forget Automounts.

/etc/dfs/dfstab is not the only place you have to be carefull, check your automounts. if you use NIS+, you can centrally push more secure configs to all your NFS clients.

Problems: FDQN please…

I work in an environment which has multiple domains with multiple seach domains listed on /etc/resolv.conf. It would ,hence, be prudent if you try to use only FDQN (Fully qualified Domain names).

Problems: netgroups

I’ve heard some horror stories with netgroups. The biggest I think is that solaris exports the directory to the entire world if someone missplet a netgroup. Thats a real horror story.

Other improvements: secure RPC Solaris allows Secure RPC communication which can make NFS a little more secure. Linux does support it too ( I think ).

http://www.cco.caltech.edu/~refguide/sheets/nfs-security.html#intro

http://www.lanl.gov/projects/ia/stds/ia7a01.html

 
Comments Off

Posted in security, unix

 

Linux in 2000

01 Jan

The other day, someone asked me what my worst nightmare was. I told him, “Linux world domination.” Surprising as it may seem coming from a Linux advocate, the fact is that this issue is being debated in Linux circles the world over. With the absence of competition linux may not have much to look forward to.

However since that will take a long time to happen, we look at the past how linux succeeded as it is today and where its headed towards in the near future.

Battling Since Birth

Linux was born in the midst of the Minix generation of Intel 286s. Minix, which did not allow free distribution of code, was the de facto OS for university curricula back then (1991). After winning over Minix, Linux fought the software crunch of the early 90s and integrated many common Unix tools ported over in the first quarter of this decade. And just before it stood against Microsoft in the late 90s to fight for a pie in the desktop segment, Linux fought hard to get X-Windows applications ported to it. And today, as I look back at the decade gone by, Linux is fighting its toughest battle yet-the people. In short, Linux can succeed only as much as people can accept it.

Two years ago I wasn’t so sure if Linux would ever gain commercial acceptance. Linux was always developed by and for a niche group of developers, and commercialization was the last thing on their mind. From the day it was created by Linus Torvalds, it was intended for education and research purposes only, with its source code free for all to see. Until the end of the millennium. The first signs of acceptance became visible in 1998, seven years after its birth, when the corporate world stepped in. Though still not considered a desktop environment, it was put to test by the likes of Sun and SGI for stability issues and came through with flying colors. The rest, as they say, is history. The Internet, as we know it today, was soon running on various versions of Linux. Apart from companies like Red Hat who centered their business around Linux, others like Netscape, Oracle, Creative, Corel, and Novell also stepped in. 1999 saw an end to the lack of commercial support, which had long plagued Linux as an acceptance criterion for larger corporations.

1999 was also the year when the Linux community made a unified effort to educate the press about how money could be made from a free OS. The press had been largely ignorant of the growth of Linux, because it was always looked at as a free OS, which did not have any business viability. The most notable development on this front was the emergence of Linux advocacy guidelines, which are today being followed by most organizations that support Linux.

The year would be particularly remembered for the Microsoft versus DOJ case, which boosted Linux in its own way. With a view to show that Microsoft was not a monopoly, Microsoft conducted a business analysis of Linux within its own closed doors. Now known as the Halloween I and II documents (www.opensource. org/halloween/), the revelations of these tests were a huge draw for the press. The document sparked a full-blown debate, so much so that Linux came to be talked about by the masses as well. It was apparent after that debate that Linux was in fact being considered as a direct opponent to Microsoft’s Windows NT. IBM, SGI, Sun, and SCO were the next to join the Linux bandwagon. Red Hat’s successful IPO in August ‘99 redefined the business viability of Linux. While things went quite well for Linux, it did have its share of ups and downs. The initial success of Linux blinded many Linux supporters to the OS’s shortcomings. The comparative tests of Windows and Linux conducted by PC Week magazine and Mindcraft, an independent organization, produced results that the Linux community didn’t want to hear. Notwithstanding the doubts raised over the authenticity of these tests, it became quite apparent in successive tests that Linux did need optimization of a few core components, which were way too slow to stand competition from Windows NT. Linus Torvalds himself was not pleased with the results, and since then a lot of work has gone into optimizing the code.

Linux (Em)bedded

Another area where optimized code could help Linux is that of Linux-embedded devices. Over the past few months, global focus has been steadily shifting from “one computer in every home” to “computers on the go.” Cellular phones, MP3 players, and other handheld devices that talk to web servers already exist, as do handheld devices that organize your daily to-do lists. What we haven’t seen too much of yet are watches, refrigerators, and microwave ovens with OSes built into them. Some of these are already happening, but some are waiting for the right OS. Not only is the open source mechanism of Linux ideal for manufacturers to use as a building block for such intelligent devices, it would also be economically disastrous to start work on a new OS from scratch. Cobalt, in 1997, was one of the first companies that started using Linux for its appliances (cache engines and web servers). A few others have since come up; some of them have also developed routers based on Linux. I’m sure the new millennium would see a lot more of these Linux-based devices. 3Com’s Palm and Microsoft’s Windows CE have shown that there is a growing market for handheld devices. A number of Linux ports to smaller chips are in progress, some of which are uCsimm, ARM, and the Palm.

Penguins and Icebergs

Some of the most mind-blowing graphics in the movie Titanic were rendered using a collection of Alpha processor-based machines running Linux. Linux was used in a research project at NASA to create such parallel-processing supercomputers. Now code-named Beowulf Linux, the system has the capability to harness the processing power of multiple machines. C-DAC’s PARAM supercomputer-which used its own OS and was developed from scratch-is till date unrealistic for most Indian universities to afford. Now with the Linux Beowulf cluster ready for market adoption, research centers in India could set up much more powerful supercomputers with relatively low investment. It won’t be long when a couple of old Pentium or 486 boxes would be put together to make Bollywood version of Titanic. Moving from the server to the desktop was perhaps the biggest challenge in Linux’s history. The desktop is the playground that determines the success of an OS. The user friendliness and intuitiveness of the Linux GUI has been one of the most ignored issues for a long time. The lack of an open version of Motif, which was used on the Sun platform, was one of the reasons behind the slow start of Linux in the desktop arena. It was not until 1997 that any one could come up with an alternative and better Window Manager. KDE and Gnome, the two most popular environments today for Linux newbies, have gone a long way in enhancing Linux’s sheen. Companies like Dell, Compaq, SGI, and Gateway-which would not have associated with Linux three years back-are now supporting Linux by selling Linux preinstalled machines along with similar Windows offers. Some hardware manufacturers are afraid of jumping onto the Linux bandwagon due to certain licensing restrictions in Linux, but that too is slowly becoming a thing of the past. DVD support is an excellent example of hardware support going wrong. The licensing specifications for DVDs forbid open source completely; this has left Linux users high and dry. Creative Technologies was one of the first big organizations to actively promote open source by opening up its specifications.

Linux Everywhere

But licensing is not the reason why Unix failed in the 90s. It failed because it could not stand united against Microsoft products. Though some analysts see a similar future for Linux, GNU licensing does address some of these issues. Organizations like FSF, which backs Linux with legal support, are recognized for their contribution to popularizing Linux. Linus Trovalds, who holds the trademark “Linux,” is still the principal maintainer of the Linux kernel around which all Linux apps are built. For each critical development tool in Linux distribution, a committee works out how to get things done. Though it might be a bit slow, this democratic setup ensures that Linux doesn’t go the way Unix did. With Linux stocks (of companies like Red Hat) touching the sky, it’s anybody’s guess what Linux would be like in the new millennium. Red Hat today has enough market capitalization to buy a few companies. Cygnus, a well-known development tool builder, was one of the first companies to be acquired by RedHat. Sun Microsystems has been busy pulling in companies like StarOffice and the like. Corel Corp. has launched its own version of Linux, as you can see from the PC World CD this month. At present there are about twenty different versions of Linux with different configurations and for different environments. All running on the same kernel and other tools. It’s difficult to predict the impact of Windows 2000 on the success of Linux, or whether my wristwatch would one day run Linux. But one thing is sure to happen: We would see some radical changes in the way software is written, shared, and maintained around the world.

A version of this article was contributed to PCWorld Jan 2000 issue.

 
Comments Off

Posted in linux, unix