computing

Creating a super-pendrive

Hi,

Its been really long since I blogged. Before going ahead, want to tell you that will be posting my endeavours on running virtual machines soon.

This, however, is my experience/guide for creating a super pen drive (a.k.a. a usb disk that if booted from presents a grub menu with options – GParted, Fedora, .. and any other linux distro hopefully, and has Fedora persistence, and has a spare partition too for you to use it as a normal pen drive).

It wasn’t as straight forward as I thought (like… all other things).

Steps:

  1. Decide the partitioning layout and partition the pen drive.
  2. Install grub on one partition
  3. Install GParted on another
  4. Install Fedora on another
  5. Install grub again on the partition in step 2 :P, and setup grub.conf
  6. Sit back, and enjoy (may need to crouch forward in some cases.. )

A super neat trick:

To check at any time how you pen drive would behave if you boot from it, use the command : qemu -hda /dev/sdb -m 256 -vga std

This command reduced my research time to one third.

1) Decide the partitioning layout

I used GParted on my Fedora installation (on my harddisk) to set up the partition of the pen drive.

2) Install grub on one partition

Make sure you manually mount the grub partition. The /media/something folder in which it gets mounted automatically gave me some trouble. So unmounted it from there, and mounted /dev/sdb5 to /mnt/usbgrub

grub-install –no-floppy –root-directory=/mnt/usbgrub/ /dev/sdb

3) Install GParted on another

Use UNetbootin for GParted

4) Install Fedora on another

Need to make sure that usb disk is ext3 before doing this. When I did this on a vfat disk, I got an “error 22” when trying to boot from the usb disk.

Use liveusb-creator or live-iso-to-disk for Fedora.

Persistence is buggy according to http://forums.fedoraforum.org/archive/index.php/t-219250.html so instead, a better solution is to directly install fedora on the pen drive on a separate partition, like in http://linuxsoftwareblog.com/blog/?p=156

5) Install grub again on the partition in step 2 :P, and setup grub.conf

mount /dev/sdb5 /mnt/usbgrub/

grub-install –no-floppy –root-directory=usbgrub/ /dev/sdb

Now, to setup the grub menu:

cd /mnt/usbgrub/boot/grub/

cp /boot/grub/grub.conf .

cp /boot/grub/splash.xpm.gz .

gedit grub.conf

# grub.conf generated by anaconda
#
# Note that you do not have to rerun grub after making changes to this file
# NOTICE:  You have a /boot partition.  This means that
#          all kernel and initrd paths are relative to /boot/, eg.
#          root (hd0,1)
#          kernel /vmlinuz-version ro root=/dev/mapper/VGSahil-LVRoot
#          initrd /initrd-version.img
#boot=/dev/sda
default=0
timeout=5
splashimage=(hd0,4)/boot/grub/splash.xpm.gz
hiddenmenu
title Fedora (2.6.30.8-64.fc11.i686.PAE)
root (hd1,1)
kernel /vmlinuz-2.6.30.8-64.fc11.i686.PAE ro root=/dev/mapper/VGSahil-LVRoot rhgb quiet
initrd /initrd-2.6.30.8-64.fc11.i686.PAE.img

title GParted Live
root (hd0,5)
kernel /live/vmlinuz1 boot=live union=aufs    noswap noprompt acpi=off irqpoll noapic noapm nodma nomce nolapic nosmp ip=frommedia vga=normal
initrd /live/initrd1.img

title Omega 11 Live
root (hd0,6)
kernel /syslinux/vmlinuz0 root=/dev/sdb7  rw liveimg quiet  rhgb
initrd /syslinux/initrd0.img

Making Fedora rpms/yum work – Offline

This is guide relevant to those who wish to spread fedora to friends and loved ones who don’t necessarily always have an internet connection (or a good one atleast). The problem faced in such situations, almost always (talking from my experience), is that there are a huge number of rpms that need to be downloaded to make fedora capable of playing media, and fill it with good stuff like k3b, amarok etc.

This isn’t always possible with the skimpy internet connections that our not-so-tech-savvy aunts have. (We’ll assume that it is our aunt on whose system we need to install fedora for the sake of this guide.)

So, I devised a way of spreading fedora to our aunt’s system, without getting embarrassed  by that fact that we weren’t able to run mp3 on their system.

The way to do this, is to install a fresh copy of fedora on our system, and then bring it to perfect shape by installing many more rpms, and while doing this, keeping a copy of the rpms required, and then copying this repository of rpms (which we are sure don’t require any more rpms as dependencies as we install them on our own system in offline mode) on a pen drive and taking it along with the fedora installation media to our aunt’s home. And after installing fedora on her system, we simply install all the rpms on her system.

Steps:
On our system:
Download all rpms required for the extra packages (the package rpms + dependencies)
yumdownloader --destdir=rpmsForAunt --resolve rpmName(s)

OR

create a service pack of all pending updates or certain rpms using gpk-service-pack

yum install gnome-packagekit-extra

OR

Edit /etc/yum.conf and change the value of keepcache to 1. After the update is done, the downloaded rpm files then can be found in (and copied from) subfolders named “packages” in /var/cache/yum. When you’re done with them you can get rid of them to save disk space with yum clean packages.
On our aunt’s system:

  1. Install fedora.
  2. Install the extra downloaded rpms:
    1. You need to disable all repositories before yum localinstall will work without net access. To do so,
      go to System > Administration > Add/Remove Software and go the System > Software Sources and uncheck all sources.
    2. Installing the rpms:(1 : see footnote)
      cd rpmsForAunt
      yumlocalinstall --nogpgcheck *

      The above command is to be run for every category of rpms below after copying the resultant directories on our Aunt’s system.

For getting all updates:
I wrote a script for downloading all updates (after a fresh install) to a directory:
for i in `yum list updates | grep fc11 | cut -d ' ' -f 1`
do
echo Now downloading rpms for package $i
yumdownloader --destdir=localUpdate --resolve $i
done

For getting all media rpms:
rpm -ivh http://rpm.livna.org/livna-release.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-livna
rpm -ivh http://linuxdownload.adobe.com/adobe-release/adobe-release-i386-1.0-1.noarch.rpm
rpm --import /etc/pki/rpm-gpg/RPM-GPG-KEY-adobe-linux

(the above is required only so that you (and not your aunt) can download the rpms)
yumdownloader --destdir=localRpmsForMedia --resolve libdvdcss vlc flash-plugin xine xine-lib-extras xine-lib-extras-freeworld mplayer mplayer-gui gecko-mediaplayer mencoder amarok rhythmbox gstreamer-plugins-ugly gstreamer-plugins-bad gstreamer-ffmpeg audacious audacious-plugins-freeworld* k3b

A few more rpms that I use:
yumdownloader --destdir=localRpmsOther system-config-lvm gparted digikam m17n-db-*

(1)I faced an issue while bash was updated using this method. It said transaction failed.
To resolve this, I ran
yum-complete-transaction --clean
rpm -e bash

The above command listed two bash versions (I don’t remember the version numbers), on saying rpm -e bash.version1, it said there are many dependencies, then I tried rpm -e bash.version and it worked. Then, I went back to the yumlocalinstall step and then that worked.

Things done in college – technology

Here is a list of things I have done in the past three years. I have written for the sake of personal record.

As a member of Delta :
Created a “PC Based Oscilloscope” in IIT Bombay, as a summer project. Used java servlets on the server side and a java applet on the client side. Was responsible for the whole of the software side – 2006 Summers
Worked in Pragyan CMS V1, which finally got implemented in our college website – 2005-2007
Made Dalal Street, a stock market simulator using java servlets on the server side and using eclipse to make a java based ui compiled using gcj to eliminate the need of jvm to run the final executable – 2006 Dec – 2007 Jan
Used CVS for the development of Dalal Street, understood the importance of a code versioning system.

As being a part of Delta Core (Technology Changes) :
Implemented LDAP using openldap, in Delta, allowing everyone to have a central authentication server, with a common login everywhere, where everywhere includes :
system login in Sun Lab comps
Implemented NFS on Delta, which gets mounted on all Sun Lab comps, using the default nfs service provided by default on fedora, so everyone has the same home irrespective of the comp they login to, which they do through their ldap accounts.
Implemented pure-ftpd on Delta, configured it to work through ldap, allowing everyone to access their home drives even from “outside” (the user labs).
Setup, and advocated use of Doku for information keeping, made it work through LDAP.
Implemented and introduced SVN on Delta, setup three repositories : pragyan, delta and dalal, delta for the use of all delta projects.
Implemented and introduced trac on delta, setup three repositories : pragyan, delta and dalal. Customized all of these three. Learned how to customize through .egg files.
Made svn and trac work through httpd authentication, which used LDAP to get authentication details. (this was hell)
Revived delta as a student group – meaning, made sure many meetings were held, made sure everyone knew each other, everyone contributed something to delta and felt a part of the group, made sure many treats were held, and chucked a few inactive members out of delta.
Created Pragyan CMS V2, from scratch.

As being Pragyan’08 Systems head :
Implemented mail system through postfix, made its authentication work through ldap. Implemented mailman like features using contact attribute in ldap and aliases in postfix.
Made dovecot work through ldap too.
Learned what SSL certificates are, how they work, created a self signed ssl certificate for pragyan.org, using tinyCA2 provided in Fedora, and made it use it. (basically, allowed the use of https://pragyan.org/…)
Implemented FDS (Fedora Directory Server) as a much better alternative to LDAP on Pragyan Server.

Normal forms

Sitting here in Morgan Stanley, it’s hard to find free time. Things are always on the run.

But one day, I did find free time, and did what I had wanted to do for a very long time – learn about Normal forms in RDBMS.

Here is what I gathered from these websites :

1st Normal Form:

  1. No multivalued attribute
    StudID|Course
    12345|3100,3600,3900
    54321|1300,2300,3400
    (Wrong)
  2. No repeating group
    StudID|Course1|Course2|Course3
    12345 | 3100 | 3600 | 3900
    54321 | 1300 | 2300 | 3400
    (Wrong)
  3. Presence of atleast one unique identifier (addn of ID field)
    StudID|Course
    12345 | 3100
    12345 | 3600
    54321 | 1300
    54321 | 2300
    (Right)

There needs to be one composite key atleast. (One or more fields that can be used to distinguish every row uniquely). In the above example, if it was possible for the same student to take the same course twice, then we would have to add another uID field to bring the DB to the first normal form.

2nd Normal Form:

The DB is in this if all non-prime attributes are completely dependent on a candidate key. (and not on a part of it).
i.e. no repetition of key detail in one detail

| Candidate Key |
|StudID CourseID | StudNm ProfID ProfName
|123456 310000 | April00 6789 David
|123456 410000 | April00 2345 David
|123456 210000 | April00 6789 David

Candidate Key = StidID + CourseID

But StudNm (student name) depends only on StudID (and not on StudID + CourseID)
Had prof been just a function of course, even that would have to be removed.
Hence it is not in 2nd NF.
The ProfName-ProfID dependency is removed in the 3rd NF (where no attribute can be dependent on a non-key attribute in that table).
No part key dependencies can exist in 2nd NF.

None of the non-prime attributes of the table are functionally dependent on a part (proper subset) of a candidate key; in other words, all functional dependencies of non-prime attributes on candidate keys are full functional dependencies. For example, in an “Employees’ Skills” table whose attributes are Employee ID, Employee Address, and Skill, the combination of Employee ID and Skill uniquely identifies records within the table. Given that Employee Address depends on only one of those attributes – namely, Employee ID – the table is not in 2NF.

  • Candidate key: A candidate key is a minimal superkey, that is, a superkey for which we can say that no proper subset of it is also a superkey. {Employee Id, Skill} would be a candidate key for the “Employees’ Skills” table.
  • uperkey: A superkey is an attribute or set of attributes that uniquely identifies rows within a table; in other words, two distinct rows are always guaranteed to have distinct superkeys. {Employee ID, Employee Address, Skill} would be a superkey for the “Employees’ Skills” table; {Employee ID, Skill} would also be a superkey.
  • Non-prime attribute: A non-prime attribute is an attribute that does not occur in any candidate key. Employee Address would be a non-prime attribute in the “Employees’ Skills” table.
  • Primary key: Most DBMSs require a table to be defined as having a single unique key, rather than a number of possible unique keys. A primary key is a candidate key which the database designer has designated for this purpose.

Some thoughts :

  • In and after 2nd NF, all tables have only one candidate key. -> FALSE
  • All candidate keys in a table map to each other one to one. -> TRUE
  • What normalization stage ensures that only one candidate key is remaining?
    • None. Beacause normalization aims at reducing data duplication and inconsistencies. Candidate keys in a table causes no redundency.
  • If a table in 1st NF has no composite candidate keys (candidate keys consisting of more then one attributes), then it is automatically in 2nd NF.

3rd Normal Form:

No non-transitive dependencies:
A non-transitive dependency is one in which an attribute is dependent on a non-key attribute in that table.
For example, in the ProfID example given above, to bring it to 3rd normal form, ProfName and ProfID would have to brought to a separate table. Because ProfName is a non-transitive dependency.

  • In 2nd Normal form, all candidate keys become mutually independent. FALSE
  • In 2nd Normal form, any candidate key is a candidate key (i.e. is a sufficient and necessary dependency) for all other attributes. FALSE – this happens in 3rd NF.
  • In 3rd Normal form, non-prime attributes become dependent on a candidate key. TRUE