Can I Upgrade an Ubuntu Instance on Amazon Cloud ????

Upgrading an EBS Instance


For the majority of the existence of EC2 there was no way to change the kernel that an instance was using. With the addition of EBS instances, that changed. I wanted to explain how you can take advantage of that EBS feature by upgrading a Ubuntu 10.04 LTS (Lucid Lynx) instance launched from a Beta-2 AMI to the Release Candidate. This same basic process should also allow you to upgrade across a release, perhaps from a 9.04 Alestic instance to Ubuntu 10.04 LTS.

In reality, if you're hoping to upgrade you're kernel and EBS instance, its because you already have one running and need to upgrade. But for the sake of this excercise, we'll launch a new instance based on the Beta-2 image in the us-east-1 region and then connect to it.

$ ec2-run-instances --key mykey ami-4be50b22
# wait a bit
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-184-73-101-171.compute-1.amazonaws.com
$ ssh -i mykey.pem ubuntu@ec2-184-73-101-171.compute-1.amazonaws.com


Now, on the instance we'll go ahead and do the upgrade. Whenever I'm working on ec2, I like to use GNU screen to protect against lost network.

% screen -S upgrade


This is the same basic process as upgrading any Ubuntu system. First update and then upgrade. Here, I've run '--dry-run' to point out that there would be kernel upgrades.

Also, the ec2 images suffer from a bug where grub will be installed and prompt you for some information even though its of no value. Because I know what those prompts will be, I'm go ahead and set them here so you're not interrupted during the dist-upgrade.


% sudo apt-get update
% sudo apt-get dist-upgrade --dry-run | grep "linux.*ec2"
libparted0debian1 linux-image-2.6.32-21-virtual linux-image-2.6.32-305-ec2
libpolkit-gobject-1-0 libpython2.6 libss2 libudev0 linux-ec2 linux-firmware
linux-image-ec2 linux-image-virtual linux-virtual locales module-init-tools
Inst linux-image-2.6.32-305-ec2 (2.6.32-305.9 Ubuntu:10.04/lucid)
Inst linux-ec2 [2.6.32.304.5] (2.6.32.305.6 Ubuntu:10.04/lucid) []
Inst linux-image-ec2 [2.6.32.304.5] (2.6.32.305.6 Ubuntu:10.04/lucid)
Conf linux-image-2.6.32-305-ec2 (2.6.32-305.9 Ubuntu:10.04/lucid)
Conf linux-image-ec2 (2.6.32.305.6 Ubuntu:10.04/lucid)
Conf linux-ec2 (2.6.32.305.6 Ubuntu:10.04/lucid)


% echo grub-pc grub2/linux_cmdline string | sudo debconf-set-selections
% echo grub-pc grub-pc/install_devices_empty boolean true | sudo debconf-set-selections


% sudo apt-get dist-upgrade
..
94 upgraded, 3 newly installed, 0 to remove and 0 not upgraded.
Need to get 84.1MB of archives.
..


Above, If you were upgrading from a previous release to 10.04, then you would use 'do-release-upgrade' from the 'update-manager-core' package.

At this point we've got 2 kernels installed, the new one and the old one. Unsurprisingly, we're booted into the old one.


% dpkg-query --show | grep "linux.*ec2"
linux-ec2 2.6.32.305.6
linux-image-2.6.32-304-ec2 2.6.32-304.8
linux-image-2.6.32-305-ec2 2.6.32-305.9
linux-image-ec2 2.6.32.305.6
% uname -r
2.6.32-304-ec2


Above, we can see that the 2.6.32-305.9 version of the kernel is the newest one. Its installed locally, but to boot it we have to find the aki of the version that is published by Ubuntu to ec2. The Ubuntu kernels are registered in ec2 such that you can correlate the dpkg version to the registered aki. We're going to query all the images, save that output to a file and then search for results that are owned by the Canonical user, and match our version string and arch.


$ owner=099720109477; # this is the canonical user's id
$ ver=2.6.32-305.9; arch=i386
$ ec2-describe-images --all > /tmp/images.list
$ awk '-F\t' '$4 == o && $3 ~ v && $8 == a { print $2, $3 }' \
a=${arch} "o=${owner}" "v=${ver}" /tmp/images.list
aki-1f02ec76 099720109477/ubuntu-kernels-milestone/ubuntu-lucid-i386-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel
aki-d324caba 099720109477/ubuntu-kernels-testing/ubuntu-lucid-i386-linux-image-2.6.32-305-ec2-v-2.6.32-305.9-kernel



That shows us that we have 2 kernels available matching that explicit version. One is "testing", and one is "milestone". These are actually the same thing. We label the kernels differently so the user easily knows what is testing and what is "released". The released kernel version will be labeled with 'ubuntu-kernels'. The RC kernel version gets labeled "ubuntu-kernels-milestone".

In order to change the kernel, we have to stop the instance, modify the 'kernel' attribute, and then start it up again.

$ ec2-stop-instances i-23453048
$ ec2-modify-instance-attribute --kernel aki-1f02ec76
kernel i-23453048 aki-1f02ec76


$ ec2-start-instances i-23453048
# wait a bit
$ ec2-describe-instances | awk '-F\t' '$1 == "INSTANCE" { print $4 }'
ec2-184-73-116-205.compute-1.amazonaws.com




So, in theory, we should have booted into our nice and shiny-new kernel. Lets test that theory:


$ ssh ubuntu@ec2-184-73-116-205.compute-1.amazonaws.com 'uname -r'
2.6.32-305-ec2



There you have it! This process can be applied to upgrading the RC to Release (which likely won't have a kernel change), or, eventually to upgrading your 10.04 LTS instance to a Maverick one.

Resolving “rsync warning: some files vanished before they could be transferred” on ec2-bundle-vol in an aws ec2 instance

This article explains how to resolve some of the common errors that you may encounter when bundling an AMI ( Amazon Machine Image) of an AWS EC2 instance. It is indeed a very good practice bundling your AMI frequently as the latest updates of your app gets stored directly to S3. So even if your instance gets crashed you can launch another instance of your latest ami and get your site back live in minutes!.

But bundling can be a big hassle if we get stuck with rsync errors during the process. There are quite a few cases where rsync fails during bundling.

1) No space left on the device

This usually happens when we specify the size of the / partition with the -s parameter that specifies size, in MB of the image file to create. The maximum size is 10240 MB. Removing the “-s” flag and letting the system determine the size of the image itself will fix it. Also the bundling process will try to include all softlinks ( if any ) to other partitions of larger size ( greater than 10GB ). This would try creating an image file larger than 10GB size and hence throw bundle errors. So check for any softlinks before bundling.

2) Size of /tmp

By Default ec2-bundle-vol will try to bundle the image in /tmp. So if the size of /tmp is small this will throw bundling errors too. Specifying “-d /mnt” to point to the ephemeral drive rather than bundling the image in /tmp by default is a fix for this.

3) rsync warning: some files vanished before they could be transferred

This one is quite an interesting error as it is a warning that some files are being deleted while the bundling process is happening. In my case I had an instance with around 2GB free space in / partition. Also there was a 100GB EBS mounted on /home_ebs. I tried bundling the instance by excluding the 100GB EBS, but still it gave rsync error.
view source
print?
1 # ec2-bundle-vol -r i386 -d /mnt -p "myami" -u xxxxxxxxx -k /mnt/pk-xxxxxxxxxxxxxxxx.pem -c /mnt/cert-xxxxxxxxxxxxx.pem -s 10240 -e /home_ebs --debug

Copying / into the image file /mnt/myami…
Excluding:
/sys
/proc
/dev/pts
/proc/sys/fs/binfmt_misc
/dev
/media
/mnt
/proc
/sys
/home_ebs
/mnt/myami.
/mnt/img-mnt
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.003113 seconds, 337 MB/s
mke2fs 1.39 (20-June-2010)

file has vanished: “/var/log/dcpumon/toplog.1276842601″
file has vanished: “/var/log/dcpumon/toplog.1276842901″
file has vanished: “/var/log/dcpumon/toplog.1276843201″

rsync warning: some files vanished before they could be transferred (code 24) at main.c(892) [sender=2.6.8]
Executing: umount -d /mnt/img-mnt

ERROR: execution failed: “rsync -rlpgoD -t -r -S -l –exclude /sys –exclude /proc –exclude /dev/pts –exclude /proc/sys/fs/binfmt_misc –exclude /dev –exclude /media –exclude /mnt –exclude /proc –exclude /sys –exclude /home_ebs –exclude /mnt/myami –exclude /mnt/img-mnt -X /* /mnt/img-mnt 2>&1 > /dev/null”

Running Bundle Volume from within a running installation may be problematic because partially written files may get copied into the AMI. To minimize the risk the best option is to stop all non essential services and try rebundling. In my case I removed the /var/log/dcpumon directory and went for a rebudling. That did the trick !.

Please note that using the “–debug” option with ec2-bundle-vol command will be of immense help in troubleshooting bundling errors. Hope this helps anyone out there ! :)

If your Nightmare came true and you Terminated that instance Accidentaly !

If you are wondering what inspired me to write on this topic then you've guessed it right:-)..Well I have been there and done that ..lol.Now imagine you have an instance that is made by one of the Big B's of tech firms and it so happens that the instance simply refused to be bundled(Some cost factor involved here so they wanted to play safe I guess).Now here comes the real thing you have managed to configure the instance and are prepared to showcase this to a client and hence you have no option but to keep it running, But Wait! Hey if you cant bundle then how do you make sure that on the D day things will be just perfect and Mr Murphy's Laws will not come true.
So here is the plan you call your entire team on a weekend and start the recreation of a duplicate instance so that there are two parallel instances up and running and even if one fails the other is there to support.The team comes starts the process of recreation and suddenly the nightmare comes true ..Holy Cow The Main instance has been terminated ....Now what ? .. Watch this space.Coming Soon ....

Every Wondered How to launch a linux Instance automatically ?

I don't know about you, But whenever we need to launch an Instance the only option I could see is to login to the AWS console and select the AMI and then follow the usually process .Now the following command can come to your rescue .

ec2-run-instances AMI [-n INSTANCE_COUNT] [-g GROUP [-g GROUP ...]] [-k KEY] [-d USER_DATA | -f FILE_NAME]

Now if you observe the only important option that is not present is the Launch zone
viz. US-EAST..etc .Hence this is what you can do use "-Z" option as follows

$ec2-run-instances ami-xxxxxxxx -z us-east-1a

The above command will allow you to specify the launch zone so that you can safely attach the storage once its up.Amazon are you Listening why haven't you mentioned this on your website ."http://docs.amazonwebservices.com/AmazonEC2/dg/2006-10-01/CLTRG-run-instances.html"

So What Is Cloud Computing Exactly ?

The Wikipedia says "Cloud computing is a Internet-based computing, whereby shared resources, software, and information are provided to computers and other devices on demand, like the electricity grid" .But this is all about the technical jargon in a lay mans word what I would say is that cloud computing is what a Taxi is to an Automobile its basically A Pay As You go kind of a model in which an Individual,SME,Corporates hire or lend Computing services .So lets assume you have a website which has a regular number of hits but on certain occasions you see that the Hits are almost double of what you had expected, In that case the only option left traditionally is to Purchase a new Machine and configure the same Web server or have virtualisation implemented.But these technologies come at a cost which you may have to invest for a short period of time(In this case for the days in which you see above average Hits ).So here is where cloud computing comes to your rescue.So in case your server was on the cloud all you have to do is add a couple of servers which are a replication of the original one and use them untill you require them and once done you can Bundle the server or simply terminate .Even the scaling can be automated .In coming days I shall update the various types of Clouds that are picking up in the Tech World

Bundling errors in Linux Instances On AWS?

I have come across the following error while bundling since quiet some time now so though of posting the problem with the solution

1048576 bytes (1.0 MB) copied, 0.001952 seconds, 537 MB/s
mke2fs 1.39 (29-May-2006)
mount: could not find any free loop device
ERROR: execution failed: "mount -o loop /mnt/latin-bpo-dev-database-image2010-08-13 /mnt/img-mnt"
Copying / into the image file /mnt/latin-bpo-dev-database-image2010-08-13...

The reason this happens is cause the loopback device that is used while bundling has a maximum limit of 8 and in case the previous
onces havent been cleaninly unmounted and hence the kernel simply refuses to allow a new mount which is done during the bundling process

The Solution to the problem is as follows

[root@:~] losetup -a
[root@:~] losetup -d /dev/loop7
[root@:~] losetup -d /dev/loop6


Once done with this try Bundling the instances

Well Times Change and so did I :-)

Hello Geeks,
Well its been long since I am back with Blogging and I guess will keep up to this habit.In the coming days I shall Blog More of Cloud Computing and Mobile Computing .I shall try to keep you guys abreast of the latest Tech surrounding these topics .Off-course that goes without saying Linux is still gonna be around so Happy Reading and do let me know your comments ..
Thanks
Jiten