Whilst doing a little storage vMotion using VMware this week, I got a little lesson in patience from VMware and thought that I would share my experiences to help a few others who are looking at doing storage migrations of larger VM's. My experiences are with VMware ESX 3.5 and Virtual Center 2.5. Not the most recent, I know, but we haven't had time yet to upgrade to vSphere 4.

The point of my migration was to get a particular vmdk off a slower set of disks and only a faster set of disks. Due to the fact that ESX 3.5 doesn't directly support moving only one vmdk at a time, I had to do a little dance to get the one vmdk that I wanted to move, moved. I had to move the virtual machine itself and ended up moving a couple of extra disks in order to to get it to move.

Near as I could figure, Storage vMotion (with ESX 3.5) has the following stipulations:

1. The virtual machine files must move.
2. The virtual machine files must move to a datastore that is large enough to hold the largest vmdk.
3. Apparently, the host needs enough memory free as the size of memory allocated to the already-running virtual machine (during the move, memory usage spiked which caused all kinds of problems for me since this was a large virtual machine, but not really the point). I haven't verified this requirement yet, but that's what my initial thoughts are after seeing the behavior.

Seeing the migration options for vSphere 4, I wanted to cry on how difficult my life was made by these requirements, but that's another story. We'll be scheduling that upgrade shortly. :)

Anyway, I ran the Storage vMotion. At which point, I managed to bring the virtual machines on my host to a screeching halt, not knowing number 3 right off hand. After killing a few virtual machines and moving a few others away from this host, we were back under way.

Near the end of the Storage vMotion (at 90% to be specific), the interface stayed at the same percent for several minutes and I was greeted with this friendly error:

A general system error occurred: failed to reparent/commit disk(s) (vim.fault.Timedout)

Uh-oh. A quick Google serch found this VMware knowledgebase article.

Unfortunately, I missed the "Incorrectly" throws a timeout error message and panicked a bit. I started digging around in the destination and original datastores and found tons of "DMotion" files everywhere. While desperately looking for solutions around the web, my Virtual Center screen refreshed with the new datastores being associated with the disks and moved over. Yup, while I was freaking out, the whole thing just took care of itself.

Apparently, when working with Virtual Center, one must always have a bit of patience and remember to double-check timeouts inside Virtual Center with the ESX hosts directly. I suppose I should have known better as I've seen this kind of behavior while working with snapshots in the past. If you come across this post while searching for this error, take a few minutes to relax and let VMware do its thing in the background while Virtual Center shows stupid error.

While playing with WordPress on nginx for my last post, I discovered that a majority of the how-tos out there on running PHP/MySQL applications using nginx left a bit to desired. Here's the steps that I took to get my application (WordPress, specifically) working.

Install nginx, MySQL, and PHP

First, let's install nginx and PHP along with a few PHP libraries:

sudo apt-get install php5-mhash php5-mysql php5-odbc curl php5-curl php5-gd php5-imap nginx php5-cgi php5-cli php5-common

If you didn't already have MySQL installed on your server, you'll need that too:

sudo apt-get install mysql-server

The installer will prompt you to enter a root password. Make sure it's a fairly good password, but also be sure to record it as you'll need it later.

Install spawn-fcgi

Spawn-fcgi used to be included with lighttpd, but has been moved to its own project, so it can be downloaded separately. Unfortunately, the spawn-fcgi project is not in the Ubuntu repositories, so it has to be installed separately. First, download the tarball from the spawn-fcgi project page. As of this writing, it's on version 1.6.2. For this particular version, run the following from a directory your user can download to:

wget http://www.lighttpd.net/download/spawn-fcgi-1.6.2.tar.gz

Untar it:

tar zxf spawn-fcgi-1.6.2.tar.gz

Make sure you have the compilation tools:

sudo apt-get install build-essential

Now, navigate into the spawn-fcgi download directory and compile:

cd spawn-fcgi-1.6.2 ./configure make

Now, let's install it into /usr/bin:

cd src sudo cp spawn-fcgi /usr/bin/

Now, let's make the init script. Copy the following example into /etc/init.d/fastcgi:

#!/bin/bash PHP_SCRIPT=/usr/bin/php-fastcgi RETVAL=0 case "$1" in 'start') $PHP_SCRIPT RETVAL=$? ;; 'stop') killall -9 php5-cgi RETVAL=$? ;; 'restart') killall -9 php5-cgi $PHP_SCRIPT RETVAL=$? ;; *) echo “Usage: php-fastcgi {start|stop|restart}” exit 1 ;; esac exit $RETVAL

Next, let's create the script to launch the PHP CGI process. Copy the following example text into /usr/bin/php-fastcgi:

/usr/bin/spawn-fcgi -a 127.0.0.1 -p 9000 -C 5 -u www-data -g www-data -f /usr/bin/php5-cgi

Make sure the new scripts are executable:

chmod +x /usr/bin/php-fastcgi /etc/init.d/fastcgi

You should be able to start up your fastcgi process now with the following:

/etc/init.d/fastcgi start

Make the fastcgi process start at boot:

sudo update-rc.d fastcgi defaults

Setup nginx site

I used the following as my site file. A majority of it was taken from the default site and parts from other how-tos. You can rewrite the /etc/nginx/sites-available/default with this templated page (in my example, I assumed that the site is called site.com and that you are using WordPress at /var/www/wordpress). Be sure to change the "root" and "SCRIPT_FILENAME" lines.

server { listen 80; server_name site.com; access_log /var/log/nginx/localhost.access.log; location / { root /var/www/wordpress; index index.php; if (-f $request_filename) { expires 30d; break; } if (!-e $request_filename) { rewrite ^(.+)$ /index.php?q=$1 last; } } error_page 500 502 503 504 /50x.html; location = /50x.html { root /var/www/nginx-default; } # pass the PHP scripts to FastCGI server listening on 127.0.0.1:9000 # location ~ .php$ { fastcgi_pass 127.0.0.1:9000; fastcgi_index index.php; fastcgi_param QUERY_STRING $query_string; fastcgi_param REQUEST_METHOD $request_method; fastcgi_param CONTENT_TYPE $content_type; fastcgi_param CONTENT_LENGTH $content_length; fastcgi_param SCRIPT_FILENAME /var/www/wordpress$fastcgi_script_name; include /etc/nginx/fastcgi_params; } }

Before restarting nginx, make sure everything is cool with the config and correct any errors:

sudo nginx -t

Now, restart nginx with the new changes:

sudo /etc/init.d/nginx restart

Install your application

You can now install your application as you normally would using Apache. In this example, you can download the WordPress packages to /var/www/wordpress and install from there, making sure that the files are owned by the www-data user.

It's a slightly older post, but after I read Workswithu's report on the four features that they believed set Ubuntu apart, I had to agree whole-heartedly with the fourth one regarding implementation of auto-complete on the shell. As I was reading, I constantly dropped to the shell to discover new tab-complete features that I didn't know about, including apt-get that I really didn't notice but used every day. Anyway, today I discovered yet another.

This kind of defeats the purpose of DNS, but I discovered that if you put a host into the /etc/hosts file on a Ubuntu 9.04, then both SSH and rsync tab-complete for you when you start typing the hostname. I'm not sure about scp, but I'm sure it works the same way.

Extending this functionality further, it would be awesome if I could tab-complete from my known_hosts file, but that doesn't seem to work.

Anyway, if you find yourself SSHing or rsyncing files to a host or set of hosts all the time, this trick can be very time-saving. Perhaps not quite as time-saving as being able to SSH with the click of a mouse, but often its not very convenient to leave the keyboard just to save a few keystrokes at the shell.

I just got my Google Voice invitation today. It's pretty sweet. It allows you to give out one phone number and configure it to ring multiple phones depending upon who is calling and when they are calling. It also packs voicemail and SMS into a convenient place along with a host of other features. The other thing that it does is it completes the communications circle of products that I use from Google.

After thinking about it for a few minutes, I use Google for everything. It knows my web searches, my web site's stats, and manages my ads. I also use Gmail, Google Talk, Google Calendar, Google Reader, now Google Voice, and occasionally Google Docs. When I get lost I pull out the Google Maps app on my phone. I'm even playing around with their Android operating system and am considering buying a phone based upon it when Verizon gets a model. Wow. That's one heck of a list when I fire them off one at a time. I'm not even using all the services available to me.

It's no wonder Microsoft wants some of Google's search market share. Whenever I search I come up with new Google services to sign up for. One of the biggest advantages for Google up to this point (at least for me) as been the fact that Google didn't have an operating system. This caused them to have to integrate and build tools for other operating systems. Microsoft has the luxury of building its own OS plus its online offerings so it can integrate easily into its own products. We'll see what happens when Chrome OS comes out, but that's a long way out. For now I guess Google's got all my data..

There's plenty of situations where you need to to open a shell to run something as a different user within a script. Scripts like the Zimbra start-stop scripts can't be run using sudo (Honestly, I don't know why and haven't questioned it). For my script, the root user doesn't have the zmcontrol utility in its path, so it had to actually spawn a shell as the zimbra user to run. Since it wasn't terribly obvious after a few minutes of googling, I decided to post how I did it. My example uses the zmcontrol utility which has to be run as the zimbra user:

su -l -c "zmcontrol start" zimbra

Last night I was playing with my MythTV box setting up a frontend on my MacBook and I noticed after I was done playing that the MythTV backend server's disk was constantly making noise. When I stopped the MythTV backend services, the noise stopped. Then once I started it back up the activity continue the way it was before. After doing a little Googling, I found out that others had had this problem on Ubuntu 8.04 shortly after installing MythTV. Later in the bug report someone discovered that on their machine, the constant disk activity was MythTV logging that it couldn't connect to the MySQL server. Turns out they were exactly right. In my /var/log/mythtv/mythbackend.log file, it was constantly writing:

QSqlQuery::exec: database not open QSqlQuery::exec: database not open 2009-06-26 23:50:02.359 DB Error (KickDatabase): Query was: SELECT NULL; No error type from QSqlError? Strange... QSqlQuery::prepare: database not open QSqlQuery::exec: database not open 2009-06-26 23:50:02.412 DB Error (SimpleDBStorage::load()): Query was: SELECT station FROM record WHERE recordid = :WHERERECORDID No error type from QSqlError? Strange... 2009-06-26 23:50:02.413 Unable to connect to database! 2009-06-26 23:50:02.414 Driver error was [1/1045]: QMYSQL3: Unable to connect Database error was: Access denied for user 'mythtv'@'localhost' (using password: YES)

Turns out, I had been messing with the MySQL permissions to get my MythTV frontend machine to work and hadn't noticed that I broke the backend. If you install MythTV and notice that you get constant disk access after it starts, this is likely the culprit.

Since I like to be an example to others of what not to do, I thought I'd share my fun today. Today I was doing a little cleanup on our systems and removed an empty directory that had been mounted by another system via NFS. (HINT: if you're stuck in this position and need help fast, scroll to the end, otherwise I'm going to tell you the story) Not really realizing that I had done this, I went on with my day as normal. Later in the day, I was asked why all of a sudden certain operations were timing out on that system. The first thing I checked was disk space on the device which returned some mount points, but not all and "sat there doing nothing". After a little while I closed the session and opened up a new one. Running a mount returned all the mount points that allegedly existed, but the list from my 'df -h' stopped at the mount right before the NFS mount point in question. Doing a 'umount -f /mnt/point' simply returned a "device is busy" error.

The particular directory that I had deleted was an old directory that we were using for Virtualmin to do its backups to and had moved. Since it was complaining I thought I'd just try doing a bounce of the Webmin/Virtualmin services quick to see if that helped since Virtualmin should've been the only application that needed to have files open in that directory. But, alas, didn't make a difference. Then I had the bright idea of trying to find out what processes might've tried to have files open in that directory. I ran a 'lsof | grep /mnt/point' and it "sat there doing nothing" again..

After my failed attempts, I decided to try to recreate the directory on the other end, re-export the share and hopefully my system would pick back up where it left off. Unfortunately, I didn't realize this, but my export had already been deleted from the /etc/exports file as well so recreating the directory and re-exporting the NFS shares did me no good.

A quick Google search led me to a post in the Ubuntu forums with a user saying they had to reboot. As this server is in production, a reboot was not an option at this point (or at least the worst possible scenario). For sure if the system was rebooted, the mount point wouldn't be busy anymore but that's no good. Later in the thread (after the unnecessary reboot) I found that one can let umount do a "lazy" umount. This lets the OS umount a mount point while still maintaining any references to files in that mount point until those processes lets go. Anyway, if you're still with me, the command is:

sudo umount -l /mnt/point

Voila! No more timeouts checking disk space, lsof's, and the mount point is no longer mounted.

Quite often we read about how small things are so much more complicated on Linux than on Windows. I set out on a journey today to add a printer via USB to my Ubuntu desktop and share it out to my MacBook at my new apartment. If there's one thing in this sick, twisted world that I hate, it's printers.

Given that background, I was shocked at just how easy it was to setup the printer, share it out, and connect to the shared printer using my Macs and other Linux machines on my home network. The printer that I worked with was just a cheap inkjet - an HP Deskjet D2330. I had a heck of a time with the driver for OS X directly connected, so I wasn't expecting much. I plugged in the USB cable and voila, my Ubuntu desktop popped up a message that the printer was ready to rock. I opened up OpenOffice.org and printed out a doc just to make sure my desktop wasn't just playing with my heartstrings. Sure enough, it printed my doc just fine.

Now, on to sharing. Doing a little poking around in the GUI, I found the printer configuration at System -> Administration -> Printing. Under the policies, the "Shared" check box was checked, so I popped open the System Preferences on my MacBook and tried to add a printer. I couldn't find the printer in any of the discovered list. A quick Google search reminded me that the CUPS configuration web GUI could be found at: http://localhost:631 when CUPS was installed and running. I navigated to the page and found the check box called "Share published printers connected to this system". I checked it and clicked "Change Settings". When prompted to authenticate, I used my desktop login username and password.

When I flipped over to my MacBook, I opened up System Preferences again, added a printer and there my printer was. I clicked add and it was ready to go within moments.

After my successes on my MacBook, I decided to press my luck just a bit more by adding my Linux laptop as well. On my laptop, I run Xubuntu 9.04 and it was almost as easy to setup. I just opened the printer configuration window via Applications -> System -> Printing and clicked on the new printer button. The Xubuntu machine scanned for printers on the network, discovered my printer and added it. I can't say I've had nearly that smooth of an experience adding any sort of printer to Windows.

An Apache server that I was working with was having an issue starting in a reasonable amount of time. This particular server was running on Ubuntu 8.04 on top of VMware ESX 3.5. The service would start eventually, but would hang for a fairly significant amount of time. In the Apache error logs it would show the following line: [notice] Digest: generating secret for digest authentication ...

Eventually, the digest generation would finish and my websites would come back. Sometimes it would take just a few seconds and other times it would take about 30 seconds. After doing a little digging, I found that a number of people were having this issue, with very few answers.

The root problem, as I found out, was that the OS was running out of entropy. One can see how much entropy is available with the following command:

sudo cat /proc/sys/kernel/random/entropy_avail

My system was returning a value somewhere around 150-200 and went down to nothing while Apache was generating its digtest. When I looked at the same file on other systems, they were all 2000+, an obvious problem.

According to a couple of blog posts, installing rng-tools and running the rngd daemon seemed to be the answer.

I did a quick install of rng-tools:

sudo apt-get install rng-tools

However, rngd failed to start, reporting that it couldn't find the hardware generator:

/etc/init.d/rng-tools: Cannot find a hardware RNG device to use. invoke-rc.d: initscript rng-tools, action "start" failed.

After a little more searching, I found out that you simply have to change the source for the generation to /dev/urandom:

sudo vim /etc/default/rng-tools

And changed the line:

#HRNGDEVICE=/dev/hwrng

To:

HRNGDEVICE=/dev/urandom

Save the changes and start rngd:

sudo /etc/init.d/rngd start

Now, 'catting' my entropy_avail file displayed over 2000 like my other systems and Apache starts right up on a restart.

Yahoo! Zimbra Desktop is almost reaching full release status, but it still has one limitation: only one installation on a machine is supported. The problem lies in the fact that Zimbra Desktop uses a Jetty server for the backend application and each instance is configured to use the same port. We can hack Zimbra Desktop by changing the port from the default to another port on the system and then add another installation. This hack worked on Ubuntu 8.10 and should work on any Linux distro. I tried it on my MacBook, but Zimbra Desktop put a directory in the /Applications folder that contained prism and the installer didn't allow me to continue. I haven't tried this on Windows, but I imagine it should work in a similar way.

Disclaimer: This hack is not in any way supported by Zimbra or by the author. I am not responsible if your Zimbra Desktop data becomes unusable. Please backup your Zimbra Desktop installation before beginning if you already have data in Zimbra Desktop. Also, this hack will most likely break upgrades and have to be re-applied. A working knowledge of some Linux shell is recommended. Now that that's out of the way, let's begin.

1. Install Zimbra Desktop as normal.

Download the installer from Zimbra's website and install it as normal on the first user's account. This is the user's account that we will change the web server ports. Once Zimbra Desktop is installed, stop the Zimbra sync services from the command line:

~/zimbra/Zimbra Desktop/Zimbra Desktop stop

You can check that the services have actually been stopped by running:

ps aux | grep zimbra

2. Edit Jetty and Prism config files.

Open up a terminal window in Linux, 'cd' to the Zimbra Desktop directory (~/zimbra/Zimbra Desktop) and find all config files containing the port number:

grep -r 7633 * | grep -v logs

The output will display the file name followed by the line that contains "7633" which is the default listening port for Zimbra Desktop. You can ignore any line that says "Binary file <filename> matches". The following files should have been returned and will need to be edited:

conf/localconfig.xml jetty/etc/jetty.xml jetty/webapps/service/WEB-INF/web.xml jetty/webapps/zimbra/js/Startup1_2_all.js jetty/webapps/zimbra/js/NewWindow_2_all.js jetty/webapps/zimbra/js/zimbraMail/core/ZmZimbraMail.js Zimbra Desktop.webapp/webapp.ini

Open up each file with the editor of your choice and change the text "7633" with "7634" or some other port of your choosing. Just make sure they are all the same.

3. Start Zimbra Desktop again.

You can now start the instance for user:

~/zimbra/Zimbra Desktop/Zimbra Desktop start

If all config files have been changed, the Zimbra Desktop sync engine will start as normal. You can launch the client from the desktop shortcut, or you can connect using another browser. To get the URL to connect to, consult the file: ~/zimbra/Zimbra Desktop/Zimbra Desktop.webapp/webapp.ini and look for the line "uri=". Otherwise, you will receive an error from Jetty saying that you are unauthorized. If you receive an error, be sure to check that all files have been changed by running the grep comand at the beginning of step 2.

4. Login as second user and install.

Login as the second user that you want to install Zimbra Desktop as. You will then be able to install Zimbra Desktop as normal.

5. Rinse and repeat.

You can stop at this point if you don't have any other users or if you don't plan on doing upgrades. I haven't tested this with upgrades, but I imagine that an upgrade will return it to the normal ports. In which case the upgrade will fail and data loss could result. I would recommend repeating steps 1-3 and changing the port in step 3 to 7635 (or whatever port you should choose).

That's it! You can now have multiple users with their own Zimbra Desktop instance on your computer. Again.. this how-to is a total hack and not supported by anyone, but it worked for me. Upgrades to the software will most likely break the setup and could result in data loss. Of course you're already doing this, but be sure to run a backup before trying any upgrades to the software.