Posts Tagged ‘performance’
Velocity 2010 Wrapup
I got the opportunity to attend the Velocity 2010 Web Performance and Operations Conference last week with most of the rest of the guys from the Ops team at SugarCRM. It took place right around the same time as some other stuff I had going on, so I wasn’t able to hang out “after hours” as much as I would have liked, but it was great to listen in on some great sessions and talk to people far smarter than I!
Some of the sessions to highlight:
- “A day in the life of Facebook operations” – Gathered some info on what happens when you scale bigger than pretty much everyone else in the world.
- “Datacenter Infrastructure Innovation” by James Hamilton – I’ve seen a video of an older version of his presentation, but still some new info coming in about stupid things we’ve been doing for years in our datacenters.
- “In the belly of the whale: Operations at Twitter” – similar to the Facebook presentation, but the name of one of their monitoring tools was worth the mention: “Whale Watcher”
There were also a number of sessions on different performance tools. Hopefully posts on some tested out for real are soon to come.
On the slightly less technical side, there were some great talks about culture, including quite a few talks on DevOps (more on that in a later post):
- “Creating Cultural Change” by John Rauser – some great examples on how to create change of culture within your organization by trickery, humor, and other methods
- “Ops Meta-Metrics: The currency you use to pay for change” by John Allspaw – was a great talk that could also be a blog post of its own. Basically, as operations people, we tend to be afraid of change because we associate it with downtime. To get around that, we change more often and track just how often things really do go wrong.
- “The Upside of Downtime: How to Turn a Disaster Into an Opportunity” – Talk on how to use communication and transparency from downtime to create trust from your customers.
- “The Mobile Web: The next frontier” – presentation by Strangeloop on the monetary value of performance
Another great set of videos from the conference is the “Choose Your Own Adventure” talks with Adam Jacob from Opscode. You can head over to one of the videos and see them all listed in the related videos. I didn’t get to that session, but watched all the videos and wish I had now
.
It was a great few days and was followed by DevOpsDay USA over at the LinkedIn campus, more on that to come!
Tags: performance
Filed under Uncategorized :
Comments (1) :
Jun 30th, 2010
How to Treat Painfully Slow DNS Lookups in OS X
I’m not really sure what made this suddenly appear, but I’ve noticed over the past few days that my initial connections to websites have started taking longer and longer. Today, it became painful as the browser would say it was “Looking up example.com” for a good 3-4 seconds (if not longer) before loading the page. Once the lookup completed, the site usually came down pretty fast. Thanks to a (not so quick) Google search, I ran across this forum thread. It turns out, disabling IPv6 in OS X speeds things up quite a bit. To disable, go to System Preferences -> select the network interface you’re using (probably Airport) -> click Advanced.
Then, in the TCP/IP tab, change the drop down next to “Configure IPv6″ to “Off” instead of “Automatic”.
This worked for me. My browser and other various apps no longer take several seconds before loading pages on the web!
Tags: networking, os x, performance
Filed under How-Tos / Tips :
Comments (0) :
Apr 15th, 2010
Don’t Trust Your Computer.. Or Anyone Else’s
Last week was a bad week for tech people all over the world. There were outages reported at several major datacenter providers, including the likes of Rackspace, Equinix, and Google. As any good system administrator will tell you, what goes up will come down. Having a contingency plan is critical to any business that relies upon its web services (who doesn’t anymore?). Any person who spends any time working on computer systems knows that you can’t trust them any farther than you can throw them. You have to keep a close eye on them and monitor them closely. Somehow, they always end up surprising you.
Fortunately for me, I wasn’t directly affected by any of these outages. Small as it is, my blog stayed online as it isn’t hosted in any of those datacenters. Unfortunately, I did have a bit of an issue last week and it went on for some time without being caught. I’m not sure that my issues were related to any of these major outages, but one of the plugins I’m using makes a call to an external system that had some problems. I’m not going to point out which it was since it could have happened to any of them, but the lesson that I learned is worth mentioning.
What Happened
I use a plugin that pulls data off another system. That system was down which caused page load times to skyrocket. During the day I was checking my AdSense account noticing that pageviews were quite low, but didn’t really have time to look into it. Later that night I saw that pageviews were still really low so I decided to look into it. The Wordpress backend of my site was pretty snappy so I went to view the frontend. The page took forever to load. My site is on a shared web server, but I just so happen to be the system administrator at that hosting company so I quick logged into the shell of the web server. Loads were normal with no I/O wait, low CPU usage, and normal memory usage.
After checking loads of my own systems – something that’s already monitored, but I checked anyway – I popped open Firebug and reloaded the page. Bingo. I found the culprit. It was the service I talked about earlier that was slow to respond. I disabled the plugin and the site worked again immediately.
What I Learned
While my site is extremely small and I would not invest much time in this at this point, as you add external services to a site, it’s important to monitor those services as well as your own. Just because you watch that services are alive and responsive on your own web server, the services on others’ web servers are just as important as they can have a detrimental effect on your services as well. Of course, there’s nothing that you can do to control those services outside of SLA’s, but in my case, if I knew about the outage I could have simply disabled the service to minimize the effect on my site. Okay, I know I wasn’t losing millions of dollars or pageviews. I barely lost tens of page views over the whole thing, but as software and web services start relying upon ‘cloud’ services, keeping a close eye on all of those services is key.
Tags: cloud computing, performance
Filed under Tech Trends :
Comments (0) :
Jul 7th, 2009
Increasing Ext3 I/O Performance
In today’s computers, CPU throughput and amount of memory are no longer the bottleneck in performance. Ask any server administrator and they will tell you that one of the biggest concerns with responsiveness of server-side applications is the throughput to and from disk. If you’re using the ext3 filesystem, it turns out there’s a tip that you can use to gain up to a 40% performance boost.
With the default mount options in most distributions, the filesystem will log the last access time to a file. Hackosis recently wrote a quick tutorial on how to disable the access logging in the /etc/fstab file. A little research shows reveals a conversation in 2007 between kernel developers such as Linus Torvalds and Ingo Molnar in which Ingo states his experience with changing the setting:
yeah, it’s really ugly. But otherwise i’ve got no real complaint about
ext3 – with the obligatory qualification that “noatime,nodiratime” in
/etc/fstab is a must. This speeds up things very visibly – especially
when lots of files are accessed. It’s kind of weird that every Linux
desktop and server is hurt by a noticeable IO performance slowdown due
to the constant atime updates, while there’s just two real users of it:
tmpwatch [which can be configured to use ctime so it's not a big issue]
and some backup tools. (Ok, and mail-notify too i guess.) Out of tens of
thousands of applications. So for most file workloads we give Windows a
20%-30% performance edge, for almost nothing. (for RAM-starved kernel
builds the performance difference between atime and noatime+nodiratime
setups is more on the order of 40%)
Using the Hackosis’ tip on disabling the access logging could help boost performance on any systems you have using ext3. Of course, I’d recommend trying it out in a test system before deploying it in a production system, especially since changing the options means you have to re-mount your filesystems. I usually use ReiserFS, but if you are using something like Red Hat or CentOS, the kernel builds don’t support the xfs, jfs, or reiser filesystems unless you compile the options in yourself. Granted, you have to have a “RAM-starved” system first as Ingo states, but if you are experiencing performance issues on a server, there’s a good chance it’s a RAM-starved system or low I/O throughput.
Tags: linux, performance
Filed under How-Tos / Tips :
Comments (1) :
Nov 4th, 2008
Western Digital to Release 20,000RPM Hard Drive
With solid state disks on the horizon, Western Digital has been rumored to release a 20,000RPM hard drive. The news came from bet-tech.net that Western Digital is making the drive as a response to pressure to start converting to solid state disks. Currently, the biggest bottleneck in computer systems seems to be I/O speed. We have processors with 4 cores, we tote notebooks with 4GB of memory, we have 80GB iPods for storing music and yet we’re still running on the same speed disks as years ago. Storage capacity in drives has seemed to become a non-issue for a lot of new systems. On Newegg.com right now, there are Western Digital drives with a capacity of 1TB for a trivial $159. RAIDs can improve the read speeds, capacity, and reliability of storage systems, but drives still remain somewhat slow. These drives will be expensive, no doubt, but hopefully a vast improvement on their 10,000 and 15,000RPM predecessors. Bit-tech has also reported that sources at Western Digital have said that a priority on the drive is also noise, which I imagine would become a problem with a disk spinning that fast. They plan on releasing the drive as a 2.5″ form factor and creating a 3.5″ housing to allow it to fit into larger bays. Solid state disks are definitely coming and definitely here to stay, however they’re not here yet. Cost and capacity are currently the two biggest concerns, as well as some technical concerns of the drives over conventional hard disk drives.
I haven’t seen a projected release date, however it better come fast as solid state disks are on their way.
Tags: hard drives, performance, western digital
Filed under News :
Comments (1) :
Aug 18th, 2008

