MESSAGE
| DATE | 2004-04-26 |
| FROM | From: "Inker, Evan"
|
| SUBJECT | Subject: [hangout] A Sunny Forecast for Open-Source
|
A Sunny Forecast for Open-Source
Weather.com's move to an all-open-source Web site infrastructure has enabled
the company to lower costs while meeting increased capacity demands.
Field Report by Julia King
APRIL 26, 2004 (COMPUTERWORLD) - Four years ago, Weather.com, the online
counterpart of The Weather Channel Interactive Inc.'s 24-hour TV channel,
relied entirely on proprietary commercial software to serve up millions of
Web pages of maps, forecasts and hour-by-hour weather data every day.
Today, the Atlanta-based Web site serves more than 50 million pages on
stormy days, but it runs almost entirely on open-source software and
commodity hardware. And since the move to the new architecture, it has
slashed IT costs by one-third and increased Web site processing capacity by
30%.
"Where it makes sense, we will always look at open-source alternatives,"
says CIO Dan Agronow. The reason is simple, he says: Despite the
self-serving air of fear, uncertainty and doubt (FUD) that commercial
vendors create around open-source software, lots of open-source products
work very well and can be deployed and run for about half the cost of
commercial products.
Agronow recalls one time when an IBM sales representative warned him that
he'd likely lose his job for dumping IBM's WebSphere application server and
formal support program for an open-source alternative.
"We've heard a lot of the FUD about how you can't replace Netscape with
Apache, or WebSphere with Tomcat [application server software], but when
we've tried it, we haven't seen the gotchas that the vendors all tell us
about," says Agronow, who worked at IBM as a technical project manager for
14 years before joining Weather.com.
"My experience is we have actually received better support of open-source
software than we have with commercial software," he adds.
The Linux Switch
But that's not to say there haven't been technical challenges. One of those
surfaced in 2001 when Weather.com was still running WebSphere but decided
for financial reasons to change operating systems, migrating from a Sun
environment of Solaris running on Sun 420R servers to Linux running on IBM
xServer 330 servers.
"We had problems like installation scripts not working or the GUI not
connecting to do the proper administration. There were various things that
were subtle differences between the platforms that hadn't been totally
worked out on Linux," recalls Jon Badenell, Weather.com's chief architect.
"Nothing was a showstopper, but it was not a turnkey installation either."
Working with IBM, Weather.com's 23-member team of systems administrators,
developers and architects resolved all of the inconsistencies. In the
process, they boosted both their confidence and skills as open-source
experts. And Weather.com saved hundreds of thousands of dollars by moving
off the Sun servers, Badenell says. "Literally, in some cases it was orders
of magnitude cheaper to go to the Linux boxes," he says. "We replaced
machines that were $500,000 with machines that were $50,000."
Tomcat vs. WebSphere
Bolstered by its success with Linux on Intel-based machines, the IT team
began looking for an open-source application server to replace WebSphere.
Again, cutting costs was a major driver. Another was reducing the
complexities and overhead associated with running the complex and
feature-rich WebSphere.
The WebSphere servers were showing signs of strain and required repeated
restarts as Weather.com's traffic load steadily increased, spiking to more
than 18 million page views one day during a snowstorm in January 2002
[QuickLink 45818]
"Our Web site is big, and we get a huge number of hits, but we don't do a
lot of complicated stuff. It's not transactional, and users are reading
data, not submitting it, so we didn't use three quarters of what WebSphere
actually offered," says Badenell. "There was an overhead penalty from just
the size of the installation and the administration of it."
Weather.com's software developers also found WebSphere to be cumbersome and
slow. As a work-around, they frequently developed applications using another
tool and then ported them to the WebSphere application server.
"It was hard to run WebSphere and an IDE [integrated development
environment] because of all the resources WebSphere took," recalls Jeff
Cunningham, who leads the Internet application development team at
Weather.com. "You had to run an instance of DB2 on your machine because
WebSphere stored its configurations in DB2, so you had to have all that
overhead. It was just really slow. I just started using Tomcat for
development because it was so much faster."
There was also the issue of IBM's response. "There was kind of a gauntlet
laid down," says Joey Reynolds, senior systems administrator-supervisor at
Weather.com. IBM's WebSphere developers were familiar with Weather.com's
software code because they had worked closely with the Web site's IT team to
resolve earlier performance problems. "They said, 'We don't think you guys
can do this, and you'll end up staying with WebSphere and paying support,'"
says Reynolds. (IBM declined to comment for this story.)
But the development staff was undaunted."There are tremendously bright
individuals here, and to challenge them to go that little extra bit is a
dangerous thing if you want to keep their business," Reynolds says.
Moreover, Weather.com developers had been using Tomcat and therefore, says
Reynolds, "we had already seen that the open-source community was adept at
answering our questions. It wasn't like we were blind."
The development team considered several open-source application servers,
Cunningham says, including Resin from Caucho Technology Inc. and offerings
from Hewlett-Packard Co. and GemStone Systems Inc. "At one point, I had
three or four on my machine," he recalls.
But a majority of team members preferred Tomcat, so the group decided to
pilot-test the software with a new version of the Web site's local activity
page, which dynamically serves up weather data for selected cities. For the
test, the team had configured the servers to switch back and forth between
WebSphere and Tomcat.
"When we rolled out that page, we discovered Tomcat was significantly
faster," recalls Badenell. "Because that page is close to 60% of our total
page views, it was at that point that we decided we would go ahead and make
the switch over to Tomcat. We were holding our breath, and it worked out,
and we haven't turned back since."
The switch to Tomcat software and Intel-based commodity servers also enables
the Web site to add capacity quickly and relatively inexpensively. "In our
architecture, which is very flat, scalability comes by buying more machines
and throwing more Web servers on them. It's much more cost-justifiable to
add 30% more capacity by buying 12 more machines," says Tim Bolser, director
of application development. "We don't have to write a check to IBM [for
WebSphere licensing fees], and it gives us a lot more flexibility in terms
of deploying assets."
"On a typical day, we do 30 million database calls [to the Web site's main
Oracle database] for just the desktop application," says Agronow. "We're
able to handle that with Tomcat and open-source because the infrastructure
gives us that capability. All of the servers are created generically, so we
can scale horizontally. As our capacity increases because downloads of the
desktop application are increasing, we just add another generic box, and
that adds capacity."
All told, Weather.com has 75 pure Web site servers, 12 servers supporting
its desktop products and 20 servers to support miscellaneous requirements.
It also has dozens of development and test servers, bringing the total
number of servers to about 180.
The Support Factor
The site's software developers also are happy, says Bolser. "Part of what we
like with open-source is you can look under the hood and see things," he
says. "With commercial software, if there's a hole and it gets exposed,
you're relying on the vendor to fix it, but if it's open-source, either the
open-source community or you can plug the hole yourself. Because technical
people are skeptical by nature, having more access to the code actually
makes some people feel more comfortable and secure, rather than less."
Robin Bloor, an IT analyst at Baroudi Bloor in Arlington, Mass., says
receiving a high level of support from open-source communities is typical,
especially for what he calls "flagship" open-source products, such as
Apache, Linux and Tomcat.
"The people who contribute to the creation of the product are an online
community and continue to contribute to its support," Bloor says. "The
person you talk to about support may even write a little piece of code for
you for a very specific problem."
Looking ahead, Agronow says he wants to optimize the Tomcat software and
Weather.com's overall server environment for Intel Corp.'s P4 processors.
"If it was optimized, we'd get even better performance out of it. That's the
one disappointment I have with Tomcat -- it doesn't seem to be optimized for
the latest generation of processors. And we want speed. Speed is what gives
us performance and increases capacity," he says.
Weather.com is also working on swapping out its Oracle database for the
open-source MySQL.
Agronow says the IT team has clearly demonstrated that open-source makes
sense for Weather.com. "It saves us money, and every time we did [a
migration] we got more confident about the next one," he says.
And that confidence extends beyond the IT staff, Agronow says. "Now when I
talk to senior management about moving from Oracle to MySQL they don't ask
me, 'Are you sure?'" he says. "They ask me, 'When?'"
****************************************************************************
This message contains confidential information and is intended only
for the individual or entity named. If you are not the named addressee
you should not disseminate, distribute or copy this e-mail.
Please notify the sender immediately by e-mail if you have received
this e-mail by mistake and delete this e-mail from your system.
E-mail transmission cannot be guaranteed to be secure or error-free
as information could be intercepted, corrupted, lost, destroyed, arrive
late or incomplete, or contain viruses. The sender therefore does not
accept liability for any errors or omissions in the contents of this
message which arise as a result of e-mail transmission.
If verification is required please request a hard-copy version.
This message is provided for informational purposes and should not
be construed as an invitation or offer to buy or sell any securities or
related financial instruments.
GAM operates in many jurisdictions and is
regulated or licensed in those jurisdictions as required.
****************************************************************************
____________________________
NYLXS: New Yorker Free Software Users Scene
Fair Use -
because it's either fair use or useless....
NYLXS is a trademark of NYLXS, Inc
|
|
