On a windy day last summer, Thane Paulsen first glimpsed what he believes will be the future of information retrieval, and it was wearing a propeller beanie. Several beanies, in fact. Mike Bergman, co-founder and chairman of VisualMetrics Inc., had brought some of his key developers to Paulsen's Sioux Falls, S.D., offices, intent on demonstrating a new approach to Web searching, and they were dressed to impress. "I looked out the window," Paulsen says, "and those propellers were spinning so fast it looked like the whole gang was about to take off."
Whether that's an ideal metaphor or just wishful thinking remains to be seen, but Paulsen is betting that VisualMetric's technology will fly. His firm, Paulsen Marketing Communications Inc., has teamed with Bergman's company to create BrightPlanet.com LLC, the latest entrant in the ongoing battle to build a better search engine. While Yahoo, Lycos, and similar search engines may be among the most established sites on the Web, they and their competitors are nonetheless feverishly developing new techniques for scouring the Internet to find the exact information a user requires. And companies are discovering that better search engines are critical to their own Web sites, whether devoted to E-commerce or designed to share information across the enterprise.
"There's a renewed interest in search," says Paul Hagen, an analyst at Forrester Research Inc., in Cambridge, Mass. In the rush to create a Web site, he says, companies often give the search function little thought. They may simply bolt on a low-cost piece of utility software or a database query tool that's not intended for efficient Web searching. That means that the most relevant information goes undiscovered, or is included in a vast number of "hits" that users must sort through one at a time. The problem may get worse: not only are new pages being added to the Web at an astounding pace, but companies are changing the way they store information, relying on databases rather than static Web pages, which can frustrate most search engines.
That's where BrightPlanet (www.brightplanet.com) sees opportunity. It claims that the information stored in databases, which it dubs the "Deep Web," is 500 times larger than what can be found on existing search engines. For computer users who get thousands of hits on even the most arcane queries, this may not be good news. But BrightPlanet says its LexiBot technology allows users to formulate specific searches and find just what they're looking for. This requires special software, priced at $89.95 and available to anyone, although BrightPlanet expects the bulk of its business to come from private agreements with corporations and Web exchanges eager to index and search specific realms of data.
"An agricultural B2B site," Paulsen says, by way of example, "may find tremendous value in the information stored in USDA databases, but most search engines can't dig in and find it and present it to users. Ours can."
That sort of corporate customization is a market many search-engine firms are pursuing. AltaVista Co.'s Business Solutions Group (www.altavista.com), for example, licenses software to E-tailers and other entities that provides customizable indexing and high- powered search functionality. One customer, Buy.com Inc., sells almost 2 million items, and wants customers to find what they need without frustration.
"The prices on some of our goods, like PCs, change constantly," says Marc Raygoza, director of Web applications for the Aliso Viejo, Calif., firm. "We were using a database query tool, but it couldn't handle the job, so an investment in a search engine like AltaVista was essential." Hagen says companies can expect to spend in the mid-to-high six figures for a robust search engine and the associated development and integration work, an investment he claims will pay for itself many times over in increased sales and the ability to analyze visitors' site usage.