The History of Search Engines

In the old days of search engines there were only a small variety of ways to search the internet. These were mostly for academics, Multiple Search Engine Logosscientists, students, and government officials searching for or reviewing research in a field. These are not anything most people would be familiar with today. They had names like Archie, gopher, Veronica, and Jughead.

Around 1994 Carnegie Mellon University introduced the first search engine as we would recognize them today with Lycos. WebCrawler soon followed as AOL’s primary search engine. In those days it was he who repeats their keywords the most wins. So if I wanted my page about Halloween to come up first all I had to do was repeat the word “Halloween” the most times on my Lycos Logopage. He with the most repeats wins. This made for some really ugly pages because at the end of the page there was a ton of tiny text with thousands of key word repeats to get high ranks.

Back then it was easier for search engine guys like myself because instead of waiting weeks or months to see a significant change in ranking there was only about a 6 hour to 1 day wait from the time I would do a modification to a page until I saw how well the change treated me in the results for my keyword.

This spawned a new term called a serp or serps (the plural of serp). What it stands for is Search Engine Results Page. Basically that’s the page that appears when you type a term into Google and it sends you the results back.

The next major breakthrough was Yahoo. This was a directory and you went there and listed your site and it was in their directory. They webcrawler Logosaid each site was reviewed by a human but I have my doubts. Yahoo really spelled out its own doom when It started charging to place a listing there. What they didn’t realize was that the content they needed desperately (Web Site Listings) were provided free by webmasters in the hopes that the immense traffic at Yahoo would get them more hits.

Once Yahoo started charging the webmasters, especially the small personal page and small business pages backed away. They couldn’t afford the fees and also if they were providing the content for free why should they pay in addition to be listed. The whole thing was a hugely stupid idea to charge the webmasters.

Later on Excite came onto the scene with a ton of other search engines. I personally liked excite a lot. My search to buy ratio was higher on it then any other search engine. A funny story, well not Excite Logoreally funny because it cost me money, but anyway. For several months I held the number 1 listing for my key word. It made me literally hundreds of dollars a day. Then I just disappeared from the listings at excite. My revenue dropped substantially. As “LUCK” would have it I got a call from a sales rep at excite about a week later. He offered me, I think it was for $3,000.00 a month long sponsored listing at the top of the search results. Similar to what we see on Google now called sponsored results or something like that. So I thought ok I am making about 400 a day extra just from Excite so 400 X 30 days is $12,000.00, so pay 3K to get back 12K, that was a no brainer. Well until the day came that I had been anxiously awaiting, when my listing went live. What a FLOP. Everyone would click on the “Organic” listing but not even 1% clicked the sponsored listing. Needless to say I learned a lot about advertising and PPC ads that day.

Now all these search engines are out there and so the problem started because someone who sold say Fords could type in the word Buick on their web page 10,000 times and come up ahead of an actual Buick dealer. People wanted accurate results not spammed search engine results so the search algorithms were born.

First they went after people who hid their key words by making the background color of their page the same color as the text. This meant Yahoo Logothere might be a lot of dead space at the bottom of the page but it wasn’t that irritating to the web surfers. So the search engines started looking for that and either banning a site from their engine or lowering their rankings substantially.

Things became more complex. Meta tags came into play, word density and frequency, Header lines and descriptive text and even image names and alt tags, along with directory names, page names, and even domain names became all weighted measures in the equations that determined if your business was going to get any exposure or not. A low ranking meant nobody would ever see it.

Google and others then said…..”Hey!!! I know, if a page is really a good page and reflects the content that’s described I bet a ton of people will link to it.” This is where back links came into play. Of google kidcourse this spawned a new type of spam. Paid Link Farms. This could be done with a flat payment or with a series of link trades designed to hide the fact it’s a link swap from the search engines. The links didn’t come from real web pages about a topic but from sites designed to solely host links back to their customers.

Of course the search engines figured this out too and first started assigning a rank or page rank to not only a site but to pages. Some sites (higher page rank) had more weight in the equation. Some would be considered an Expert Site. A few examples of expert sites would be Ebay relating to auctions, WordPress concerning blogs, or any site that for whatever reason seemed to be the defacto standard for their industry. Amazon is a good example.

Next not only was the link weighting a factor but links to your site from “Bad Neighborhoods” (Link Farms, known link sellers, adult sites, etc) devalued your site if there was a link from a bad neighbor to google\ page rankyour website. Other factors came into play like Off Site optimization, press releases, blogs, you tube postings, facebook shares and likes, and even Tweets.

Designing a page to rank high in the search engines is now both an art and a science. Do too much and even though It doesn’t violate any rules you may still get penalized, don’t do enough and you rank low to begin with. So you have a fine balancing act between on page optimization, links, off page optimization, and then factor in social media and also how many clicks you get on your results listing compared to the websites above you and below you in the SERPs.

This is a big hint: If your page title and description is keyword rich and entices the person reading it to click your site instead of a site from someone else then that’s as good as it gets. If your Title and description are enticing to the surfer but not that key word rich that is a good compromise. However, if it is just keyword rich but not that enticing to the surfer then it will become a downfall to your rankings.

People and businesses are constantly experimenting with new types of search engines and new ways to deliver information from the deweysearch engine to the consumer. One of my favorites was the Microsoft experiment with Ms. Dewey. The site is now down but you would type in your query and not only would your results be displayed, courtesy of MS Search but she would also in a very sultry voice give you her comments. The programmers had a great time with this system and pre-programmed in answers to questions that had no bearing on a search but they knew people would ask. She was also not the most patient of search engines and verbally reminded you that she was waiting for your input.

I think we will see this history being written over and over again as time goes by and more new and innovative systems come online.