The History of Search Engine Optimization

The 1990’s

A screenshot of Tim Berners-Lee’s first page of the internet

Search Engine Optimization began with the birth of the search industry and the need to structure and organize all of the information the World Wide Web had to offer.

In 1993, one of the earliest search engines, Excite, was born, and transformed the way information was categorized through the use of keywords and “keyword crawling.” This is the process through which bots or “spiders” scan millions of web pages for keywords related to user’s queries.

In this model, search results lists were based on a website’s relevance, keywords and backlinks, and to this day, this is still the case.

The First Search Engines

Much of what Google controls now was once divided up by many different search engines. Here are just a few:

• Archie
• World Wide Web Virtual Library
• ALIWEB
• Wandex
• Infoseek
• Yahoo
• Excite

Soon after the inception of search engines like Excite, Archie and ALIWEB, corporate moguls Yahoo (1994) and Google (1997) followed with less primitive, more user-friendly user interfaces. These became far more popular than their predecessors, and Google eventually conquered the search engine market.

Archie’s search query form

However, despite Google’s sophisticated algorithms, many online companies still thought that they could outwit their new system, and Google was faced with legions of outlaw webmasters hell-bent on thwarting their search system. It was very much like the wild west, with Google as acting sheriff.

Black Hat Spamming Tactics

Many companies would employ brute-force spamming tactics to score higher on the search results list. Search engines had not yet adopted adequate rules to govern the hierarchy of search results. Now, what did this mean for users?

• Excessive Backlinks
• Spamming Keywords
• Tiny text, hidden text, hidden links
• Duplicate content
• Spam blogs and comment spam
• Linkfarming

As the SEO community, Moz, puts it, “It was discovered that by taking some rather simple actions, search engine results could be manipulated and money could be made from the internet.”

In this highly lucrative new market, search was “spammy,” more difficult to navigate and filled with unreliable content. Companies who paid the most money to webmasters to manipulate Google’s algorithms scored higher on results lists rather than web pages with the most relevant and reputable content.

During this time, it was up to the user to interpret the validity of their search results. This kind of behavior discouraged users from wholeheartedly trusting the information they were finding on the web, making web navigation a headache.

The Miracle at Stanford University

When Larry Page and Sergey Brin, the founders of Google, set out to create their engine, one of the problems they wanted to address was preventing the corporate takeover of the Internet.

In 1998, the pair published a paper at Stanford titles “The Anatomy of a Large-Scale Hypertextual Web Search Engine,” where they wrote:

“…the predominant business model for commercial search engines is advertising. The goals of the advertising business model do not always correspond to providing quality search to users.”

In that same paper, Brin and Page first mentioned Page Rank, the tech that Google uses today to help rank search results based on quality, and not keywords alone.

This highly sophisticated algorithm sorted search results so efficiently, that all other search engine’s fell behind, and Google’s coup d’etat of the search engine industry was complete. Users migrated to Google at an alarming rate, making them the undisputed king of search.

From their cramped, cluttered dormitory on Stanford University’s campus Brin and Page laid the groundwork for the greatest database of knowledge known to man.

The Early 2000’s

Luckily, there is a bright side to modern SEO, called “white hat” SEO. A website or company employing “white hat” tactics for digital marketing would use the following procedures to promote their products:

• Regularly update their webpages with new, relevant content written for human readers
• Adequate keyword analysis
• Backlinking and linkbuilding to improve link popularity
• Localizing content for users in particular areas
• Personalizing content for user-focused results

Google’s First Algorithm Updates

The early 2000’s marked the attempt by Google and other search engines to make the internet less ad-centric and more user-friendly. Google updates nicknamed “Florida” and “Penguin” oversaw the demise of thousands of spammy, “black-hat” powered websites, which fell from the top of search results lists to the very bottom. Soon, only the most ethical, user-friendly websites topped the lists.

In 2005, Google, Yahoo and MSN united for the “nofollow” attribute which was created in part to decrease the amount of spammy links and comments on websites, especially blogs.

It soon became clear to webmasters and web architects, that in order for their sites to get traffic, they would have to ethically optimize their web pages to score higher on results list. This is when truly ethical Search Engine Optimization began, and the internet was reshaped into what it looks like today.

The Mid 2000’s

Social Media

In 2008, with the emergence of new media such as Twitter (2006), Facebook (2004), and Youtube (2005) Google launched Google’s Universal Search which included news media, images and video in search.

Also in 2008, Google Suggest was launched to improve usability and offered users more relevant content by displaying suggested search options. This is simply the name given to Google’s auto-complete function.

In 2009, Google’s algorithm codenamed Caffeine sped up crawling speeds and expanded search indexes to the point where search results were instant and sprawling with verifiable results.

This massive shift toward user-focused content helped encourage new approaches to SEO tactics. Marketers began using social media to help create a seamless optimized search experience, with credible backlinks from reliable news sources.

In December of 2009, real-time search was released with search results including tweets and breaking news. It became immediately apparent that SEO wasn’t exclusively for webmasters. Journalists, Web Copywriters and Social Media Marketers became heavily involved in SEO to promote and optimize their content.

The 2010’s

The 2010’s saw scores of powerful new updates by Google nicknamed Panda (2011), Penguin (2012), and Payday Loan (2013). These updates utterly altered the face of SEO practices, virtually wiping out things like content farms, subtly “black hat” media websites and even user queries which might bait spammy results.

Big Name Black Hat Scandals

Panda focused on further punishing sites for unethically “gaming” Google’s algorithms. A famous example of this would be the Overstock.com and JC Penny scandals of 2011.

Overstock.com Scandal

At the time, domains ending with .edu had higher reliability under Google’s current algorithm. Overstock.com used this advantage by asking colleges and universities to link to its site and use keywords like “mattress pads” and “notebooks”. These links would cause Overstock.com to jump the rankings when any keywords relating to school supplies were used. Overstock.com eventually ended this practice, but not until Google severely penalized them.

Codename “Penguin”

Google’s “Penguin” updates targeted web pages that used subtle “black-hat” SEO tactics. An example of this would be a web page which professed to answer questions such as “how to burn fat around the mid-section,” which, for the most part, would be informative, but would be intermittently littered with ads and spam that had nothing to do with the pages H1 or heading.

The Knowledge Graph

In 2012, Google officially launched “Knowledge Graph,” a familiar feature you might recognize today as the panels to the right of your search results list. This new feature would be used to display facts about things like:

• Movies, Actors, Directors
• Cities and Countries
• Music
• Planets and Spacecraft
• Politicians and Historical Figure
• Planets and Spacecraft

With this new feature, SERPS (search engine results pages) offered results to questions with direct answers. Google had compiled a library of over 3.5 billion facts, and became much like a modern encyclopedia. This eliminated the need for users to scan through web pages to find trustworthy content to simple, direct questions like: “when was George Washington born?”

The Mobile Revolution

The most important SEO announcement of the 2010’s is probably Google’s mobile update of 2015. This update forced web pages to become mobile-friendly if they wanted to score higher in search results lists. This meant that SEO wasn’t just about keywords and content; responsive web page design was important too.

The Future

Despite the vast expansion of SEO over the past 25 years, the future of SEO seems more promising than ever. Here are just a few promising new developments in Search:

• Voice Search
• Mobile Search
• Chatbots
• Job Search Engines

Voice search is a massive new instrument for search engines and the introduction of hardware such as Amazon’s “Alexa” and Google’ “Home” device have made voice search a part of everyday life for people around the world. Google will doubtlessly optimize and update this new tool in the years to come, and SEO operators will have to follow suit if they want their methods to continue to work.

Mobile accessibility, as in 2015, will continue to be an important part of SEO’s future. Google has reported that almost half of all searches conducted are done from mobile devices like smartphones. Companies will have to optimize their methods for users that are “on the go” and don’t have the time or resources to conduct searches from a stationary device.

SEO will be fully integrated into all assets Google can access and should be a first thought to any online content. Marketers will need to develop a stong, consistent brand presence across all digital channels in order to maintain social influence.

SEO will continue to evolve and fill these needs, possibly leveraging data from external platforms to personalize search and offer additional value. You can prepare your brand for this shift by optimizing new kinds of content (such as in-app content) around how users search. Drive exposure and engagement with concise and direct content, optimized for user intent.

Artificial Intelligence

With the emergence of AI, the digital marketplace is now populated with plug-ins such as chatbots; computer generated “employees” which act like search engine for your websites and products, designed to answer and respond to queries related to your site’s products and services.

The Best Ways to Prepare for the Future of SEO

Overall, the future of SEO is bright, and if anything, is doubling its power and influence using the newest, cutting edge technology. Here are a few ways to plan for the evolution of SEO

1. Continue using ethical, “white hat” tactics which adhere to Google’s code of conduct
2. Stay informed on Google’s updates which will tell you where the market is heading
3. Strive to build strong relationships with users, as they will most likely return to your site if you treat them well and streamline their search/ shopping process.
4. Use Google Adwords and Google Analytics to monitor and promote the growth of your brand
5. Hire digital marketing companies to ethically manage and update your web pages/ social media pages