Everything You Need to Know About Search Engines

Search engines make our online lives much easier - instead of having to remember exact web addresses, or even exact names of business or services we’re looking for, we can go to a search engine and simply type in a word or phrase related to what we are looking for. The result is a host of information related to whatever we typed in. But how does that information get populated?

How do search engines work?

A search engine has three primary functions.

Crawling is the first step. This means the search engine has scanned the website and collected as much information as it can from each page. It has scanned titles, images, keywords, linked pages, and other relevant things to gather the information needed.

Search engines use what are referred to as spiders, which are automated bots, to visit pages as quickly as possible. Today, Google bots can read thousands of pages a second. They use page links to figure out where to go next. So, the spider will collect all the links of a page to create its list of pages to visit next. This repeats, and the crawling bots will revisit past pages to see if any changes may have occurred.

Some people like to compare the internet to a big subway system - each stop is a unique page, either a webpage or another type of file on a website. The search engines use the links to crawl through all the stops along the way. Non-text files, such as videos, images or audio files, cannot be interpreted unless meta tags are attached. Google offers a complete list of the types of files that can be indexed here.

Once the bots find a page, they then decipher the code in them, and stockpile this information to be used later when someone submits a search query.

The History of Search Engines

Prior to search engines, the web consisted of File Transfer Protocol (FTP) sites. Users would navigate them to find specific files. But as internet usage increased, there needed to be a way for users to quickly and easily find the information they needed. Search engines were created to make it easier to navigate.

Archie was the first such engine created. In 1990, Alan Emtage, a university student in Montreal, created Archie as a school project. Archie essentially created an index of computer files stored on FTP sites. In 1991, another college student named Mark McCahill created Gopher, which also searched plain text references in files.

Mosaic followed in 1993, but these still didn’t look quite like what we’re used to today. Around the same time, Wandex was the first technology to crawl the web, indexing and cataloging along the way. In 1994, WebCrawler began to index the full texts of sites, rather than just web page titles.

Yahoo has been around since 1994, and was originally a directory of sites cataloged by human editors. In 2002 it shifted to be crawler based, and in 2004 it created its own search index.

Google wasn’t born until a few years later, in 1997. As Stanford PhD students, founders Larry Page and Sergey Brin started researching search engines based on relevancy ranking. They believed that search engines should be able to analyze and then rank websites based on how often certain search terms appeared.

Other search engines have come and gone over time, and a few are still active today. Windows Live (formerly MSN Search) and Bing are also used today. But, according to Net Market Share, Google accounts for nearly 73% of search engine users.

Anatomy of a search engine page

In short, a search engine is a software program or script that searches for keywords, and then offers the user results based on those keywords. How search engines are set up to operate is an important way to learn more about what you need to do to operate a successful SEO campaign.

The search begins when a user types in a search query. Google bots have already indexed and canvassed the internet, storing the information you need. Google will then look up the information you searched for from its vast index of information found from crawling the internet.

Since Google is the most commonly used search engine, let’s take a look at what a simple Google search is like:

When your Google returns your search, you’ll see something much like the above. So, what exactly are you looking at?

The areas at the top, as well as the bottom, consist of ads, or sponsored results. We’ll explain these in a bit more detail in a bit.

Next, you’ll find Google Local Pack results. The results found here are localized to the location of the person searching and usually consist of the three most relevant businesses in that area. Sometimes, you can find ads in the local pack results too.

Finally, the organic results will appear. These are natural and unpaid listings that appear when your website is found to have relevant, quality content that matches the search. Good search engine optimization can help you show up at the top of this list.

Sponsored results vs. Organic results

When Google returns your search, you’ll find two different types of results. We refer to these as organic or natural results, and paid or sponsored results.

Organic results will be the ones Google finds based on relevance. Paid or sponsored results are those that are paid for by a company. They are essentially advertisements that a company has paid to have show up when internet users search for certain terms. You’ll be able to tell the sponsored results from the organic ones because they will typically be set apart from the organic results.

Sponsored posts are auction-based, so the location your ad will appear will depend on your bid on the keyword you want to appear for. If a company has a higher bid per keyword, they will appear before yours. But these posts aren’t only decided by bid amount, but also on other factors such as quality score, CTR and relevancy.

Research shows that the majority of internet users prefer organic results. In fact, 70% of the links users click on are organic, and 70-80% of users ignore paid posts and go directly to the organic listings (https://www.imforza.com/blog/8-seo-stats-that-are-hard-to-ignore/). However, if you are selling a certain product or service, paid ads may work better in your favor. “High commercial intent” searches outperform organic results. Commercial intent refers to transactional searches - the keywords the searcher uses signal a strong intent to conduct a transaction. This could mean the searcher wants to buy something, learn more about a service, or something else that could lead to a future sale.

Sponsored posts push the organic posts down, meaning a user will need to scroll to find the organic posts. On a mobile device, which is more and more becoming how people search, users typically need to scroll quite a bit to find the organic results.

But which one is better? It isn’t necessarily a one or the other - in a good SEM program, organic and sponsored posts can work in tandem. While a sponsored post won’t make your organic post appear higher in organic searches, the two can complement each other. Utilizing both will provide you with more real estate on the search engine, making your name appear more often in front of users.

But, there are some things to keep in mind when it comes to search engine users. Because paid or sponsored results are actually ads, they often work better for shoppers, rather than an internet user who is doing research. Internet users who are gathering research may most likely prefer organic search results. Organic search results are important because they will return content relevant to what the user was searching for.

How do people interact with search engines?

In order to have a successful online marketing program, it is important to think about internet users. What is your target market looking for, and how will they find you?

MOZ.com, one of the leaders in search engine marketing information, explains that there are typically three ways in which users interact with a search engine. These three types of queries are:

The search engine’s job is to then determine whether a user will be happy with the information they find on your site - will it meet their search query? The search engine wants to provide relevant and useful information to its users, so your job is make sure your customers will be happy with the information returned to them. That means you need to build your website and fract your content with users in mind. When building your website, if you think first about what a user wants, you’ll also be satisfying the search engines.

So, build for users first and then Search Engines when you build or optimize a site. Each search begins with the intention of seeking information (either finding a place, finding a product, finding an answer to a question, etc..) The site that fulfills this quest for information will most certainly rank the best (at least it deserves to rank the best) and search engines such as Google are getting smarter everyday and can easily identify which content or site is worth ranking higher or not. Relevancy is the key for search engines as it is directly connected to how much users will be happy with the returned results.

So a user will have a need that can be met by searching for it online. Then, the user will formulate the search query - it can be a keyword or phrase, or a question that includes keywords. One the query is submitted, the search engine uses its warehoused information to return relevant search results. The user browses the results for the best match, and clicks on that. If the user is unsatisfied with the information, they’ll return to begin a search all over again. 75% of users won’t scroll past the first page of results (https://www.imforza.com/blog/8-seo-stats-that-are-hard-to-ignore/).

Being listed at the top of the results gives a business the most visibility. The top ten results on Google result in an over 50% click through rate (https://www.internetmarketingninjas.com/blog/google/announcing-2017-click-rate-study/). But not only will users see your listing more, they will also trust it more. If it shows up in the top few results, users trust its relevance to their search query.

And that leads us to section 2 - is search engine optimization and marketing really necessary?