It’s been a busy summer here at ASTOUNDZ! From launching new websites, starting new SEO campaigns, and working with clients to build successful PPC ad campaigns, the team has still found time to research and implement new solutions. The digital landscape only continues to get more competitive, especially in SEO, and our team is always looking for new solutions that give us an advantage. One that we’ve had great success with that we thought needed sharing is what we’ve coined as ‘Instant Indexing’.
So, if you’ve asked yourself recently ‘why isn’t my new website showing up on Google when I look for it?’ then continue reading to learn more about what Instant Indexing is and how it helps new website launches get found and begin ranking faster.
Crawling: The First Step in Getting Your Website to Show Up On Google
Simply publishing a website does not guarantee it will show up on Google Search, or any search engine for that matter. In order for a Search Engine to begin ranking a new website, each page must be scanned first. The technical term for being scanned is called, crawling. Sounds easy enough though, right? Pretty much!
Over the years Google has changed how it crawls websites, in both positive and negative ways for webmasters. For example, as users have migrated over to browsing the web more frequently on mobile devices than desktop computers, Google has shifted to crawling websites on a mobile-first basis. Depending on how mobile-friendly a site is, this change could severely impact how many pages Google crawls.
The Fundamentals of Crawling
Keeping up with how Google changes the way it crawls websites can be very challenging, especially when managing large sites. Despite these changes, there are still some key fundamentals to follow that will help in having a website fully crawled:
- Make Pages Findable: The pages should be linked from somewhere known to Google: from other known pages: from your homepage, from other known pages on your site, from other sites, or from a sitemap.
- Verify & Submit Your Domain: Every search engine provides webmasters a search console/platform to submit key information and check on a website’s health. Verifying your domain and submitting your Sitemap XML file to each search engine will manually trigger a crawl.
- Using The Correct Tags: By default, a website will carry the ‘index’ and ‘follow’ tags, but depending on how a website is launched certain tags like ‘noindex’ and ‘nofollow’ will be brought over from the development environment.
Depending on the size & scope of a website, optimizing the Robots.txt file will help guide how Google crawls a site and prevent crawl budget issues. However, for most sites, this is an unnecessary step to take.
Indexing: Getting Your Website to Begin Ranking on Search
If crawling is passive & (relatively) easy, then Indexing is hands-on and a challenge. For the majority of websites, following the fundamentals listed above will still be enough to have your entire website crawled – the challenge begins after this, as Google can still decide to not index your webpage(s).
Google explains that “after a page is crawled, Google tries to understand what the page is about. This stage is called indexing and it includes processing and analyzing the textual content and key content tags and attributes, such as title elements and alt attributes, images, videos, and more.” In other words, indexing is a 2 step process that starts with the analysis of a webpage and follows with the algorithmic decision to place a webpage in a directory. The challenge for most people will be this last step, as there isn’t much guidance to follow if Google ultimately decides to not rank a page.
Using Search Console to Diagnose Indexing Issues
Let’s say you’ve published your site and see your homepage show up but no other pages. This indicates that you have indexing issues and not crawling issues. The first step to both confirm this and begin understanding what needs to be done, you need to fully verify your domain in Google Search Console.
From there, you’ll be able to see what specific reasons Google has as to why it isn’t ranking your web pages. You’ll most likely notice 2 rows with a good amount of pages listed:
- Discovered – currently not indexed: The page was found by Google, but not crawled yet. Typically, Google wanted to crawl the URL but this was expected to overload the site; therefore Google rescheduled the crawl. You can manually submit these URLs to Google to get crawled and move on to the next step in the process – Indexing.
- Crawled – currently not indexed: The page was crawled by Google, but not indexed. It may or may not be indexed in the future; no need to resubmit this URL for crawling.
In both cases, we see that Google recognizes the web pages but fails to make it to Google’s directory and begin ranking. The question here is how to solve these issues.
An Instant Indexing Solution
Our team has been running into these issues a lot lately, and more specifically having pages listed under ‘Discovered – currently not indexed’. While Google doesn’t simply crawl and index a website overnight, it also doesn’t drag on for weeks on end with no progress made in having pages crawled. As a result, we had to look for solutions to keep our SEO campaigns progressing at a reasonable pace. What we found has been coined as ‘Instant Indexing’ and while the name is a bit of a white lie because it isn’t literally instant, it is very fast.
Below are two websites that had been stuck in the mud for a while.
In the first example, only the homepage was being crawled and indexed by Google. After trying to manually submit different pages and make some tweaks, we looked to this new tool.
Almost immediately after leveraging the Instant Indexing tool, the number of pages listed as ‘Valid’ shot up.
For a larger site this tool also worked wonders:
While all the pages didn’t get listed as valid in one go like the first example, you can clearly see the initial jump it made after leveraging this tool.
My Site is Indexed – Getting to the First Page of Search
While crawling & indexing are 2 of the most important steps in having your website rank, it is still just the starting point for an SEO campaign. Your website has to provide strong content, trust signals (reviews/testimonials), and a healthy amount of links to make the jump up to the first page. If you need that extra help, our team of SEO experts is here to help!
Simply reach out to us and a representative will be in touch with you soon.
