Milin Enterprists
Milin Enterprises
Chapter 7

Submit, Spider-Watch,
List & Rank

STEP 1: Submit Your URLs
Submitting your site to the engines is a tedious affair, because each engine likes to receive submissions in different ways
  • they each have their own submission pages
  • some only want to receive your home page. Some let you submit every page of your site.
  • some allow you to submit many pages in a day, others only one. And, to make it all nastier, they change policies from time to time!
Still, itís a necessary task. Yes, there are submission services and software. But, unless I really know someone, I trust important stuff like this only to myself. Iíd suggest the same for you.

Luckily, a mere few major free Search Engines account, either directly (through their own search services) or indirectly (through partner sites), for about 95% of your traffic. Letís stay on the right side of the 80-20 Rule (actually 95-5 here!) and only submit to them.

Go to the submit URLs of the following engines and research their current submission policies. Then start submitting

There is no need to submit to any other engines, not even
  • AOL Search 

  • your listings are displayed automatically in AOLís Search Engine when you bid on keywords at Overture which uses Inktomi for its backup search results, so youíre covered when you submit to Inktomi via HotBot.
  • MSN Search 

  • you are listed automatically in MSNís Search Engine when you submit to Inktomi. You submit to the Inktomi index through HotBot or through the paid inclusion program available at or
  • Netscape Search 

  • you are listed automatically in Netscapeís Search Engine when you bid on keywords at Overture which uses Inktomi for its backup search results, so youíre covered when you submit to Inktomi via HotBot. Also, Netscape use Google as a secondary Search Engine. So youíre covered there, too. 
Overtureís network partners often only display between the first and third place listings.
You can no longer batch submit URLS to AltaVista. It has come up with a creative way to force a human to actually go the submission page in order to submit, you have to enter the 8 characters from a displayed graphic into a Submission Code box, then enter your URL and then press the Submit button.
Check it out yourself...

The big problem with all submission software? They donít tell you the policies of each engine (how many pages are OK to submit, how often). And, of course, they donít tell you whether youíve been spidered, etc. So itís easy to happily batch submit your URLs until you run afoul of the engines by violating their policies.

STEP 2: Track Enginesí Spiders
Even after you do submit, some of the engines take weeks, even months, to send their spiders over to your site to bring back the goodies. The pros donít wait around. They submit. Then they check their siteís log files, looking to see which of the major engineís spiders have visited, and which pages they took back to their mother ships. If they donít see an engineís spider within a certain amount of time after submitting (varies for each engine), theyíll resubmit, according to each engineís acceptable limits.

Yes, tracking spiders is rather complicated, and far more tedious than submitting. Here are several resources to get you started

STEP 3: Track Your Listings
Site Build It!'s World Submitter will resubmit automatically if any Search Engine doesn't index within a reasonable amount of time after a spider visit or if an engine ever drops you from the index.

SBI!'s ListChecker watches for when each engine lists each of your pages that have been spidered or indexed. When it finds your URL in the index of an engine, it reports this date in the Submit-Spider-List Report. 

Youíve been spidered! That means youíre in, right? Not really. It just means that that engine knows about you. Now you have to watch for when each engine lists each of your pages (that have been spidered). The pros call this being indexed. They track each engine to see when their pages get indexed (the pages are officially in the database, but not yet ranked).

For example, at AltaVista, enter your URL ( into the search box. It will show you every page from that domain that it has indexed. Yes, there is a way to check this at each engine. Unfortunately, every engine has a different protocol. For more info on how to track which pages have been indexed at each engine

STEP 4: Track Your Rankings
Great! Youíre indexed. That means people are finding you, right? Geez, I hate to be a party-pooper, but no. Once youíre indexed, thereís a lag before your pages actually show up in search results. And even then, some of your pages will score better than others. So now what do the professionals do? They track their ranking for all their keywords, at each engine. There are several ways to track how your pages rank. Hereís a nice resource, organized to help you do it manually.

Even so, tracking your rankings manually takes time, too much time. To make things worse, as youíll see in chapter 8, you have to do it over and over again, each time you make a change to a Web page,  re-submit, spider, index, rank, for each page, for each engine.

Luckily there are services or software that will do it for you. Hereís the best of a not-quite-perfect bunch (Iíve left out several abysmal services)

  1. Position Agent (part of Submit It!) 

  2. At $49 per month, itís pricey
    Take the free trial at the bottom of the page. Youíll see that not all the engines are covered. And, if you check, the results are not correct for all the engines.
  3. Top-10 

  4. donít be put off by the long form that you must fill out to get the free report. Itís a good report, mostly accurate (only missed one engine by a few spots), and is almost complete (only missing Google)
  5. AgentWebRanking Suite

  6. as opposed to the above two, this is client-side software (i.e., it runs on your computer instead of on the Internet). Itís free. And many people speak well of it. Download and try it for yourself
  7. WebPositionGold

  8. probably the Cadillac for ranking. Also client-side software. But itís expensive, and very complicated to use. This kind of tool is best in the hands of consultants who want to build traffic for businesses.
Site Build It! has some awesome reporting tools. After a page has been submitted, spidered and indexed, SBI! will start to report on how that page ranks for its keyword.

The Keyword Ranking Report will tell you how each page ranks during a search for its Specific Keyword. If the ranking is beyond the top 30 for many of the engines, you can click on the "Analyze It!" button. This function will give you suggestions on how to optimize your page.

The Keyword Searches Report will tell you which keywords people are using to find you, will show you how many times you were found with each of these keywords, and will provide a direct search-link so that you can see how you rank at a particular engine for a particular keyword. 

Yes, it's a quagmire. Tracking how all your pages rank is definitely tedious stuff.

So why do all the pros do it? Because itís so important to know where you stand. Every situation is different. You have to get the lay of the land, your land, and see where you stand in it. For example, you might be writing about succulent plants, there is an entire world of related Web sites, directory listings, links flying around all over the place and this set of circumstances is different than the one for fashion.

Suppose you wrote two near-identical pages. The only difference is that you replaced the word ďfashionĒ with ďcactusĒ (you should never create such search and replace pages, of course, this is an example to make a point). Your fashion page ranks Page 1, #1 for every Search Engine. Does that mean that your cactus page will also be Page 1, #1? Nope. Because the Web world for fashion is totally different from that of the cactus. The point of tracking how you rank is to get the lay of your land.

Once you see how your pages fit in your particular Web world (and it likely wonít be great at first), you start tweaking. Weíre getting a bit ahead of ourselves, but, as weíll see in the next chapter of this discussion, the key to building traffic through the free, major Search Engines, is to try different things until you get the feel for what ranks highly for your particular theme.

Thatís why I recommended in chapter 6 to experiment with different numbers of Specific Keywords, as well as General Keywords and common synonyms in titles, headlines, etc. If you have many differently constructed pages, youíll notice which pages rank high and which donít. And that gives you the basis to improve the ones that donít. It takes some time and effort. Thatís why the major Search Engines are not really free. They do cost you something, time, after all, is money. The payoff, however, is terrific, substantial, nearly free (once you work out what succeeds for you), sustained, and targeted traffic.

So hereís what youíve done so far. In chapter 6, you built a Theme-Based Content Site, full of Keyword-Focused Content Pages. As weíve discussed, youíve overdelivered on content for your human visitors. And, for the engines, you have also varied your formulas to see what works.

Earlier we talked about how important it is to have a feedback loop from building to tracking. Now that you have feedback from tracking, itís time to take traffic-building action, which brings us to

BIG TECHNIQUE #1 The Free Major Search Engines

This is still the best way to build traffic on the Net, if you know what youíre doing. And you soon will!

<- Previous Page Previous Page
Table of Contents
Next Page Next Page ->
©Milin Enterprises, San Antonio TX, 2001-2003