Estibot TotalSearch™ Data

TotalSearch™ is by far the coolest data you will acquire for your domain name. It represents the total number of searches across all known search engines, in the U.S. search market. It is an estimate of the true search popularity for the exact phrase match of the search term that your domain is made of. The data is from 2008, so it is well representative of the current search market. Forget Overture, this is the new benchmark.

TotalCPC™ An up-to-date estimate of the highest advertiser bid for the search term from 2008.

TotalCompetition™ Represents the amount of advertiser competition that exists for the search term, again the data is from 2008.

These data are available for the single appraisal at this point. I will get to work on improving the bulk appraisals, and will include the data soon.

 I first started toying with the idea of automated appraisals around Christmas time 2006. The reason for my interest was that I was fairly new to domaining, and wasn’t sure how to tell which domains had value and which did not. I had hand regged quite a number of domains (as we all seem to do when we first start out), most of which turned out to be what are referred to in the business as crap domain names. Hand regging a bunch of crap names seems to be the baptism of fire we all have to go through before we decide to try and learn a thing or two about domaining. I figured there must exist some general rules for determining domain value. I set out to discover those rules, and somewhere in the process I started thinking, if indeed there are rules, then it should be possible to devise a computer program that would employ those rules to come up with a value estimation. After all, IBM’s Deep Blue beat Kasparov in Chess, surely domain valuation should be simpler than that. 

I searched the Net for domain appraisals, and found some sites that would offer automated appraisals for free. The most popular such sites then were Leapfish and Nameboy. I tested them with a largish number of domain names but wasn’t satisfied with the results they produced. I liked what they’d done, but their results just didn’t compare to known sale prices very well. 

I decided to attempt to make such a program for my own use, to help me make informed decisions about buying and selling domain names. Armed with a knowledge of programming in Basic (from owning a Sinclair Spectrum in the 80’s) and some idea of Pascal (yes, there used to be a programming language by that name), I set out to embrace the task. Pretty soon I found out that my programming skills, such as they were, were horribly outdated and I had to look at other options. I dediced to go mathematical, which would require no programming, just mathematical analysis done by a spreadsheet program. I fired up my Excel and started experimenting. 

I gathered information about the various factors that were generally considered important for domain value, and soon discovered that what makes or breaks a domain name is the quality of the search terms, or keywords, within the domain. I learned about search popularity, term frequency, and also discovered other parameters that were considered important, such as the TLD, length, hyphens, digits and so on. I started gathering that data for a number of domains, and tried to figure out a mathematical method for determining domain value based on those factors. After a few weeks of experimenting I realized I was not getting anywhere. No matter how I tried, the model would spit out seemingly random values, certainly not much better than what I’d seen Nameboy or Leapfish do.

 Thanks to my then girlfriend and now mother of my 4-month-old son (one thing leads to another as they say…), I decided to take another approach. She suggested I try to employ scientific principles in the analysis. I have a fairly extensive training in the natural sciences, with two doctorate degrees, one of them being a Ph.D. in Neuroscience. So I decided to apply what I had been trained to do: find order in apparent chaos - correlations, associations, and ultimately, hopefully, rules and laws that would explain the workings of the domain market. Using the very same scientific principles as I’ve used in my Neuroscience research, I gathered keyword and other data for thousands of domains, and finally after a few weeks I found a way to produce a reasonably good correlation between the characteristics of a domain name and its market value. For the statistically inclined, I reached a p<0.05 for positive correlation between what my model spat out, and the known sale price of domains - in large samples. 

I spent another 3 months honing the algorithm, and in the process made some surprising findings, for instance that the length of a domain name, contrary to popular belief, does not correlate with domain value except in the extremes of the spectrum. I analyzed each factor separately, and all combinations of known factors, and settled on, originally about 8 parameters that could be used for determining value. Since then, the algorithm has swollen and more factors have come into play, but in the end, I was fairly happy with my algorithm as a rough indicator of potential value. It still had severe shortcomings, and would be easily fooled by abnormalities in the search popularity data. I was using Overture data, and as we know, Overture has some serious disadvantages, resulting in overestimations of value for a large number of domain names. 

What I had in my hands was a single mathematical formula for domain valuation. It looked something like this: 

$VALUE=((((c*sqrt(h*i)+E*j+f*k)*SIGN(e-c*l)+ABS((c*m+e*n+f*o)*SIGN(a-b*q)))/2)+r/(g^(g+(g-2)))+(-SIGN(e-c*s)*f*t)+f*u+(-SIGN(e-a*v)*g*w)+h*x+(-SIGN(c-a*y)*e*z)+d*a+c/z)*x/SQRT(g)+100/h 

Just one look at that formula and I knew I was at the end of the road for improving it. I found it hard to edit, as I wasn’t quite sure what was what anymore. As the mathematically inclined will see instantly, I’m no mathematician and decided to pursue other ways to improve the valuation algorithm. 

I needed to learn programming again. 

Since I was going to have to learn programming, the idea came to me, why not learn a language that I could use for other purposes as well, such as making websites. The obvious choice, then, was PHP. After a crash course in PHP programming, I found that I was able to hone the algorithm programmatically, and make the mathematical bit a lot neater. Finally I arrived at an algo that I thought was as good as I could reasonably expect to achieve. It worked for me; even with its limitations, it helped me make sense of domain values. 

By that time I had been hanging out at the NamePros.com domaining forum, and had made many new friends in the world of domaining. I wanted to share with them my creation, such as it was, and decided the best way to do it was to publish it online. With the help of Bill from NameClerk.com and of AvailableDomainNames.com fame, I found the perfect domain name for it, Estimation Bot -> Estibot.com, and finally I had a bunch of friendly NamePros members to alpha test it privately, and, encouraged by their positive feedback, Estibot.com went live in June of 2007.

The original website development was done by eTechSupport.net for about $800. Since then I have learned enough PHP to have redone everything by myself. Only the center column colors are what’s left of the original website. The new look is in part thanks to another NamePros member, Bannen, who designed the new logo (asking for absolutely nothing in return) and also has designed a whole website layout for me, which I haven’t had a chance to implement yet. If you haven’t seen Bannen’s Ice Photography yet, you’re missing on some of the more beautiful things in life - go check it out now!

So you see, Estibot was never intended to become a website, or otherwise public for that matter - I made it for my own use, but looking back, releasing it was the right thing to do. The reception was cautiously positive, but, predictably, it got mixed reviews in the domaining peers. Some loved it, some absolutely hated it. The consensus, however, seemed to be that it was the best auto-appraisal system to date, for what that was worth. Knowing well the limitations of automatic appraisals - or human appraisals for that matter - it was good enough for me, and I decided to keep at it, keep improving it. 

The last changes to the algorithm were done around Christmas time 2007, a full year after the initiation of the project. I had spend countless hours on this thing, I decided it was time to focus on developing the website more and leave the algorithm at that for now, because it had come to the point in complexity where even a small change could have unpredictable results. That is not to say that I won’t try to improve it in the future - I’m just taking a holiday from the mathematical side of it, if you will. 

To my utter amazement, the website got pretty popular, and today it has an Alexa rank of about 18,000, with about 750,000 hits per month. In order to keep it functional, I had to get a dedicated server for it, and added Adsense and affiliate links in order to cover the server costs. Originally I used screen scraping of Overture, Google and other services to get the keyword data, but, having educated myself of good Webmaster practices, opted out of that and started using API access to various services to retrieve the data.

Website growth is a two-edged sword - as the popularity grew, I was starting to exceed the limits my API access quotas, Google even banned the site at some point, and I had to constantly keep tweaking things to keep up. I have since then added paid API contracts to retrieve some of the data. Each “free” appraisal actually costs me money, which is one of the reasons I decided to go premium, but more on that later. 

Some of the more significant later improvements include the website traffic estimation, which I developed - it uses, among other things, paid data retrieval from Alexa. It enabled me to add the “value considering traffic” feature, which I think works reasonably well, although you can’t really valuate a website based on traffic alone. But it’s a nice addition, and the traffic estimation seems to do a fairly good job at estimating external website traffic at ballpark level. 

But Estibot would only be a shadow of what it is now if it weren’t for the support of two very generous members of the NamePros community. The intervention of these men really made all the difference. Call it serendipity if you will, but these two events turned a whole new page in the development of Estibot. First, Justin of NameBio.com (you may know him as Spade from NamePros) offered to give me access to his veritable database of domain sales known as NameBio.com. I confessed to him that I had extracted most of my original data for the analysis from his site, and he offered the whole database for my use, again asking for nothing in return. Estibot is now able to compare entered domains to Justin’s database in realtime, resulting in even better appraisals, and also there is now a “related sales” feature which helps users to instantly see prices retrieved by related domain names. Justin and I are talking about deepening the integration now that the subscription service has been started up.

Another huge turn of events was the offer from another NamePros member, Michael Goldman, to provide me with access to his gigantic database of archived search data, enabling me to offer realtime search popularity data and bulk tools that can retrieve data very fast. I cannot stress how important this was, and I am to this day extremely grateful to this fellow domainer. All Michael asked for in return for this invaluable resource was that I make him a few simple bulk tools to enable him to run bulk estimations and keyword data retrieval. Needless to say, this was a major show of generosity and support for my project. Michael posts interesting domain news at his website, Domain Magnate. 

So you see, Estibot is actually the product of many people - fellow domainers who have supported me, given me money in the form of donations, access to their data, volunteered to test my tools, and publicly supported me on their website, driving more traffic to Estibot. It’s a great community, NamePros, people are ready to help and bend backwards to assist a fellow domainer in his endeavours to make something of value. I hope I can pay these favors forward, as I certainly can’t pay many of them back. 

I hope you enjoyed reading about the history of Estibot, and my take-home message to all is this: network with people, be polite, help each other out and good things will come your way. 

As I said before, success in website development is a two-edged sword. Since Estibot became popular, it has been operating at a loss that even the ads and the user donations could not turn around. Domainers don’t click on ads, donations are few and far between and affiliate marketing ain’t what it used to be. I have a mortgage to pay, family to feed, and I could not justify running Estibot anymore, considering the efforts it demanded from me, time away from my family in the evenings (I have a day job as well, you know), and having to pay server and API access costs out of my own pocket. I had to make a choice: take Estibot down altogether, or make parts of it subscription-based. I decided to try the latter, hoping that it would enable me to keep developing this service, and hopefully if I get enough subscribers, justifying the continuation of the service, I can go back to improving the free tools as well, which is what I originally intended to do. More about the subscription based service in my next entry. Until then, keep on domainin’