Creating a robots.txt file
The easiest way to create a robots.txt file is to use the Generate robots.txt tool in Webmaster Tools. Once you've created the file, you can use the Analyze robots.txt tool to make sure that it's behaving as you expect.
Once you've created your robots.txt file, save it to the root of your domain with the name robots.txt. This is where robots will check for your file. If it's saved elsewhere, they won't find it.
You can also create the robots.txt file manually, using any text editor. It should be an ASCII-encoded text file, not an HTML file. The filename should be lowercase.
Syntax
The simplest robots.txt file uses two rules:
User-agent: the robot the following rule applies to
Disallow: the URL you want to block
These two lines are considered a single entry in the file. You can include as many entries as you want. You can include multiple Disallow lines and multiple user-agents in one entry.
What should be listed on the User-agent line?
A user-agent is a specific search engine robot. The Web Robots Database lists many common bots. You can set an entry to apply to a specific bot (by listing the name) or you can set it to apply to all bots (by listing an asterisk). An entry that applies to all bots looks like this:
User-agent: *
Google uses several different bots (user-agents). The bot we use for our web search is Googlebot. Our other bots like Googlebot-Mobile and Googlebot-Image follow rules you set up for Googlebot, but you can set up specific rules for these specific bots as well.
What should be listed on the Disallow line?
The Disallow line lists the pages you want to block. You can list a specific URL or a pattern. The entry should begin with a forward slash (/).
To block the entire site, use a forward slash.
Disallow: /To block a directory and everything in it, follow the directory name with a forward slash.
Disallow: /junk-directory/ To block a page, list the page.
Disallow: /private_file.htmlTo remove a specific image from Google image search, add the following:
User-agent: Googlebot-Image
Disallow: /images/dogs.jpg To remove all images on your site from Google image search:
User-agent: Googlebot-Image
Disallow: / To block files of a specific file type (for example, .gif), use the following:
User-agent: Googlebot
Disallow: /*.gif$To prevent pages on your site from being crawled, while still displaying AdSense ads on those pages, disallow all bots other than Mediapartners-Google. This keeps the pages from appearing in search results, but allows the Mediapartners-Google robot to analyze the pages to determine the ads to show. The Mediapartners-Google robot doesn't share pages with the other Google user-agents. For example:
User-agent: *
Disallow: /folder1/
User-agent: Mediapartners-Google
Allow: /folder1/Note that directives are case-sensitive. For instance, Disallow: /junk_file.asp would block http://www.example.com/junk_file.asp, but would allow http://www.example.com/Junk_file.asp.
Pattern matching
Googlebot (but not all search engines) respects some pattern matching.
To match a sequence of characters, use an asterisk (*). For instance, to block access to all subdirectories that begin with private:
User-agent: Googlebot
Disallow: /private*/To block access to all URLs that include a question mark (?) (more specifically, any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string):
User-agent: Googlebot
Disallow: /*?To specify matching the end of a URL, use $. For instance, to block any URLs that end with .xls:
User-agent: Googlebot
Disallow: /*.xls$You can use this pattern matching in combination with the Allow directive. For instance, if a ? indicates a session ID, you may want to exclude all URLs that contain them to ensure Googlebot doesn't crawl duplicate pages. But URLs that end with a ? may be the version of the page that you do want included. For this situation, you can set your robots.txt file as follows:
User-agent: *
Allow: /*?$
Disallow: /*?The Disallow: / *? directive will block any URL that includes a ? (more specifically, it will block any URL that begins with your domain name, followed by any string, followed by a question mark, followed by any string).
The Allow: /*?$ directive will allow any URL that ends in a ? (more specifically, it will allow any URL that begins with your domain name, followed by a string, followed by a ?, with no characters after the ?).
WEB DESIGN EXAMPLES>> 
To request more information click here to fill out our form.
Often simple things really matter. And the use of keywords in linking text is one of those simple things that turn out to be hugely important. Linking text, sometimes called anchor text, is the clickable text that takes people to another page or resource. You need to choose that text carefully.
If I was refer people to this article I could use the linking text, ‘Latest article’: the text is meaningful, clear and wouldn't need to change every time I published a new article. However, I'd be missing out on some important opportunities. Using the actual title of the article, ‘Keyword phrases in linking text’, is a much better option.
-
Links ARE Usability
Foremost, links provide simple navigation between documents on a website. But in addition to regular site navigation, links dramatically increase the usability of site information since words within an article can be linked - or extended out in a tree-like manner - to many other documents of importance, like definitions, resources and items of interest, on-going, without interrupting the flow of the originating concept.
Links make your site - and the internet - more usable, and useful.
-
Links Create Direct Traffic
Clicking a link from a trusted source is one of the most popular ways for people to find a new website. If a user is on a website they trust, they are likely to follow links that it recommends - or at the least, users will follow links that make the destination site appear to provide some valuable information.
More links = more traffic.
-
Links Build User Trust in You
If users see a link to you on a popular or respected site, their level of trust in what you say or what you offer is likely to be higher. MySpace generates more traffic to eBay and Amazon than MSN search does - because navigating through MySpace is about trusting what 'friends' recommend. Blogs create this same link-trust factor. Blog readers have a certain 'trust' in the author.
Word-of-mouth is the best advertising there is.
-
Links Build Search Engine Trust in Your Site
There is no doubt that link score is one of the most important factors the major search engines consider when determining where websites are ranked. The reasoning is simple: if you have many links pointing to your site from other sites, your content must contain quality, important information. A link is like a vote. Although links from important 'authority' sites carry more 'vote weight', ultimately, total 'link/vote score' is one of the heaviest algorithmic factors considered in rankings.
Sites with links from trusted sites are 'trust-worthy'.
-
Links Tell Search Engines and Users What Your Pages Are About
To determine what a web page is about search engines look at a few critical elements on a web page, but just as important (if not more) is the text used in links that point to your page - from within your site and from outside sites. E.G.: If a page has 10 links from outside sites, and each link uses the word 'pizza' as the link text, it stands to reason that the page is about pizza. You could probably determine the page topic without even looking at the page. And, most likely, the link text is why users clicked the link in the first place.
If you could describe yourself in two words, what would they be?
-
Links Establish Your Site in 'Communities'
Typically, the types of sites that easily offer and create links to your site are community type sites that are updated on a regular basis, either by a main administrator, or by users themselves. Examples are; blogs, forums, wikis, MySpace, and other social networking sites. See MySpace example in point #3. Communities are based in trust.
"Never doubt that a small group of thoughtful, committed citizens can change the world. Indeed, it is the only thing that ever has." - Margaret Mead
-
Links Are Free Advertising
Unless you've paid for your link - it is free advertising for you.
That was simple, eh?
-
Links Are Trackable and Provide Market Intelligence
There are a few methods of determining who links to you, and this information can be very valuable in providing market intelligence such as;
- competitor strengths/weaknesses (find out who links to them, request a link)
- consumer/user opinion (what are people saying about you/competitors on blogs/forums)
- potential suppliers (industry insiders who liked/linked to your site)
- source of testimonials
- reveals potential PR hazards (negative posts/links)
- reveals strategic partnerships (next point)
To determine who links to you, you can first check your traffic logs for 'referrers'. Once seeing who has generated some traffic for you, you can find out who the search engines have listed that link to you. Here's how you can check:
- at Goolge, search: link:www.yoursite.com
- at Yahoo and MSN, search: linkdomain:www.yoursite.com
Find out who, what, where, when, why, how - and use it.
-
Links Reveal Strategic Partnerships
Upon reviewing your traffic logs or doing a simple link check on any of the major search engines (as mentioned above) you may encounter some sites that link to you that can reveal sources of mutual benefit, such as;
- marketing partnerships (develop reciprocal ad campaign)
- service or product suppliers
- potential employees
- potential contractors
- or for support/education/help
"There's GOLD in them there hills..."
-
Links Are Assets - They Add Value to Your Property
Your website is your internet property, and it has value. What gives your website saleable value (typically) is the amount of traffic the site can generate. As we've shown, links are one of the most important factors in generating substantial website traffic, search engine rankings, trust, publicity, market intelligence and more. A substantial link network offers significant value to the overall package when determining a website's value.
The marketplace creates the asset, and its value
To request more information click here to fill out our form.
We have helped dozens of small businesses and start up companies to get their
products and services noticed so that they can achieve success they have planned
on for themselves and their families.
We ARE A FLORIDA BASED DESIGN STUDIO, SERVING ALL OF THE USA.
OUR TOLL FREE # IS: 888-808-8449 or FILL OUT OUR CONTACT FORM
If you would like to take your web site to the next level then give us a call:
Logo Master Web Design Services - (561) 624-9982
Logo Master web design services
can help design your business web sites,CD-ROM presentations, logos and graphics. We design Flash animations, scan photos and make digital photo alterations. Logo Master can also help you with search engine optimization , web hosting, domain name registration, video and audio streaming from your web site or CD or DVD.
To access our web site properly please click on the button to upgrade your browser.
We're listed in Palm Beach Gardens web design section of Florida web design
directory at Finders - US web design directory
