Refactoring, cleanups and documentation

Author: jimsmart

README.md

@@ -10,11 +10,65 @@
 grobotstxt is a native Go port of [Google's robots.txt parser and matcher C++ 
 library](https://github.com/google/robotstxt).
 
-- Direct function-for-function conversion/port.
-- Preserves all behaviour of original library.
-- All 100% of original test suite functionality.
-- The code is not pretty :/
-- But the tests all pass! :)
+- Direct function-for-function conversion/port
+- Preserves all behaviour of original library
+- All 100% of original test suite functionality
+- Minor language-specific cleanups
+
+The original package includes a standalone binary but that has not yet been ported as part of this package.
+
+## Installation
+```bash
+$ go get github.com/jimsmart/grobotstxt
+```
+
+```go
+import "github.com/jimsmart/grobotstxt"
+```
+
+### Dependencies
+
+- Standard library.
+- [Ginkgo](https://onsi.github.io/ginkgo/) and [Gomega](https://onsi.github.io/gomega/) if you wish to run the tests.
+
+## Examples
+
+```go
+import "github.com/jimsmart/grobotstxt"
+
+// Fetched robots.txt file.
+robotsTxt := `
+    # robots.txt with restricted area
+
+    User-agent: *
+    Disallow: /members/*
+
+    Sitemap: http://example.net/sitemap.xml
+`
+
+// User-agent of bot.
+const userAgent = "FooBot/1.0"
+
+// Target URI.
+uri := "http://example.net/members/index.html"
+
+
+// Is bot allowed to visit this page?
+ok := grobotstxt.AgentAllowed(robotsTxt, userAgent, uri)
+
+```
+
+Additionally, one can also extract all Sitemap URIs from a given robots.txt file:
+
+```go
+sitemaps := grobotstxt.Sitemaps(robotsTxt)
+```
+
+See GoDocs for further information.
+
+## Documentation
+
+GoDocs [https://godoc.org/github.com/jimsmart/grobotstxt](https://godoc.org/github.com/jimsmart/grobotstxt)
 
 ## Testing
 
@@ -26,6 +80,16 @@ For a full coverage report, try:
 $ go test -coverprofile=coverage.out && go tool cover -html=coverage.out
 ```
 
+## Notes
+
+Parsing of robots.txt files themselves is done exactly as in the production
+version of Googlebot, including how percent codes and unicode characters in
+patterns are handled. The user must ensure however that the URI passed to the
+AgentAllowed and AgentsAllowed functions, or to the URI parameter
+of the robots tool, follows the format specified by RFC3986, since this library
+will not perform full normalization of those URI parameters. Only if the URI is
+in this format, the matching will be done according to the REP specification.
+
 ## License
 
 Package grobotstxt is licensed under the terms of the

doc.go

@@ -1,4 +1,4 @@
 // Package grobotstxt is a Go port of Google's robots.txt parser and matcher C++ library.
 //
-// See https://github.com/google/robotstxt
+// See: https://github.com/google/robotstxt
 package grobotstxt

examples_test.go

@@ -0,0 +1,40 @@
+package grobotstxt_test
+
+import (
+	"fmt"
+
+	"github.com/jimsmart/grobotstxt"
+)
+
+func ExampleAgentAllowed() {
+
+	robotsTxt := `
+	# robots.txt with restricted area
+
+	User-agent: *
+	Disallow: /members/*
+`
+	ok := grobotstxt.AgentAllowed(robotsTxt, "FooBot/1.0", "http://example.net/members/index.html")
+	fmt.Println(ok)
+
+	// Output:
+	// false
+}
+
+func ExampleSitemaps() {
+
+	robotsTxt := `
+	# robots.txt with sitemaps
+
+	User-agent: *
+	Disallow: /members/*
+
+	Sitemap: http://example.net/sitemap.xml
+	Sitemap: http://example.net/sitemap2.xml
+`
+	sitemaps := grobotstxt.Sitemaps(robotsTxt)
+	fmt.Println(sitemaps)
+
+	// Output:
+	// [http://example.net/sitemap.xml http://example.net/sitemap2.xml]
+}

robots_cc.go

@@ -595,7 +595,8 @@ func NewRobotsMatcher() *RobotsMatcher {
 	return &m
 }
 
-// init initialises the userAgents and path for this RobotsMatcher.
+// init Initialises next path and user-agents to check. Path must contain only the
+// path, params, and query (if any) of the url and must start with a '/'.
 func (m *RobotsMatcher) init(userAgents []string, path string) {
 	// Line :478
 	m.path = path