Add awesome bot (#3662)

* add awesome_bot to check urls

* allow dupes

* make url checking dependent on commit message

* add web.archive.org to whitelist

cuz awesome_bot is having trouble with those links

* fix travis.yaml

* check urls

* need ruby

* another site to avoid

* check urls

* make regex more generic

* sudo deprecated

* make travis linter happy

check url

* try skip results

check url

* timeout was what was killing us

check url

* testing fixing urls

check url

* dont allow timeout

check url

* testing timeout

check url

* put fpb back

check url

* enable to check_urls on arbitrary files

check_urls:free-programming-books.md

* try again

check_urls:free-programming-books.md

* test again

check_urls:free-programming-books.md

* travis doesnt like :

check_urls=free-programming-books.md

* allow arbitrary number of files to be checked

check_urls=free-programming-books.md

* woops forgot to fix the trigger regex

check_urls=free-programming-books.md free-programming-books-en.md

* use for loop instead of xargs

check_urls=free-programming-books.md free-programming-books-en.md

* add documentation

.travis.yml

@@ -1,7 +1,24 @@
-language: node_js
+language: shell
-sudo: false
+dist: xenial
-node_js:
+os: linux
+
+jobs:
+  include:
+    -
+      language: node_js
+      node_js:
         - 6
-before_script:
+      before_script:
         - npm install -g free-programming-books-lint
-script: fpb-lint .
+      script:
+        - fpb-lint .
+
+    -
+      language: ruby
+      if: commit_message =~ /check_urls=/
+      rvm: 2.4.1
+      before_script:
+        - gem install awesome_bot
+      script:
+        - INPUT=$(echo $TRAVIS_COMMIT_MESSAGE | awk -F= '{print $2}')
+        - for i in $INPUT; do echo "processing $i"; awesome_bot $i --allow-redirect --allow-dupe --skip-save-results; done

CONTRIBUTING.md

@@ -118,3 +118,15 @@ GOOD: * [A Very Awesome Book (1970)](https://example.org/book.html) - Jane Roe
 ```
 GOOD: * [Will Be Awesome Soon Book](http://example.com/book2.html) - John Doe (HTML) (:construction: *in process*)
 ```
+
+### Automation
+- Formatting rules enforcement is automated via [Travis CI](https://travis-ci.com) using [fpb-lint](https://github.com/vhf/free-programming-books-lint) (see [.travis.yml](.travis.yml))
+- URL validation uses [awesome_bot](https://github.com/dkhamsing/awesome_bot)
+- To trigger URL validation, push a commit that includes a commit message containing `check_urls=file_to_check`:
+
+```
+check_urls=free-programming-books.md free-programming-books-en.md
+```
+
+- You may specify more than one file to check, using a single space to separate each entry.
+- If you specify more than one file, results of the build is based on the result of the last file checked. You should be aware that you may get passing green builds due to this so be sure to inspect the build log at the end of the pull request by clicking on "Show all checks" -> "Details".