Website crawler and fulltext indexer.

Driller is a command line Ruby based web crawler based on Anemone. Driller can crawl website and reports error pages and slow pages and generates HTML reports.

This is a crawler framework.

A simple directory crawler DSL.

SemanticCrawler is a ruby library that encapsulates data gathering from different sources. Currently microdata from websites, country information from Freebase, Factbook and FAO (Food and Agriculture Organization of the United Nations), crisis information from GDACS.org and geo data from LinkedGe...

BFS webcrawler that implements Observable

JavaScript enabled web crawler kit

A web scrawler to get a Marmiton's recipe

your friendly neighborhood web crawler

Gem for crawling data from external sources

render_static allows you to make your single-page apps (Backbone, Angular, etc) built on Rails SEO-friendly. It works by injecting a small rack middleware that will render pages as plain html, when the requester is one of the most common crawlers/bots out there (Google, Yahoo Baidu and Bing)

This little library helps people download images from different subs much easier. It's actually like a crawler for the images posted on a subreddit. Actually, it's a great tool to have your favorite memes locally!

webget gem - a web (go get) crawler incl. web cache

s7o crawler optimized for programmer happiness and sustainable productivity.

Crawl the senegalese web, looking for jobs using the excellent wombat gem

A simple solution to provide on-demand service access (e.g. port 80 on webserver), where a more robust and secure VPN solution is not available. Essentially, it is a more user-friendly form of "port knocking". The original proof-of-concept implementation was run for almost three years by ...

This gem crawls the latest CircleCI artifact file you specified. For Example, you can get the result JSON of simplecov.gem etc.

This gem helps Crawler Writers to interact with the PromoQui REST API

With just a few lines of code, developers can effortlessly integrate this gem into their projects, enabling seamless retrieval of page titles from HTML documents. Whether you're building web scrapers, crawlers, or any application that requires fetching webpage titles, WebTitle streamlines the pro...

Using paperclip to generate images from sensible attributes like e-mails and telephone numbers, in order to reduce crawler's success