No commit activity in last 3 years
No release in over 3 years
Also notifies you when certain thresholds are hit. I have found this much more accurate than New Relic
2005
2006
2007
2008
2009
2010
2011
2012
2013
2014
2015
2016
2017
2018
2019
2020
2021
2022
2023
2024
 Dependencies

Runtime

 Project Readme

Heroku Mongo Watcher

Command line utility to monitor both your mongo and heroku instances, and to alert you when things are heating up

The Origin

I have a pretty 'spiky' application that can go from having 10_000 requests per minute to 100_000, we need to notified when things are heating up so we can turn the appropriate dials. We found new relic to be too slow (and not accurate enough once throughput levels got high), so we built this.

It needed to accomplish the following:

  • See Mongostats and heroku stats at the same time, the key ones being requests per minute, average response time lock %, and error counts
  • Have multiple ways of notifying stake holders: colors, beeps and email notifications
  • Be able to parse the web log for certain errors and other logged events and aggregate data on them

The output looks like the following ...

|<---- heroku stats ------------------------------------------------------------------->|<----mongo stats ------------------------------------------------>|
| dyno reqs    art   max    r_err  w_err  %err   wait  queue   slowest                  |insrt query updt  flt  lck  lck:mrq qr|qw   netI/O      time      |
   6   3096     57    870      0      2  0.06%      0      0   /assets/companions/50104c|   41     0   79    0 2.2%   0.71    0|0   36k/30k      15:59:29
   6   2705     80   3314      0      0   0.0%      0      0   /assets/companions/50104b|   34     0   67    0   2%   0.74    0|0   30k/25k      16:00:29
   6   2469    122   5708      0      0   0.0%      0      0   /ads/gw/50074348451823003|   30     0   57    0 1.7%   0.69    0|0   26k/22k      16:01:29
   6   2465     89   1347      0      0   0.0%      0      0   /assets/videos/501050b991|   30     0   59    0 1.8%   0.73    0|0   27k/22k      16:02:29
   6   2301     83   1912      0      4  0.17%      0      0   /assets/companions/501050|   28     0   57    0 1.7%   0.74    0|0   25k/21k      16:03:29
   6   1951     64    830      0      0   0.0%      0      0   /ads/gw/50074348451823003|   24     0   45    0 1.4%   0.72    0|0   21k/18k      16:04:29

Legend

dynos Number of running web instances
reqs number of requests per sample
art average request time
max max request time
r_err number of router errors, i.e. timeouts
w_err number of web errros (see below)
%err total errors divided by total requests
wait average router wait
queue average router queue
slowest path of the url that corresponds to the max request time
insert number of mongo inserts
query number of mongo queries
update number of mongo updates
faults number of mongo page faults
lck mongo lock percentage
lck:mrq ratio of lock% to 1000 requests
qr|qw number of mongo's queued read and writes
netIO size on mongo net in/ net out
time the time sampled

Web Errors (w_err)

At least for me, one of the key features is aggregating signals from my web log (I look out for certain race conditions, and other errors). You can can configure the error_messages array in your .watcher file to define which String we should report on.

In concert with that is the print_errors configuration. If set to true it will aggregate and display the errors found (see output above), set to false it will just put the total in the summary row

Prereqs

  1. need to have a heroku account, and have the heroku gem running on your machine
  2. need to be able to run mongostat (e.q. has at least read-only admin access to your mongos), and have mongo installed locally

To install

  1. gem install heroku_mongo_watcher
  2. create a .watcher file (see the examples) in your user directory ~/.watcher
  3. then run bundle exec watcher
  4. Ctrl-C out to quit

Options

  • --print-errors to print a summary of errors during each sample
  • --print-requests to print a summary of requests during each sample

##Experimental

( you probably

  • --autoscale to autoscale dynos
  • --min-dynos=n to set the min default 6
  • --max-dynos=n to set the min default 50

note: you can set these defaults in your .watcher file

Contact

jtushman@pipewave.com