Chris's Blog

Devops Shokunin

Guide to running redis in production

Comments Off on Guide to running redis in production

On my company blog, I wrote a guide to running redis in production environments.

Blogspam Analysis with R Part 1

Comments Off on Blogspam Analysis with R Part 1

This morning while checking the comments on this blog I was surprised at the amount of spam comments caught by the Akismet plugin, so I decided to dive in with some logfile analysis using R to see if I could lessen the scourge.

Grab the data from my nginx logs, since I get very few comments, we can assume that everything is spam.

echo '"IP", "DATE"' > ~/tmp/data_analysis/blogspam.csv
zgrep '/wp-comments-post.php' /var/log/nginx/acc* |
perl -ne 'if (m/.*:(\d{1,3}\.\d{1,3}\.\d{1,3}\.\d{1,3}) \- \- \[(\d+)\/(\w+)\/2014/)
{print "\",$1, "\",\"", $3, "/", $2, "\"\n"}' >>
~/tmp/data_analysis/blogspam.csv

Install R and start it

$ sudo apt-get install -y r-base-core
$ R

Load the data into R

spammers  <- read.csv(file="blogspam.csv", head=TRUE,sep=",")

Let’s find the biggest IP and heavest days:

 > summary(spammers)
               IP             Date      
 1.1.1.1        : 2135   Sep/07 : 1364  
 2.2.2.2        : 2069   Oct/02 : 1353  
 3.3.3.3        : 1971   Oct/03 : 1348  
 4.4.4.4        : 1864   Sep/09 : 1344  
 5.5.5.5        : 1819   Oct/01 : 1333  
 6.6.6.6        : 1712   Sep/30 : 1328  
 (Other)        :50435   (Other):53935

Histogram by IP Frequency

iplist <- as.data.frame(table(spammers$IP))
hist(iplist$Freq, breaks=100, xlab="ip distribution",
      main="Spammer IPs",  col="darkblue")

IP_distribution

This shows that there is no single IP causing all of the trouble, so there is no simple solution of blocking a single IP.

Graph the number of spam comments per day.
Note: you need to sort the data by date or your lines will be all over the place and the graph unreadable

dates <- as.data.frame(table(spammers$Date))
datessorted <- dates[order(as.Date(dates$Var1,format = "%b/%d")),]
plot(as.POSIXct(datessorted$Var1,format = "%b/%d"),
  datessorted$Freq, main="spam comments", xlab="date", ylab="count", type="l")

timeseries

This gives me a basic idea of the problem and further analysis will be available in Part 2

Note: Since I sat down to write this post after clearing out the spam comments I now have 101 new spam comments.

Getting started with VimWiki

2 Comments »

VimWiki is an excellent tool for creating both a personal knowledgebase and journal.  I use it everyday to keep track of new techniques that I learn as well as to keep notes on how I perform tasks everyday.  Because it is completely text based with minimal formatting, it is possible to share easily and keep in git version control for a full history of how and what I learn.  In addition, the search feature allows me to quickly recall how to perform a task or find the information I need quickly.

Installation

1) Install vundle or download my dotfiles to get started with managing vim plugins

2) Install the VimWiki and Calendar plugins for vim by adding the following to your vundle file

Bundle 'mattn/calendar-vim'
Bundle 'vimwiki'

3) update all of your plugins

vim +BundleInstall +qall
mkdir -p ~/Documents/VimWiki/

4)  Remap your leader key in your .vimrc . I set mine to ,

:let mapleader=","

To find your current leader key run

:echo mapleader

and it will appear in the bottom left

5) Add the the following to your .vimrc to get started

" vimwiki stuff "
" Run multiple wikis "
let g:vimwiki_list = [
                        \{'path': '~/Documents/VimWiki/personal.wiki'},
                        \{'path': '~/Documents/VimWiki/tech.wiki'}
                \]
au BufRead,BufNewFile *.wiki set filetype=vimwiki
:autocmd FileType vimwiki map d :VimwikiMakeDiaryNote
function! ToggleCalendar()
  execute ":Calendar"
  if exists("g:calendar_open")
    if g:calendar_open == 1
      execute "q"
      unlet g:calendar_open
    else
      g:calendar_open = 1
    end
  else
    let g:calendar_open = 1
  end
endfunction
:autocmd FileType vimwiki map c :call ToggleCalendar()

 

Using the Wiki Features

 

1)  Start up Vim

ss1

 

2)  The following command will take you to the top page of your wiki

,ww

ss2

3) select yes to create the new directory and it will take you to your new Wiki index

 

4) enter information

ss3

5) to make the text actual links surround with [[ ]]

ss4

6) Move up and down when the cursor is on the link you want Company Information in this case hit return and it will generate a new wiki page

ss5

7) add data to the page

ss6

8) go back to the main page by pressing the backspace key

ss3

Searching the Wiki

1) Search for the term “blog” by running the following command

:VWS /blog/

ss8

shows (1 of 2)

ss9

2) show all of the matches

:lopen

ss10

3)  navigate up and down and hit return to select that file

ss11
Using diary

1) create a new diary entry and add data.  If you add the == Title == at the top it will be visible in the index (next step)

,w,w

ss12

2) go to the diary index

,wi

ss13

3) build a diary index

:VimwikiDiaryGenerateLinks

ss14
4) using the calendar plugin to toggle on/off

,c

ss15

5) you can navigate to the date and hit return on that date and the diary entry for that date will open!

ss16

Resources

VimWiki Quick Reference Guide  

VimWiki on Github

Opensource Infrastructure Revisited

Comments Off on Opensource Infrastructure Revisited

In a previous article, I detailed the open source projects that I used to implement a PaaS infrastructure.

Since that time the number of instances in the infrastructure has grown by 2.5X and several of the components needed to be rethought.

Capacity/Performance Management

Previous: Collectd/Visage
Replacement: Collectd/Graphite
Reasons: The collectd backend was too slow and I/O heavy
Graphite graphs are easily to embed in dashboard applications
Ability to easily transform metrics, such as average CPU across a cluster of servers

Continuous Integration

Previous: Selenium
Replacement: Custom Tests
Reasons: Selenium tests failed too often for undiscernable reasons
False positives slowed development too often

Log Collection

Previous: Rsyslog/Graylog2
Replacement: Logstash/ElasticSearch/Kibana
Reasons: Mongodb too slow in EC2 for storing and searching

Logstash offers better parsing and indexing of logs with powerful filtersElasticSearch is super fast and scales horizontally on EC2

Kibana is simple to use and allows Developers to quickly find the relevant information

All of these components are easily integrated into our dashboard application

These changes not only allow the infrastructure to scale, but provide APIs that allow easy integration with custom dashboards.

Upgrading to Puppet 3.0

1 Comment »

After spending a few days at PuppetConf and talking with Eric S. and Jeff McC. of puppetlabs, I felt compelled to upgrade the latest version of our architecture to Puppet 3.0 from 2.7. Mainly to fix plugin sync issues and for the increased performance.

Here is my list of gotchas:

0) You cannot run both and ENC and stored configs at the same time. Filed a bug (#16698) on this

1) On my puppet master I run as user puppet under unicorn+nginx. When you don’t run as root, puppet only looks in ~/.puppet for the configuration file (Bug #16637). A note is also in the release notes. I temporarily got around this by:

mv ~/.puppet ~/.puppet.old
ln -s /etc/puppet ~/.puppet
#also grab the new rack config
cp /puppet-3.0.0-rc8/ext/rack/files/config.ru .

and restarting my unicorn processes. I am much happier than I was on Passenger+Apache.

2) Ignoring deprication warnings. The following will fail

  file { '/etc/nginx/nginx.conf':
    ensure  => present,
    source  => [ "puppet:///nginx/${::fqdn}.nginx.conf",
                "puppet:///nginx/${::role}.nginx.conf",
                'puppet:///nginx/default.nginx.conf'],
  }

with this message:

err: /Stage[pre]/Nginx::Config/File[/etc/nginx/nginx.conf]: 
Could not evaluate: Error 400 on SERVER: Not authorized to call find on
/file_metadata/nginx/foo.bar.com.nginx.conf Could not retrieve file metadata for
puppet:///file_metadata/nginx/foo.bar.com.nginx.conf: Error 400 on SERVER: 
Not authorized to call find on /file_metadata/nginx/foo.bar.com.nginx.conf at 
/etc/puppet/modules/nginx/config.pp:86

You need to be sure to include the modules path between in the source.

  file { '/etc/nginx/nginx.conf':
    ensure  => present,
    source  => [ "puppet:///modules/nginx/${::fqdn}.nginx.conf",
                "puppet:///modules/nginx/${::role}.nginx.conf",
                'puppet:///modules/nginx/default.nginx.conf'],
  }

Jeff even helped by suggesting the error message was somewhat cryptic and filed Bug #16667

3) When I write the facts.yml file for mcollective I needed to remove the map call so that I did not get the following error after moving to ruby 1.9:

--- a/modules/mcollective/manifests/config.pp
+++ b/modules/mcollective/manifests/config.pp
@@ -34,7 +34,7 @@ class mcollective::config inherits mcollective {
     group    => root,
     mode     => '0400',
     loglevel => debug,
-    content  => inline_template('<%= Hash[scope.to_hash.reject 
- { |k,v| k.to_s =~ /
- (uptime_seconds|uptime|timestamp|free|path|ec2_metrics_vhostmd|
- servername|ec2_public_keys_0_openssh_key|sshrsakey|sshdsakey|serverip)/ 
- }.sort.map].to_yaml - %>'),
+    content  => inline_template('<%= Hash[scope.to_hash.reject 
+ { |k,v| k.to_s =~ /
+ (uptime_seconds|uptime|timestamp|free|path|ec2_metrics_vhostmd|
+ servername|ec2_public_keys_0_openssh_key|sshrsakey|sshdsakey|serverip)/ 
+ }.sort].to_yaml %>'),
     require  => Class['mcollective::package'],
   }

Thanks to the Eric and Jeff for helping me to roll this out.
So far, I’m super happy with the significantly faster catalog compile times and the awesome support that the PuppetLabs Community team has provided.

Sync Puppet Certs between EC2 regions

Comments Off on Sync Puppet Certs between EC2 regions

In the past I have used nginx to route all cert requests to a single cert server. This worked fine when I had limited my puppet infrastructure to a single EC2 region. However, I recently decided to have puppet masters on separate coasts.

Keeping the certs in sync requires a two-way sync, so I ruled out just rsyncing files around. I tried deploying various drbd+clustered_file_system solutions, and while the tests worked within a region, I could not get them working well through the NAT of the two regions.

A helpful IRC regular (semiosis, thanks!) suggested unison. The issue was that a cron job might be too slow and I may run into issues performing the unison sync. There’s a very useful program that monitors file systems for changes and performs actions based on inode level changes called incron. So the obvious solution was to monitor the filesystem for changes then force a unison sync.

The final solution looks like this:

Install Unison

apt-get install unison

make sure ssh works

unison -batch -auto /etc/puppet/ssl/ca/signed \
ssh://puppet@OTHERPUPPETHOST//etc/puppet/sslca/signed

write simplescript on each host

#!/bin/bash
 
/usr/bin/unison -batch -auto /etc/puppet/ssl/ca/signed \
ssh://puppet@OTHERPUPPETHOST//etc/puppet/ssl/ca/signed > /tmp/sync.log

Set the right mode

 chmod +x /bin/puppet_cert_sync

add a crontab entry to make sure it stays kosher

31 * * * * /bin/puppet_cert_sync

install incron

sudo apt-get install incron

configure it to allow user puppet

echo "puppet" >> /etc/incron.allow

add the incrontab entry

export EDITOR=vi
incrontab -e
 
/etc/puppet/ssl/ca/signed IN_CLOSE_WRITE,IN_CREATE,IN_DELETE /bin/puppet_cert_sync

Then test on one host by running

watch -n 1 ls /etc/puppet/ssl/ca/signed/testhost.pem

And on the other host run

sudo puppetca --clean testhost

Caveats: This may not work in an environment with a many new certs being created very close to each other in both environments. It is also not as highly performant as a clustered file system, but seems to work well in my use case. In addition, the default puppet ssl directory is different, so adjust as necessary.

HTTP Troubleshooting with tcpdump/tcptrace

Comments Off on HTTP Troubleshooting with tcpdump/tcptrace

Operations people are often called upon to do low level HTTP troublshooting and I often end up using tcpdump and tcptrace to break out HTTP sessions and troubleshoot.

Install tcptrace on your localmachine

apt-get install tcptrace

or for you Mac people

brew install tcptrace

Run tcpdump on your server

tcpdump -s 1500 -w /tmp/DUMP.pcap -c 5000 -i eth0 port 80 and host www.mague.com
switch reason
-s Sets the snaplength or the length to capture, by default it is often too small and you lose data that you want for analysis
-w Write a pcap file to this location. I usually prefer to perform analysis on another host
-c Capture this many packets. Not necessary, but useful if you forget to stop the capture
-i Interface to capture on. lo is the looopback and you can find interfaces by running ifconfig -a
expression limit to ports or protocols More info on filtering

Copy the dumpfile down to your local machine

mkdir -p ~/tmp/analysis
cd ~/tmp/analysis
scp remotehost:/tmp/DUMP.pcap .
tcptrace -n -xhttp DUMP.pcap

This will create a bunch of files in your directory like so:

172.16.0.20_http.xpl contains information that you can plot using xplot
http.times contains information on the timestamps when data was first fetched and completed
for troubleshooting however we are interested in the *.dat files

The request and response are in separate files with the names reversed.

For example a2b_contents.dat is the request

and b2a_contents.dat is the response

Now you can go about finding errors with grep

chris@gorilla:o ] ~/tmp/analysis 
$ grep --binary-files=text 404 *.dat
o2p_contents.dat:GET /throw/me/a/404/please HTTP/1.1
p2o_contents.dat:HTTP/1.1 404 Not Found

This is also super useful if you want to use curl later to reproduce any issue because now you can just add all of the headers that were previously sent

curl -v -H "Host: www.mague.com" -H \
    "Accept-Encoding: gzip,deflate,sdch" http://www.mague.com

Puppet ENCs and Automating Monitoring Setups with Puppet

Comments Off on Puppet ENCs and Automating Monitoring Setups with Puppet

Patrick Buckley came and spoke about how he uses Puppet Node Classifiers – Slideshare

After him, I spoke about using Puppet to configure monitoring. – Slideshare

Wrap a unix command for mcollective code example

Comments Off on Wrap a unix command for mcollective code example

Writing plugins to wrap a command in mcollective is very easy.

What is difficult is figuring out all of the conventions that you need to follow.

I’m going to walk you through an example with some tips.

Conventions you need to follow

1) Agents need to go in the agent directory.  You can find it with the following command

sudo find / -name rpcutil.ddl -exec dirname {} \;

2) The file name should be all lowercase with a file mode of 644

test@mcollectivemaster:/usr/share/mcollective/plugins/mcollective/agent$ ls -l wrapit.rb
-rw-r--r-- 1 root root 1127 2012-04-13 19:05 wrapit.rb

3)  The class name needs to match the file name with at least the first letter capitalized.

4) To more easily debug make sure the following two lines are present in your /etc/mcollective/server.cfg so you can find errors in syslog

logger_type = syslog
loglevel = debug

5) Restart mcollective between changes to the agent file

An example agent

module MCollective
  module Agent
    class Wrapit < RPC::Agent
###################################################################
      metadata :name        => "My Agent",
               :description => "Example of how to wrap a command",
               :author      => "Me <me@example.com>",
               :license     => "DWYWI",
               :version     => "0.1",
               :url         => "http://blog.mague.com",
               :timeout     => 10
###################################################################
#functions
 
      def run_command(cmd)
	cmdrun = IO.popen(cmd)
	output = cmdrun.read
	cmdrun.close
	if $?.to_i > 0
	  logger.error "WRAPIT: #{cmd}: #{output}"
	  reply.fail! "ERROR: failed with #{output}"
	end
	output
      end
###################################################################
#actions
 
      action "ping" do
	reply[:info] = run_command("/bin/ping -c 5 #{request[:target]}")
      end
 
###################################################################
    end #class
  end   #agent
end     #module

Mcollective uses the request data structure to receive information from the client and reply to send information back to the client. The actions are methods that the client calls to run.

When running commands be sure to wrap with IO.popen so that you can capture both the output and pass the error code. Do not forget to close the popen or you will leak file handles and cause trouble.

An Example Client

 
#!/usr/bin/env ruby
require 'rubygems' if RUBY_VERSION < "1.9"
require 'mcollective'
 
include MCollective::RPC
 
args = {}
action=""
 
options = rpcoptions do |parser, options|
 
  parser.on('-K', '--targ TARGET', 'Target to test against') do |v|
    args[:target] = v
    end 
 
  parser.on('-S', '--action ACTION', 'action') do |v|
    action = v
  end   
 
end
 
mc = rpcclient("wrapit", :options => options)
#mc.fact_filter "servertype", "/myservertype/"
mc.timeout = 10
mc.progress = false
 
mc.send(action, args).each do |resp|
  if resp[:statusmsg] == "OK"
    puts resp[:sender]
    puts resp[:data][:info]
  else
    puts "ERROR:"
    p resp
  end
end

The RPC includes all kinds of switches so be sure to run your script with -h to make sure none of your arguments overlap as they will be overwritten by the default switches.

You can add your own fact_filters and turn off the progress bar which is useful with web interfaces.

Be sure to check the statusmsg sent back from the agent, so you can catch any errors.

Since developer access is limited in my work environment, I wrote lots of customer agents to allow troubleshooting and deployment to be coordinated. Putting them behind a Sinatra web front end has made life easier for all.

Update:It was kindly pointed out to me by R.I. Pienaar that instead of using popen, there is a built-in run() function that is even more flexible.

cmd_output = []
cmd_error = ""
run ( "/bin/ping -c #{request[:target}",
      :environment => {"MYSETTING" => "ABC123"}
      :cwd         => "/home/testguy",
      :stdout      => cmd_output,
      :sdterr      => cmd_error,
)

How-To: Mcollective/RabbitMQ on Ubuntu

Comments Off on How-To: Mcollective/RabbitMQ on Ubuntu

1 ) Install RabbitMQ Prerequisites

apt-get install -y erlang-base erlang-nox

2 ) Install RabbitMQ from the Download Site

wget http://www.rabbitmq.com/releases/rabbitmq-server/v2.8.1/rabbitmq-server_2.8.1-1_all.deb
dpkg -i rabbitmq-server_2.8.1-1_all.deb

3 ) Enable the stomp and AMQ plugins

rabbitmq-plugins enable amqp_client
rabbitmq-plugins enable rabbitmq_stomp

4 ) Create the rabbitmq config file in /etc/rabbitmq/rabbitmq.config

%% Single broken configuration

[
  {rabbitmq_stomp, [{tcp_listeners, [{"0.0.0.0", 6163},
                                     {"::1",       6163}]}]}
].

5 ) Create the MCollective Users

rabbitmqctl add_user mcollective PASSWORD
rabbitmqctl set_user_tags mcollective administrator
rabbitmqctl set_permissions -p / mcollective ".*" ".*" ".*"

6 ) Restart RabbitMQ

/etc/init.d/rabbitmq-server restart

7 ) Edit /etc/mcollective/client.conf to point to the local host

#####
# Example config

topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/share/mcollective/plugins
logfile = /dev/null
loglevel = info

# Plugins
securityprovider = psk
plugin.psk = unset

connector = stomp
plugin.stomp.host = localhost
plugin.stomp.port = 6163
plugin.stomp.user = mcollective
plugin.stomp.password = PASSWORD

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml

8 ) Edit /etc/mcollective/server.cfg to point to the localhost

####
# Example server.cfg
topicprefix = /topic/
main_collective = mcollective
collectives = mcollective
libdir = /usr/share/mcollective/plugins
logger_type = syslog
loglevel = info
daemonize = 1
registerinterval = 30
classesfile = /var/lib/puppet/state/classes.txt

# Plugins
securityprovider = psk
plugin.psk = unset

connector = stomp
plugin.stomp.host = 127.0.0.1
plugin.stomp.port = 6163
plugin.stomp.user = mcollective
plugin.stomp.password = PASSWORD

# Facts
factsource = yaml
plugin.yaml = /etc/mcollective/facts.yaml