Chris's Blog

Devops Shokunin

Wrap a unix command for mcollective code example

Comments Off on Wrap a unix command for mcollective code example

Writing plugins to wrap a command in mcollective is very easy.

What is difficult is figuring out all of the conventions that you need to follow.

I’m going to walk you through an example with some tips.

Conventions you need to follow

1) Agents need to go in the agent directory.  You can find it with the following command

sudo find / -name rpcutil.ddl -exec dirname {} \;

2) The file name should be all lowercase with a file mode of 644

test@mcollectivemaster:/usr/share/mcollective/plugins/mcollective/agent$ ls -l wrapit.rb
-rw-r--r-- 1 root root 1127 2012-04-13 19:05 wrapit.rb

3)  The class name needs to match the file name with at least the first letter capitalized.

4) To more easily debug make sure the following two lines are present in your /etc/mcollective/server.cfg so you can find errors in syslog

logger_type = syslog
loglevel = debug

5) Restart mcollective between changes to the agent file

An example agent

module MCollective
  module Agent
    class Wrapit < RPC::Agent
###################################################################
      metadata :name        => "My Agent",
               :description => "Example of how to wrap a command",
               :author      => "Me <me@example.com>",
               :license     => "DWYWI",
               :version     => "0.1",
               :url         => "http://blog.mague.com",
               :timeout     => 10
###################################################################
#functions
 
      def run_command(cmd)
	cmdrun = IO.popen(cmd)
	output = cmdrun.read
	cmdrun.close
	if $?.to_i > 0
	  logger.error "WRAPIT: #{cmd}: #{output}"
	  reply.fail! "ERROR: failed with #{output}"
	end
	output
      end
###################################################################
#actions
 
      action "ping" do
	reply[:info] = run_command("/bin/ping -c 5 #{request[:target]}")
      end
 
###################################################################
    end #class
  end   #agent
end     #module

Mcollective uses the request data structure to receive information from the client and reply to send information back to the client. The actions are methods that the client calls to run.

When running commands be sure to wrap with IO.popen so that you can capture both the output and pass the error code. Do not forget to close the popen or you will leak file handles and cause trouble.

An Example Client

 
#!/usr/bin/env ruby
require 'rubygems' if RUBY_VERSION < "1.9"
require 'mcollective'
 
include MCollective::RPC
 
args = {}
action=""
 
options = rpcoptions do |parser, options|
 
  parser.on('-K', '--targ TARGET', 'Target to test against') do |v|
    args[:target] = v
    end 
 
  parser.on('-S', '--action ACTION', 'action') do |v|
    action = v
  end   
 
end
 
mc = rpcclient("wrapit", :options => options)
#mc.fact_filter "servertype", "/myservertype/"
mc.timeout = 10
mc.progress = false
 
mc.send(action, args).each do |resp|
  if resp[:statusmsg] == "OK"
    puts resp[:sender]
    puts resp[:data][:info]
  else
    puts "ERROR:"
    p resp
  end
end

The RPC includes all kinds of switches so be sure to run your script with -h to make sure none of your arguments overlap as they will be overwritten by the default switches.

You can add your own fact_filters and turn off the progress bar which is useful with web interfaces.

Be sure to check the statusmsg sent back from the agent, so you can catch any errors.

Since developer access is limited in my work environment, I wrote lots of customer agents to allow troubleshooting and deployment to be coordinated. Putting them behind a Sinatra web front end has made life easier for all.

Update:It was kindly pointed out to me by R.I. Pienaar that instead of using popen, there is a built-in run() function that is even more flexible.

cmd_output = []
cmd_error = ""
run ( "/bin/ping -c #{request[:target}",
      :environment => {"MYSETTING" => "ABC123"}
      :cwd         => "/home/testguy",
      :stdout      => cmd_output,
      :sdterr      => cmd_error,
)

Packaging – Deploying Ruby Applications Like an Adult – Part 2

1 Comment »

Continuing from Part 1

Build gems!

 

It’s not that hard and your efforts will be rewarded.

Here are my arguments for learning packaging

 

What’s running on your system now?

When you’re running hundreds of servers you need a programatic way of auditing what is running on your system. Compiling from source will not give this to you for every single package on your system. Git is wonderful as a source code management system but did a4f85e72894895a8269d65cb3fa2ab012804d3ef come before or after aa7c72e6a15ae37db7beb6450f4db3d30069a7dd and what developer or product manager would be able to give you a git hash as to the version they want running on production? Even with tags going back and forth is hard.

Are all of your dependencies met and consistent?

What if some dependency of a dependency is updated causing a bug? Deploying from source code and running bundler to handle dependencies means you might have different gem versions running between the time you brought up the original server to when you added a new node into the cluster. It happens and it is very time consuming to troubleshoot.

How long does it take you to deploy an application?

Takes me 20 seconds to release across a 100 node cluster. It can take up to 10 minutes to download and install all of the dependencies on my old system and then there are plenty of failures due to network issues or rate-limiting from the upstream server. Internal and external customers don’t do delayed gratification.

Can I give a gem version to a developer and be sure they’re running what’s on production so they can troubleshoot?

Yes

 


I’m still waiting for a good argument against packaging.

There are excellent gem tutorials available

Example Gemspec file for building

Before building the gem, I take another step and use

bundle install --deployment

this downloads all of the gems and compiles all of the extensions necessary to run the gem in the vendor directory. Now when you start your application with

bundle exec START_COMMND

it will use only those gems in the vendor folder. You can view the full Rakefile here

Deploying Ruby Applications Like an Adult

Comments Off on Deploying Ruby Applications Like an Adult

“Push button deploy” is something that is often hear people requesting or mentioning as something they would like to have. What’s more important, in my opinion, is to provide a reliable and scalable system for both internal and external developers to deploy code to the staging environment for clients to QA. Staging deployments should be as simple as possible. Production releases are slightly more complicated as Operations needs to carefully monitor and coordinate with external parties, but should still use the same base system.

Larger Image

Requirements for a deployment system

Scalable

  • deploying a package to 1 server or 100 servers should take the same amount of time and effort

Sane

  • Deploy only sanity checked code.
  • Break loudly.
  • Fit with developer culture.

Fast

  • Everyone likes thing to happen quickly.
  • Clients don’t do delayed gratification.

Audit-able

  • What is running right now?
  • How can I trace back to a change in the source code?
  • Is what I think really running?
  • Logs, logs, logs

Reliable

  • It’s Ops, so no one else will be available at 3am to fix it
  • Have to be able to quickly troubleshoot

Flexible

  • Requirements will change over time.
  • Owned by operations, so changes can be separate from production releases.

 

Here is what I came up with:

Larger Image

The criteria for the components chosen is described here
The next posts will go into more detail on individual components.

DNSRR – rewriting DNS for testing web applications

Comments Off on DNSRR – rewriting DNS for testing web applications

When testing web applications, it is often necessary to rewrite DNS entries to avoid XSS Javascript warnings.

Building on Rubydns my company has open sourced a quick ruby script to easily rewrite DNS queries for web testing

Available on Github

Dashboard Example with Sinatra and Mcollective

Comments Off on Dashboard Example with Sinatra and Mcollective

Having a dashboard to provide real time data to users helps minimize interruptions at work.

The combination of Sinatra handling the incoming HTTP requests and Mcollective pulling real time data from the infrastructure provides the responsiveness and self-service that saves everyone time and effort.

The example code is available on Github

Here are the screen shots running on my internal network.

Welcome Screen

Filtering Form

Results from Monitoring Agent

Results from Puppetd Agent