machine data is #bigdata. #axeda #in #m2m

Machines talk way more than humans.  A good break down by @andrewbrust

http://www.zdnet.com/blog/big-data/industrial-big-data/350

Another blog post on HOWTO PORT FORWARD with IPTABLES #in #sysadmin

My application (puppetmaster) is behind a firewall and there is no direct access from the internet to the server.  So, it was necessary to configure a host on the DMZ to act as a proxy.  This was accomplished using IPTABLES to do a forward and reverse NAT. Iptables version: v1.3.5.


This first step is very important.  Without doing this, you will scratch your head wondering why your perfectly formed IPTABLES rules don't get the job done.  


Step 1: enable ip forwarding on the system (if it hasn't already been done):

echo 1 > /proc/sys/net/ipv4/ip_forward

 Step 2: configure IPTABLES (on CentOS, add these lines to your /etc/sysconfig/iptables file and restart iptables):

iptables -A PREROUTING -p tcp -s *route_only_for_this_ip* -d *router_ip* --dport *router_port* -j DNAT --to *destination_ip*:*destination_port*

iptables -A POSTROUTING -o eth0 -d *destination_ip* -j SNAT --to-source *router_ip*

A few definitions for the above:


--dport 8140 -- the default puppetmaster listening port
*route_only_for_this_ip* -- if you want to limit incoming IPs to a single known IP (not required)
*router_port* -- which port incoming requests are accepted on.  This does not have to be the same as the destination_port.  No one should rely on security through obscurity, but using it is a best practice.
*router_ip* -- the host doing the routing (the one running these rules!)
*destination_ip* -- the server that will actually be servicing the request
*destination_port* -- the port on the above server that will accept requests


Now here's an important note.  If you input those rules and run a simple:

iptables -L

Chain INPUT (policy ACCEPT)target     prot opt source               destination         
Chain FORWARD (policy ACCEPT)target     prot opt source               destination         
Chain OUTPUT (policy ACCEPT)target     prot opt source               destination      

What?  Your rules are NOT in your ouptut!  To see them, you need to specify the nat table, like this: 

iptables -L -t nat

Chain PREROUTING (policy ACCEPT)
target     prot opt source               destination      
DNAT       tcp  --  anywhere             router_ip                tcp dpt:router_port to:destination_ip:destination_port
Chain POSTROUTING (policy ACCEPT)
target     prot opt source               destination      
SNAT       all  --  anywhere             destination_ip      to:router_ip
Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Resources:


http://www.linuxquestions.org/questions/linux-networking-3/iptables-port-forwarding-599401/
http://www.whatismyip.com -- love this tool.  From a command line using wget or curl goto: http://automation.whatismyip.com/n09230945.asp
http://www.linuxquestions.org/questions/linux-software-2/iptables-list-does-not-show-pre-or-postrouting-rules-785184/

The decline and fall of system administration #in #devops

wherein @pvenezia bemoans: "Virtualization makes it all too easy to spawn new instances rather than figuring out what went wrong. Is this the end of Unix best practices?"

http://www.infoworld.com/d/data-center/the-decline-and-fall-system-administration-375

Different takes on why monitoring sucks and what's to be done about it #in

Why monitoring sucks — for now

http://gigaom.com/2012/02/12/why-monitoring-sucks-for-now/


A new (old) model
I’d suggest that any well-designed monitoring tool can help automate the OODA loop for operations teams.
1. Deep integration
2. Contextual alerting and pattern recognition
3. Timeliness
4. High resolution
5. Dynamic configuration
What’s next for monitoring?


Why Alerts Suck and Monitoring Solutions need to become Smarter

http://www.appdynamics.com/blog/2012/01/23/why-alerts-suck-and-monitoring-solutions-need-to-become-smarter/

#1 Problem Identification – Do I have a problem?
#2 Problem Isolation – Where is my problem?
#3 Problem Resolution – How do I fix my problem?



My ideal monitoring system
http://forecastcloudy.net/2012/01/12/my-ideal-monitoring-system/

• Hosted (CloudKick, ServerDensity, CloudWatch, RevelCloud and others) vs Installed (Nagios, Munin, Ganglia, Cacti)
• Hosted solutions pricing plans use varied parameters such as price/server, price/metric, retention policy, # of metrics tracked, realtime-ness, etc.
• Poll based method – where collecting server polls the other servers/service vs. Push – where you have a client on the server that pushes locally collected data to logging/monitoring server
• Allowing custom metrics – not all systems allows monitoring, plotting, sending and alert on custom data (at least not in a easy manner)

puppet day #2 -- and I need a custom fact

Objective
One of the first things I wanted to accomplish with puppet is to track down rogue cron jobs under accounts of people that are no longer here.  The broader objective is to delete old/un-used accounts.

Problem
But there was some evidence that a few of these old accounts still had cron jobs running.  So, we couldn't just delete the old accounts, but needed to proceed cautiously to insure we didn't stomp on some cron job that was actually needed!

I was looking for puppet to tell me which systems had cron jobs under this old account.  Now, puppet is a declarative language, so something like:

if /var/spool/cron/userfoo exists, notify me, so I can take a look and see what I need to fix/replace

doesn't exist!  In puppet, you have to declare whether something should or should not exist and then puppet will take the corresponding action.  I just wanted puppet to tell me about something on my system.  I didn't want puppet to take an action!

Solution
It's up to the puppetlabs provided facter to help out here.  Puppet ships with a bundle called facter that collects a lot of bits of information about systems, like their OS, RAM, kernel version, etc.  The code to gather these facts is written in ruby and is extensible.  I needed a custom fact that would indicate whether or not /var/spool/cron/userfoo or (on solaris) /var/spool/cron/crontabs/userfoo exists.  Writing that code is actually straight forward (my first ruby code ever! yay!).  Getting that code onto my agents had an obstacle to overcome.

Problem #2
Puppet does not deliver custom facts to agents by default.  Agents and the puppetmaster need this set in /etc/puppet.conf

    pluginsync = true

This required using puppet to update the puppet.conf and restart puppet.  That's what I built.  Getting puppet to allow delivery of custom facts by default is a listed feature request: http://projects.puppetlabs.com/issues/5454


The only gotcha here is to make sure you include:

     hasrestart => true,

in your init.pp for the puppet service.  Otherwise puppet will send a stop, but not a start since it can't send a start since it is no longer running!

Resources
http://conshell.net/wiki/index.php/Puppet
grabbed this:

kill -USR1 `cat /var/run/puppet/puppetd.pid`; tail -f /var/log/syslog 

from the above link.  Which I shortened to: 

kill -USR1 `pgrep puppet`; tail -f /var/log/syslog 

Config details after the jump

Puppet installed -- let's do something!

Having got a critical mass (but not all) of my servers running puppet and talking to the puppetmaster, I was ready to start actually doing something with puppet.  So, the first thing I wanted to do, was update the motd on the servers.  I appreciate a standard look and feel when logging into a server and being provided with some useful info about the host I'm on.  Moreover, I wanted to communicate to system users that

I found this: https://github.com/aussielunix/puppet-motd, which uses a puppet template to collect a number of facts along with a really big ASCII banner, that I quite like.
                              _   

 _ __  _   _ _ __  _ __   ___| |_ 
| '_ \| | | | '_ \| '_ \ / _ \ __|
| |_) | |_| | |_) | |_) |  __/ |_ 
| .__/ \__,_| .__/| .__/ \___|\__|
|_|         |_|   |_|             
                                            _   _ 
 _ __ ___   __ _ _ __   __ _  __ _  ___  __| | | |
| '_ ` _ \ / _` | '_ \ / _` |/ _` |/ _ \/ _` | | |
| | | | | | (_| | | | | (_| | (_| |  __/ (_| | |_|
|_| |_| |_|\__,_|_| |_|\__,_|\__, |\___|\__,_| (_)
                             |___/                


Any files that have a 'Puppet' header need to be changed in puppet. 

Interesting tidbit
In my motd.erb template, I included:

Uptime:    <%= uptime %>

What happens with this, is that the "uptime" fact (and the other facts included in the template) gets evaluated on the client on every puppet run and a flat file without the puppet mock-up is laid down on the file system.  This file gets compared and reevaluated on every run.  Here's the point: every day the uptime changes and a new file is laid down in /etc/motd and the old file is backed up.  This is clearly pretty inefficient, and needs to be replaced with a function that will process/update the uptime on login, and not on every puppet run.

Resources
https://github.com/aussielunix/puppet-motd
My init.pp and motd.erb are in the comments I just discovered that I can't format in comments, so adding the file specs after the jump...

managing users with puppet

useful resource:
http://itand.me/using-puppet-to-manage-users-passwords-and-ss