“…a version got bumped and something elsewhere stopped working and it took us quite a while to work out what had changed.”

Yup, this happens to everybody, even in well managed environments. John Vincent discusses his thoughts on this problem at length here.

While configuration management tools like Chef or Puppet are a critical component of the DevOps Toolchain for building scalable infrastructure, they are only one piece of the automation puzzle. These tools do a great job helping to prevent drift but drift inevitably finds a way to sneak in. Sometimes it’s for the reasons that Mark talked about in his post. Sometimes it’s because humans are still a part of the IT process and where there are humans, chaos (in this case changes that go undocumented or unnoticed!) will ensue.

To ensure that the actual state of your environment is what you think it is, you need to verify it regularly by checking for any deviations from its previous or intended state. This step allows you to close the feedback loop of your automation system.

Just like any good security system which requires firewalls for prevention as well as intrusion detection to catch any sneaky viruses that manage to get in, a robust automation system should include both an automated deployment platform and a drift detection solution. Only then can you be confident that you know the actual state of your environment.

The post When Puppet and Chef Aren’t Quite Enough appeared first on Metafor Software.

So what was the cause? According to Google, “the outage was caused by a misconfiguration of a user authentication system, which caused a fraction of the login requests to be unintentionally concentrated on a relatively small number of servers. At the time the misconfiguration occurred, monitoring systems detected a load increase and alerted Google Engineering at 1:08 a.m. PT on April 17. However, the alert cleared and the authentication system operated normally under the current load conditions.

At 5:00 a.m. as login traffic increased, the misconfigured servers were unable to process the load. This began to cause errors for some users logging in to Google services. The request load, exacerbated by retry requests from users and automated systems such as IMAP clients, initially appeared as the cause of the login errors.”

This is a great example of the challenge IT operations face in managing configurations. Problematic changes can slip into systems at anytime, causing anomalies that stay hidden until service interruptions strike. Even an infrastructure the size of Google can be brought down by a single small change that slipped by. Could this disaster have been avoided? We think so. Anomaly detection designed for highly scalable, dynamic web environments is one of the key solutions DevOps teams can use to minimize downtime.

We’ve put together a white paper called “Anomaly Detection in the Data Center and the Cloud” to help shed some light on best practices for anomaly detection. It covers how the different types of environmental and behavioral anomalies occur, why automated provisioning is only a start, why thresholds don’t tell the whole story, and the current anomaly detection methods. Hopefully the info in this short (9 page) white paper can help you avoid some downtime (and 3am phone calls!).

Let me know if you find the white paper helpful. If you have suggestions, need help with anomalies, or just want to say “hello”, leave a comment or shoot us a tweet @metaforsoftware.

The post Google Taken Down by an Anomaly – Best Practices that Would Have Prevented Google’s Downtime appeared first on Metafor Software.

We’ve moved beyond thresholds. While thresholds are important for cataclysmic events, (database is down!) they’re not so useful for a lot of other things. They make all kinds of unwarranted assumptions about the underlying systems they are monitoring. So what if my disk space is at 91% on a large capacity drive, and has been stable at 91% for a while? I’d much rather know if the disk usage is rapidly increasing. Machine Learning/Analytics is the next wave.

Context is important. Do I really want to be alerted when I know someone is performing maintenance or backups? There was argument on both sides of this one, but the consensus is no. The trick of course is how do you tie your monitoring and alerting system to other events that are occurring (such as maintenance)?

Know your data. This is something I’ve already written about, but it bears repeating. You need to understand the statistical properties of your data in order to determine what kind of analytics to use. For example, it’s important to know if your data is normally distributed. If so, you can leverage a large number of powerful tools and techniques that are geared to this kind of data.

Don’t just look at timeline charts. We’ve fallen into the trap of looking at all the pretty charts as time series charts. When we do that, we end up missing some important characteristics. For example, a simple histogram of the data, instead of just a time chart, can tell you a lot about anomalies and distribution. Using different kinds of visualization is crucial to giving us a different aspect on our data.

Is all data important? Here we got into a great debate on whether you can just look at all the data, or be picky about which feeds are important and which aren’t. This reminded me of the great debate between Noam Chomsky and Peter Norvig of Google. On the one side, some people are saying data is data, let’s analyze everything and figure out the trends. On the other, some people are saying well not all data is important, so let’s figure out what’s important first and understand the underlying model so we don’t waste resources on the rest. Unresolved as far as I know

We all want to automate. Having humans in the way of detecting and solving DevOps issues doesn’t scale. At some point, we need systems that can detect anomalies before problems become critical, and take appropriate action. We all want to automate. The quote of the day, however, goes to Geoff George who, when asked why doesn’t he automate to the next level, replied: “Because I don’t have the budget”! How true.

I hope that covers the major points. I’m sure I missed a whole bunch of stuff, so let me know. Next post is back to analytics, I promise!

The post Beyond The Pretty Charts – A Report From #devopsdays in Austin appeared first on Metafor Software.

Solutions for spotting those pesky outliers have been around in the world of credit card processing, banking, and security for some time now. “Anomaly detection”, a.k.a. fraud detection, is a pretty common term in those industries. That term has now made its way to the dynamic data center, where system anomalies can be characterised as either static or dynamic. Well, at least that’s how we’ve categorized them.

Static anomalies happen when there are state inconsistencies between servers in a cluster, either physical or virtual, or sometimes on a single server. You may start out with identical resources — OS versions, software packages, or data – but over time, even when systems are identically provisioned using automated tools, drift happens. It’s hard to guarantee ongoing uniformity or even uniformity immediately after a release or provisioning because:

• Machines which timeout during provisioning can miss out on certain installations
• Virtual machines created during or after provisioning miss out on installations
• Software downloads from online repositories may introduce differently versioned files from different servers
• Human error and failure to follow protocol result in ad hoc fixes or changes that don’t get logged or deployed properly
• Security breach or exploits can interfere with the base provisioning

In effect, static anomalies live on your servers and are found in your environment (we sometimes also call them environment anomalies).

Dynamic anomalies, on the other hand, show up as inconsistent server behavior. Even without environment drift, dynamic anomalies can materialize. A server could exhibit behavior unlike the others or even unlike its own behavior the day before. All it takes is:

• Bad code
• Network congestion
• A security breach
• Hardware hiccups

Those are the two main buckets we use for categorizing anomalies. But we’re always tossing around different ideas. How do you see the world of system anomalies? What kind of buckets would you use? Come to think of it, are there more than two buckets? Would love to hear how other ops folks think about this. Leave a comment or tweet us, @metaforsoftware.

The post You Say Dynamic, I Say Static appeared first on Metafor Software.

The first topic I wanted to address, based on the discussions I had, was the difference between parametric and non-parametric methods. Some people mentioned that they were trying to use Holt-Winters or Pearson product-moment, or other methods. While the algorithms themselves can be very powerful, it’s important to understand the characteristics of the data before using any particular technique, otherwise the results you get cannot be trusted. Basically, statistical techniques can be broadly categorized as parametric or non-parametric. In short, parametric techniques make some major assumptions about the characteristics of the data being analyzed, where non-parametric techniques do not. There are other differences of course. For example, you can’t use parametric techniques on non-quantitative data such as labels or rank and position. One of the major assumptions you make however when you use parametric techniques is that the data under analysis has a known statistical distribution.

Let’s use some examples. Take a look at this set of load averages from my analytics server machines that I’m getting from Cacti.

Load averages of my analytics server machines from Cacti

It’s hard to tell just from visual inspection if they belong to any kind of well-known statistical distribution. One quick visual way to see the distribution of the data is to run a histogram. I use Gnu Octave as a stats workbench, but you can use any other similar tool such as Matlab or R. If I plot a quick histogram of one of the data sets:

octave>hist(loadavg01(:,1),30)

This is what I get:

A quick histogram of one of the data sets

Now all of a sudden you can see the nice Gaussian (also known as “normal”, or “bell-curve”) distribution. Knowing that my data is normally distributed, with a well-defined mean and standard deviation opens up a whole world of powerful statistical tools. An overly simplistic example, knowing the mean and standard deviation of this load average time series, given a value at any time, I can predict with a high level of confidence a range for the value of the next sample point. In other words, given the load average at time t, I can predict what the load average at time t+1 should be, within a confidence interval. This is because of what’s called the three-sigma rule, which says that 99.73% of all samples in a normal distribution fall within three standard deviations of the mean (three sigmas, get it?). This means for the load average of server 01 above, 99% of the data falls within the range of 1.64 ± 0.75, which is a relatively wide range. But if we’re willing to lower our bar by a little bit, the three sigma rule also says that 95.45% of all samples fall within a narrower band of two standard deviations (1.64±0.5 in our case), and 68.27% of samples fall within one standard deviation (1.64±0.25). Take your pick. In any case, once time t+1 comes around and I collect that data point, I can compare it with what I think it should be based on the distribution and get a sense of how “normal” or “anomalous” that value is. Presto, anomaly detection 101! And the ONLY reason I can do that with confidence is because I’ve established that my data follows a normal probability distribution.

The flip side of this awesome thing is that if I don’t have a well-behaved probability distribution, all bets are off and I can’t make those kinds of predictions. Which leads us to non-parametric techniques. While it’s somewhat easy to define the requirements of parametric techniques, non-parametric techniques are in essence “everything else”. That’s not to say that you can’t do powerful stats with non-parametric methods. Most parametric methods have an equivalent non-parametric method. It’s just that non-parametric methods are not as powerful statistically, and the results can be sometimes hard to interpret.

I’ll cover more on both parametric and non-parametric techniques in future posts, but in summary, make sure you know the characteristics of your data before making any decision on which particular predictive or statistical algorithm to use, otherwise you might be in for a surprise. And I don’t mean of the nice kind.

Let me know if you found this useful (please share this if you did). Too much math? Not enough math? Want more pretty graphs? I’d love to hear your feedback so please drop me a line in the comments.

The post Everything You Should Know About Anomaly Detection Step 1: The Basics. Know your data – Parametric or Non-parametric? appeared first on Metafor Software.

Ready to Rocket is a unique business recognition list that profiles British Columbia technology companies with the greatest potential for revenue growth. Since 2003, the Ready to Rocket list has consistently predicted revenue growth leaders and the companies most likely to attract investment.

“All of us at Metafor are honored to be a part of such a prestigious list of companies,” says Jenny Yang, CEO, Metafor Software. “If the talks today at Monitorama are any indication, anomaly detection will become a critical component in the DevOps tool chain. The coming year is going to be exciting!”

Metafor’s SaaS based solution can be set to check servers for anomalies hourly or daily, and automatically send email alerts when servers drift from their desired state. With Metafor, performance troubleshooting and release validation can be easily performed by anyone, freeing up valuable senior operations staff time. Metafor Software installs with a single command, and provides actionable insight within minutes.

See the full 2013 ICT Emerging Rockets list

Find out more about Metafor Software’s Anomaly Detection

About Metafor Software:
Metafor Software helps IT operations detect unexpected changes and anomalies in their application and infrastructure environment so they can prevent downtime, improve performance, and reduce operational risk and failed user interactions. Unlike traditional threshold based monitoring tools, Metafor’s anomaly monitoring solution applies machine learning techniques to learn system behavior patterns and alert staff when systems start deviating from their normal state. Install Metafor with a single command, and get results in minutes. More information is available at: http://metaforsoftware.com

The post Metafor Software Named to the 2013 ICT Emerging Rockets List appeared first on Metafor Software.

Metafor replaces manual troubleshooting with automated diagnostics to identify and prevent environment anomalies before they impact quality of service. The cloud based service can be set to check servers for anomalies hourly, daily, or only on certain days, and automatically send email alerts when servers drift from their desired state. With Metafor, performance troubleshooting and release validation can be easily performed by junior operations staff. Metafor Software installs with a single command, and provides actionable insight within minutes.

“We could have saved a week of work if we’d discovered Metafor earlier,” says Jeremy Hutchings, Technical Director at MetroLyrics. “They verify the state of our servers so we can see what’s actually deployed, not just what we think has been deployed.”

“Now I spend 4-5x less time on drift than I did before implementing Metafor Software,” says Kelcey Damage, Infrastructure Systems Architect at Backbone Technology. “Metafor’s anomaly monitoring is a massive time saver.”

“Metafor provides instant actionable insight,” said Toufic Boubez, CTO, Metafor Software. “It’s a standalone solution that fills a critical gap in the DevOps troubleshooting kit by preventing drift in environments where change is continuous and constant.”

Metafor offers its anomaly detection solution for free to anyone who joins its Beta program: http://metaforsoftware.com/join-our-beta

About Metafor Software:
Metafor Software helps IT operations detect unexpected changes and anomalies in their application and infrastructure environment so they can prevent downtime, improve performance, and reduce operational risk and failed user interactions. Unlike traditional threshold based monitoring tools, Metafor’s anomaly monitoring solution applies machine learning techniques to learn system behavior patterns and alert staff when systems start deviating from their normal state. Install Metafor with a single command, and get results in minutes. More information is available at: http://metaforsoftware.com

The post Metafor Software Solves Server Configuration Drift – Announces Environment Anomaly Detection Engine appeared first on Metafor Software.

Glad to see you found us. We’ve been terribly remiss in the communications department lately. Sadly, this is what happens when everyone around here is an engineer… nobody wants to write anything, except code! But from here on out we’re going to try hard to change that.  We promise to communicate more frequently and keep you up to date on all the cool goodies we’ve been building.

To start with, I’m happy to announce that we’ve released a few big features that our beta customers have been waiting for: scheduled snapshots, automated differencing, and email alerts. This latest release is all about making it easy to prevent, manage, and resolve a very special static anomaly — server drift. With these new capabilities, you will be able to troubleshoot environment bugs much faster than before and be warned of unexpected changes before they cause havoc.

Automated provisioning solutions such as Puppet and Chef reduce the chance of drift by ensuring uniform deployment of packages, services, and files. They are great tools, especially for the dynamic data center where nodes can have a very short existence. Automated provisioning is a necessary and good precaution against drift, but it doesn’t guarantee ongoing uniformity or even uniformity immediately after provisioning.

Machines which timeout during provisioning can miss out on installation. Virtual machines created during or after provisioning will miss out. Software downloads from online repositories may introduce differently versioned files from different servers. Then there are those servers we treat as pets, full of personality and ad hoc changes which are not propagated. All of these introduce inconsistencies into the system, anomalies which over time can deteriorate system integrity and cause errors and bad results, inconsistent application behavior between servers, diminished performance, and even downtime. An analogy that comes to mind is the network security firewall: firewalls offer a measure of protection, but they don’t prevent intrusions or problems from within, you still need intrusion detection technology to round out your security suite.

As pointed out by John E. Vincent in his blog Configuration Drift and Next-gen CM, configuration management systems aren’t assertive enough. They are designed to verify the state of a resource at the point they run. They don’t manage the state of those resources until they next inspect them. Configuration management tools don’t get run in response to those resources changing but in response to a user asking them to be checked. To compound the matter, without constant verification, drift can occur across nodes of different types whilst they share a verified common base block. Cliff Moon sums up the challenge pretty good: “Of all the problems to fix in Chef or Puppet, the diffusion and drift of state that occurs in idiomatic usage seems highest priority.” Our latest release fills the role pretty well I’d say.

If you’d like to get an email alert every time any of your servers drift out of their desired state, join our Beta. We’d love your feedback on our tool. Bug reports are welcome too

The post Automate and Validate the Drift Away appeared first on Metafor Software.

Storage and compute time is cheaper than ever, and the proliferation of programming environments and components – whether runtimes, databases or application servers, to name a few – has resulted in a proliferation of reusable, interchangeable, distributed application components. Through the power of Service Oriented Architecture (SOA), programmers can cobble together literally dozens of components, each designed for a specific, optimized task, to create an application. In turn, these components can live anywhere, resulting in an increasingly complex software environment that not only redefines what an application is, it redefines how we should think about the data center itself.

It doesn’t make a difference whether or not these components are mission-critical – directly supporting the business model – or not. To the IT or Development leader, these components increasingly define what goes on in the data center. Add to this the reality of distribution across in-house servers, virtual servers and the cloud, and you end up with the necessary conclusion that application software matters more than the hardware it runs on when it comes to efficient data center operations.

Oh, and did we mention that in most tech companies today you’ll find IT and Development teams potentially working in close physical proximity but actually on different planets when it comes to the creation and deployment of applications and components? If you’re lucky enough to work on a well-defined and managed DevOps team, you’ve overcome this barrier, but that doesn’t help you at all in dealing with the underlying complexity of today’s software.

The implication is that we’re forced to redefine how we allocate value in today’s data centers and server farms. It’s not the hardware it’s the software, and it’s not the application on its own, it’s the underlying components that make it hum. The question for you is: how are you tracking what these components do, where they reside, and what the impact would be on your software and data center when they change?

Whether you’re from IT, Dev or DevOps, you need to know what’s running where and why if you want a truly useful picture of what’s going on. Hardware is the easy part; where it gets really tricky is at the application and component level. You could assume, and take the chance that you’ll figure it out when something goes wrong, or you could build an architectural roadmap that plots out every detail of your environment, hardware and application down to the component level, revealing all its connections and interdependencies in what ends up being a holistic view of what’s really going on in your data center, whether you know about it or not.

This is why we started Metafor.

In our next installment we’ll explore why you need an application-centric model of your data center, not just a hardware-centric view.

The post Application Trumps Hardware appeared first on Metafor Software.

The post Kelcey Damage, Infrastructure Systems Architect appeared first on Metafor Software.