Puppet Application Orchestration: Building a load balanced and distributed multi-tier application

For those who don’t know puppet I will give following statement from the puppet website (https://puppetlabs.com/puppet/what-is-puppet) itself:

Puppet is a configuration management system that allows you to define the state of your IT infrastructure, then automatically enforces the correct state.

It allows IT administrators to model your applications in what puppet calls manifests and will make sure the configuration remains consistent in time. If you want to learn more about puppet I suggest you download the learning VM to gain experience or you can also download Puppet enterprise for free (up to ten nodes).

In the latest version, 2015.3, a cool new feature called “Application Orchestration” has been launched. Application Orchestration (AO) allows administrators to model distributed applications across multiple Puppet nodes. In this blogpost I will show how to model a load balanced multi-tiered application using AO.

Note: the official Puppet documentation explains how you can configure AO on your own puppet master


First a few concepts. Puppet now features what is called service resources. A service resource can be defined by the administrators themselves. You can think of it as an object comprising attributes identifying a certain service. This object can be used by nodes that want to use that service.
This may seem vague but think of the following ‘HTTP’ example. When we have a web server, it can offer a HTTP service resource. Following attributes are needed to identify all facets of that service:

  • Name of the web server (FQDN)
  • IP of the web server
  • Port of the web service (for example 80)

With the attributes listed above, sufficient information is provided to connect to the web server.

Next are the export and consume statements. When modeling your application, some nodes will export service resources (for example: a web server will export the HTTP resource) and another node will consume it. When consuming the service resource, a node can use all attributes contained in the service resource.

This will become clear when looking at the full example.

The problem

The multi-tiered application we will model using AO consists of:

  • A loadbalancer (HAProxy) for the web servers
  • Multiple web servers (Apache)
  • A single database (MySQL)

Every component in the application will run on the Ubuntu (14.04) operating system and will be managed by puppet.

The solution

First, we will need to think about the relations between our components. The web servers need to know the IP and port the database is currently listening on and the load balancer needs to know which IPs and ports it needs to load balance. In other words:

  • The database needs to export his “SQL” information to the web servers
  • The web servers need to export their “HTTP” information to the load balancer

Or in a diagram:

multi tier app graffle

The application we will define will be called “guestbook”. An application in AO has the same directory structure as a regular module. So in the folder

/etc/puppetlabs/code/environments/<your environment>/modules

we will create following directory structure:

guestbook directory structure

The files and templates directory can be skipped. In my examples these are needed to store my html and php files.

In the rest of this blogpost I will use production as environment. This is the default environment created by Puppet during installation.

Now that we have our basic directory structure we can define our two service resource types called HTTP and SQL.

In the /etc/puppetlabs/code/environments/production/modules/guestbook/lib/puppet/type directory we need to create two ruby files. The first file that we create will be the definition for the HTTP service resource:

Puppet::Type.newtype :http, :is_capability => true do
        newparam :name, :is_namevar => true
        newparam :http_name
        newparam :http_port
        newparam :http_ip

here we define the name for our type and the attributes of this type. Attributes can be added and removed as you wish.

The definition of the SQL service resource is analog but here we have additional attributes:

Puppet::Type.newtype :sql, :is_capability => true do
        newparam :name, :is_namevar => true
        newparam :user
        newparam :password
        newparam :port
        newparam :host
        newparam :database

Now that we have our service resources, we can start using them in our application that we will define now.

In /etc/puppetlabs/code/environments/production/modules/guestbook/manifests we need to create our init.pp file. In this file we will define our application and how the components in our application interact with each other. With all the data in this definition, Puppet will have enough information to spot dependencies between components. For example it will know that the database will need to be created first, followed by the web servers and end with the loadbalancer. More on that later.

This is the code snippet for my guestbook application:


application guestbook(
        #the default number of web servers I will have in my application. Can be overwritten via input parameters
        $number_webs = 1
        #iterate X number of times and create a Http service resource with a unique name and store it in the $webs variable
        $webs = $number_webs.map |$i| {Http["http-${name}-${i}"]}
         #Definition of the database component. Here we define that the database component will export a SQL service resource

                export => Sql["guestbook-${name}"],

        #Loop over the $webs variable and create a unique resource each time. In the definition we declare that the SQL service resource will be consumed and a HTTP service resource is exported
        $webs.each |$i, $web|{
                guestbook::web{ "guestbook-web-${i}":
                        consume => Sql["guestbook-${name}"],
                        export => $web

        #The load balancer definition does not use export or consume statements. We just pass the $webs service resources as an input
        #note: we have a require statement here. This will halt the configuration of the load balancer until the HTTP service resources are created
                balancemembers => $webs,
                require => $webs,

Now that we have defined our application, we still need to define each individual component. We will start with the database, followed by the web and eventually the loadbalancer. Comments in the code will explain what is important.

define guestbook::db (
        $user = 'myuser',
        $password = 'mypass',
        $host = $::clientcert,
        $database = $name,
        $port = 3306
        #This file holds SQL statements to create database tables in MySQL
                ensure => file,
                content => "use $database; create table posts (id INT(6) UNSIGNED AUTO_INCREMENT PRIMARY KEY, post VARCHAR(500) NOT NULL);",

        #Create a database
           user => $user,
           password => $password,
           host => "%",
           sql => "/tmp/mysql.sql",
           grant => ["ALL"]

        #Install a MySQL server and listen to all IP-addresses
        class { '::mysql::server':
           restart => true,
           override_options => {
             mysqld => {
               bind-address => ''

#Important part of the code: here we define our service resource and bind our variables to the attributes
Guestbook::Db produces Sql {
        user => $user,
        password => $password,
        host => $host,
        database => $database,
        port => $port

define guestbook::web (
                #Define input variables
                $nodename = $::fqdn,
                $database = 'db',
                $host = 'localhost',
                $username = 'user',
                $password = 'pw',
                $port = '8080'
        $servername = $host
        $path = '/var/www/html/'
        #Include apache
        class { '::apache':
                mpm_module => 'prefork',

        #Include extra apache modules
        include '::apache::mod::php'

        class { '::mysql::bindings': php_enable => true, }

# removed some file resources to shrink the number of lines

#Multiple resource statements here: we define our consume statement and the produce statement.

#Note: produce, NOT export
Guestbook::Web consumes Sql {
        username => $user,
        password => $password,
        host => $host,
        database => $database,
        port => $port

Guestbook::Web produces Http {
        http_name => $::clientcert,
        http_ip => $::ipaddress,
        http_port => '80'

define guestbook::lb(
        #Input variables: the balancemembers will contain our HTTP service resources

        # Include the HAProxy class
        class { 'haproxy':

        # The load balancer must listen on certain ports and IPs
        haproxy::listen { $::clientcert:
                ipaddress => '*',
                          ports     => '80',
                          mode      => 'http',
                          options   => {
                                  'option'  => ['httplog'],
                                  'balance' => 'roundrobin',

        #Loop over each over the HTTP service resources and create a balance member resource of each of them.
        #The service resource contains all data for the balance members to be instantiated
        $balancemembers.each |$balancemember |{
                haproxy::balancermember { $balancemember['http_name']:
                           server_names => $balancemember['http_name'],
                           listening_service => $::clientcert,
                           options => "check",
                           ipaddresses => $balancemember['http_ip'],
                           ports => $balancemember['http_port']
#Note: no consume or produce statements here.

Now we defined all our components and our application. All we need to do is instantiate it. This is something we do in the site.pp file located in the directory: /etc/puppetlabs/code/environments/production/manifests

Following code can just be added at the bottom of the site.pp file:

#Create a site section
        #Instantiate your guestbook and give it a name
                #Our application has one input parameter: the number of web servers in our application
                number_webs => 2,
                nodes =>{
                        #Bind your puppet nodes to the correct component
                        Node['p00puc03.nubera.local'] => [Guestbook::Db['guestbook']],
                        Node['p00puc04.nubera.local'] => [Guestbook::Web['guestbook-web-0']],
                        Node['p00puc06.nubera.local'] => [Guestbook::Web['guestbook-web-1']],
                        Node['p00puc05.nubera.local'] => [Guestbook::Lb['guestbook-lb']],

Everything is ready now to run the application. If you follow the AO installation and configuration documentation, will be able to login with a certain user and invoke AO commands. I used the peadmin user for this.

Using these commands I can login with the peadmin user and invoke the new puppet app and puppet job commands:

su – peadmin
puppet access login peadmin --service-url https://<puppet master>:4433/rbac-api 

Now we are logged in as peadmin and we have an access token to invoke AO commands. Next we can use puppet app show to show the details of our application.

puppet app show output

When everything seems ok we can run the application with:

puppet job run Guestbook[guestbook].

The output of this command is very verbose but I took a screenshot of an important section of the output:

puppet job run

Here we can see that Puppet knows the relations between the components and that it will invoke puppet runs in the correct order. Also note that the web server puppet runs will be invoked simultaneously.

This concludes our AO work since we now have deployed a load balanced multi-tiered  application


Puppet Application Orchestration is a cool new feature that allows administrators to easily model distributed applications. It can be a bit overwhelming at first sight (especially without a lot of puppet knowledge) but it is really worth it because it facilitates the configuration of your distributed application and components.