Bootstrap a dev environment - OIT CSI Internal

The sordid and cautionary tale of how to build a development environment for Buildie McBuildface.

Caution: Internal Document

Most of the information on this site is intended for our customers’ consumption. We do have some process notes intended to guide CSI technical staff through the environments’ implementation. In general, these notes assume familiarity with our back-end processes and are not as “user friendly” as we hope to make the rest of the site.

We include these docs here so that they are available in the off chance that they might prove of interest.

Note: As of the last update this document is in extreme flux and should not be consulted by anyone not in a position to verify it’s accuracy.

Needful Things

A dev environment consists of 3+ nodes, each serving a particular function.

Foreman server (enc and console) oit_linux::role::foreman
Puppet Master (compiles mannifests) oit_linux::role::puppet_master
Puppet CA (certificate authority) oit_linux::role::puppet_ca
Smart Proxy (tftp for now) oit_linux::role::tftoserver

Order of things

Because foreman, puppet, and the puppet CA all interoperate, it’s hard to formulate “do this, stabilize, then do that” in puppet, we install the CA as an “all in one” (AIO) standalone install, and then use that AIO to build and configure the Puppet Master and Foreman components.

We fill out (almost) all of the configuration for a dev system in hiera, using the same repos as we use for production. This allows us to Bootstrap a new dev system using the production systems, which once functional will independently serve the same content/config that we’re using in production. This allows us to confidently stage changes in a branch, test it using the dev boxes, then move it into production on the live boxes with a simple git merge.

Each node needs to be un-associated in turn with the live production system and its certificate tagged to the new dev system.

Data storage

Most of the hiera work is to share secrets between several hosts, and this can be cumbersome and error prone with the data spread among multiple hiera files.

Since “branch” and “environment” are terms already too overloaded, the work “ecosystem” is used to describe a collection of Foreman, Puppet, and PuppetCA hosts cryptographically joined together.

Ecosystem names are just used to identify a .eyaml file under oit_linux:data/ecosystems This eyaml file holds the settings for all the members of the eco system in one file, to make updating passwords and keys “easier”

Pick oauth keys for each new ecosystem

OAuth tokens are just random blobs. They should also be kept secret. Generate new ones with something like

FOREMAN_CONSUMER_KEY=$(< /dev/urandom tr -cd "[:alnum:]" | head -c 32; echo)
FOREMAN_CONSUMER_SECRET=$(< /dev/urandom tr -cd "[:alnum:]" | head -c 32; echo)

Now, use eyaml to encrypt these and load them into hiera for your new ecosystem.

/opt/puppetlabs/puppet/bin/eyaml encrypt -o block -s $FOREMAN_CONSUMER_SECRET
/opt/puppetlabs/puppet/bin/eyaml encrypt -o block -s $FOREMAN_CONSUMER_KEY

The Puppet CA always goes first

None of the node can talk unless they are signed by the same CA. Building the CA as AIO, means we have a functional puppet + foreman + ca infrastructure that we can use to bootstrap the other nodes.

Note: TODO: document the ISO/kickstart used to build the AIO.

The Foreman Server comes next

Start wuth an OS and Puppet Agent

Install a machine with an OS and a puppet agent by any means necessary.

Clean up old mess

Remove any existing certs and re-register with the AIO CA.

echo Removing old certificate files
hostname=$(hostname --fqdn)
find /etc/puppetlabs/puppet/ssl \
  -name $hostname.pem -print -delete
for i in \
    /etc/puppetlabs/puppet/ssl/certs/ca.pem \
    /etc/puppetlabs/puppet/ssl/crl.pem \
   ;
do
   echo "   $i"
   rm -f $i
done

If you have any certificates on the AIO CA for the new foreman server, remove them from the AIO CA with puppet cert clean $foreman_node_name You can do this on the AIO CA’s GUI, but caching will not be your friend here and you may get some frustrating display bugs.

In the AIO GUI, delete any host records for the new Foreman and Puppet Master. If you prefer, and hammer is working on the AIO CA, you can do this from the command line with hammer host delete --name $foreman_node_name

Get a shiny new Certificate

Now, back on the foreman server, update the /etc/puppetlabs/puppet/puppet.conf file and in the [agent] section, set

ca_server to the AIO CA server
server to the Puppet Master identified for the Ecosystem.

Request a new cert with

puppet agent -t --server=${puppetca_name}.oit.ncsu.edu \
  --dns_alt_names=${foreman_name}

The AIO CA ($puppetca_name) may autosign, or it may not. To manually sign the cert, on the AIO CA run

puppet cert sign --allow-dns-alt-names $foreman_node_name

Set the Foreman role, and get your config on

After the cert is signed, go back and re-run the puppet agent -t --server=${puppetca_name} on the new foreman server. This will download your private key and create an entry for the foreman server on the AIO CA.

Associate the new foreman with the oit_linux::role::foreman foreman class in the AIO CA GUI.

Run puppet agent -t --server=$puppetca_name on the foreman server.

You will get errors from the config portions of the puppet mannifests, since it’s trying to configure foreman, which is not yet installed. Let’s install it, shall we?

Install Foreman using the provided scripts

Several files were copied to /root/foreman-scripts when the puppet agent ran with the host classified as a foreman node.

Verify the data in params.sh This contains passwords, so it must be deleted after the host is installed.

Run

cd foreman-scripts
./install-foreman.sh

When the install is successful, puppet agent -t --server=$puppetca_name should be able to successfully complete and configure foreman correctly.

Configure Foreman

TODO: decide and document database restore and/or ./initial-hammer.sh

On the new foreman host as root:

cd foreman-scripts
./initial-hammer.sh

The initial-hammer script

sets up the proper certs for hammer
ensures there is an “NC State” organization
establishes some smart proxy entries (may not work yet)
sets up Wolftech authentication
creates the production, ncsutest, and forgelatest environments
creates the “View Dashboard” role and the All Users usergroup
creates the autosign account for the puppet ca
creates a TESTERS org,user,group and role

At the time of this writing, it attempts to create a bunch of objects to help testing – domains, subnets, operating systems and foreman templates. This may be removed in the future.

When complete, be SURE to REMOVE both the params.sh file. This has live passwords!

rm params.sh

If you remove params.created, puppet will recreate params.sh on it’s next run.

The Puppet Master comes next

On a “clean” host with the puppet agent installed…

If the host has been registered with a puppet ca, first remove the old certs, as done for foreman under [Get a shiny new Certificate].

Edit /etc/puppetlabs/puppet/puppet.conf and properly set ca_server to the AIO CA. You can also set server to the name of the Puppet Master you are creaing

Now generate a new certificate. Be sure to include the generic puppet Master name as a dns_alt_name.

puppet agent --test --dns_alt_names "puppet" --server $puppetca_name

Note: TODO write up our subject alternate name games.

Because of the --dns_alt_names, you can’t sign this certificate from the AIO CA GUI. You need to ssh to the AIO CA and run

puppet cert --allow-dns-alt-names sign $puppet_node_name

The puppet master needs both the CA (which it has) and Foreman (ditto).

Confirm params.sh and run

cd foreman-scripts
./install-foreman.sh

At this point, run puppet agent -t --server $puppetca_name until you get a clean run or something addressable manifests.

Now run plain puppet agent -t with no qualifications so the PM can talk to itself. Success here means it mostly works.

Go to the new foreman node, and

Confirm that we can see the logs on the smart proxy
Import the environments and classes from the new PM
Classify the nodes.

Copy the production /etc/puppetlabs/puppetserver/secure-eyaml/keys to the new server so it can decrypt the hiera files.

scp $production_pm_host:/etc/puppetlabs/puppetserver/secure-eyaml/keys\* \
    /etc/puppetlabs/puppetserver/secure-eyaml/keys

chown puppet.puppet /etc/puppetlabs/puppetserver/secure-eyaml/keys/*

Run r10k to prime the environment pump. 1st run can take some time.

/opt/puppetlabs/puppet/bin/r10k deploy environment -pv

Note: Much more about git needs to be explored.

Fixup the Foreman

Import the puppet environments now that r10k has imported them on the PM.

Run puppet agent -t to report to the PM, and create a host entry on foreman for itself. Classify it.

Fixup the AIO CA

The autosign password has been created on the foreman server.

Confirm that the password is properly set in /etc/puppetlabs/puppet/autosign.foreman.py file.

Confirm that /etc/puppetlabs/puppet/puppet.conf contains

[master]
autosign = /etc/puppetlabs/puppet/autosign.foreman.py { mode = 0550 }

You can test autosign by running /etc/puppetlabs/puppet/autosign.foreman.py $hostname and it will display if the host is in build mode or not.

Note that it returns a cagey does not exist error if there is a problem with the foreman password. For a host that exists but is not in build mode, you should get an explicit is not in build mode response.

When you establish the PuppetCA smart proxy, on the AIO CA, edit the configs under /etc/foreman-proxy/settings.d and disable puppet.yml and puppetca.yml

The smart proxy / tftp always comes last

Cause it ain’t really important in most cases

Tags:

Edit me