Bootstrap a dev environment - OIT CSI Internal
Caution: Internal Document
Most of the information on this site is intended for our customers’ consumption. We do have some process notes intended to guide CSI technical staff through the environments’ implementation. In general, these notes assume familiarity with our back-end processes and are not as “user friendly” as we hope to make the rest of the site.
We include these docs here so that they are available in the off chance that they might prove of interest.
Needful Things
A dev environment consists of 3+ nodes, each serving a particular function.
- Foreman server (enc and console)
oit_linux::role::foreman
- Puppet Master (compiles mannifests)
oit_linux::role::puppet_master
- Puppet CA (certificate authority)
oit_linux::role::puppet_ca
- Smart Proxy (tftp for now)
oit_linux::role::tftoserver
Order of things
Because foreman, puppet, and the puppet CA all interoperate, it’s hard to formulate “do this, stabilize, then do that” in puppet, we install the CA as an “all in one” (AIO) standalone install, and then use that AIO to build and configure the Puppet Master and Foreman components.
We fill out (almost) all of the configuration for a dev system in hiera, using the same repos as we use for production. This allows us to Bootstrap a new dev system using the production systems, which once functional will independently serve the same content/config that we’re using in production. This allows us to confidently stage changes in a branch, test it using the dev boxes, then move it into production on the live boxes with a simple git merge.
Each node needs to be un-associated in turn with the live production system and its certificate tagged to the new dev system.
Data storage
Most of the hiera work is to share secrets between several hosts, and this can be cumbersome and error prone with the data spread among multiple hiera files.
Since “branch” and “environment” are terms already too overloaded, the work “ecosystem” is used to describe a collection of Foreman, Puppet, and PuppetCA hosts cryptographically joined together.
Ecosystem names are just used to identify a .eyaml file under oit_linux:data/ecosystems
This eyaml file holds the settings for all the members of the eco system in one file, to make updating passwords and keys “easier”
Pick oauth keys for each new ecosystem
OAuth tokens are just random blobs. They should also be kept secret. Generate new ones with something like
FOREMAN_CONSUMER_KEY=$(< /dev/urandom tr -cd "[:alnum:]" | head -c 32; echo)
FOREMAN_CONSUMER_SECRET=$(< /dev/urandom tr -cd "[:alnum:]" | head -c 32; echo)
Now, use eyaml to encrypt these and load them into hiera for your new ecosystem.
/opt/puppetlabs/puppet/bin/eyaml encrypt -o block -s $FOREMAN_CONSUMER_SECRET
/opt/puppetlabs/puppet/bin/eyaml encrypt -o block -s $FOREMAN_CONSUMER_KEY
The Puppet CA always goes first
None of the node can talk unless they are signed by the same CA. Building the CA as AIO, means we have a functional puppet + foreman + ca infrastructure that we can use to bootstrap the other nodes.
The Foreman Server comes next
Start wuth an OS and Puppet Agent
Install a machine with an OS and a puppet agent by any means necessary.
Clean up old mess
Remove any existing certs and re-register with the AIO CA.
echo Removing old certificate files
hostname=$(hostname --fqdn)
find /etc/puppetlabs/puppet/ssl \
-name $hostname.pem -print -delete
for i in \
/etc/puppetlabs/puppet/ssl/certs/ca.pem \
/etc/puppetlabs/puppet/ssl/crl.pem \
;
do
echo " $i"
rm -f $i
done
If you have any certificates on the AIO CA for the new foreman server, remove them from the AIO CA with puppet cert clean $foreman_node_name
You can do this on the AIO CA’s GUI, but caching will not be your friend here and you may get some frustrating display bugs.
In the AIO GUI, delete any host records for the new Foreman and Puppet Master.
If you prefer, and hammer is working on the AIO CA, you can do this from the command line with hammer host delete --name $foreman_node_name
Get a shiny new Certificate
Now, back on the foreman server, update the /etc/puppetlabs/puppet/puppet.conf
file and in the [agent]
section, set
ca_server
to the AIO CA serverserver
to the Puppet Master identified for the Ecosystem.
Request a new cert with
puppet agent -t --server=${puppetca_name}.oit.ncsu.edu \
--dns_alt_names=${foreman_name}
The AIO CA ($puppetca_name) may autosign, or it may not. To manually sign the cert, on the AIO CA run
puppet cert sign --allow-dns-alt-names $foreman_node_name
Set the Foreman role, and get your config on
After the cert is signed, go back and re-run the puppet agent -t --server=${puppetca_name}
on the new foreman server. This will download your private key and create an entry for the foreman server on the AIO CA.
Associate the new foreman with the oit_linux::role::foreman
foreman class in the AIO CA GUI.
Run puppet agent -t --server=$puppetca_name
on the foreman server.
You will get errors from the config
portions of the puppet mannifests, since it’s trying to configure foreman, which is not yet installed. Let’s install it, shall we?
Install Foreman using the provided scripts
Several files were copied to /root/foreman-scripts
when the puppet agent ran with the host classified as a foreman node.
Verify the data in params.sh
This contains passwords, so it must be deleted after the host is installed.
Run
cd foreman-scripts
./install-foreman.sh
When the install is successful, puppet agent -t --server=$puppetca_name
should be able to successfully complete and configure foreman correctly.
Configure Foreman
TODO: decide and document database restore and/or ./initial-hammer.sh
On the new foreman host as root:
cd foreman-scripts
./initial-hammer.sh
The initial-hammer script
- sets up the proper certs for hammer
- ensures there is an “NC State” organization
- establishes some smart proxy entries (may not work yet)
- sets up Wolftech authentication
- creates the
production
,ncsutest
, andforgelatest
environments - creates the “View Dashboard” role and the
All Users
usergroup - creates the autosign account for the puppet ca
- creates a TESTERS org,user,group and role
At the time of this writing, it attempts to create a bunch of objects to help testing – domains, subnets, operating systems and foreman templates. This may be removed in the future.
When complete, be SURE to REMOVE both the params.sh
file. This has live passwords!
rm params.sh
If you remove params.created
, puppet will recreate params.sh
on it’s next run.
The Puppet Master comes next
On a “clean” host with the puppet agent installed…
If the host has been registered with a puppet ca, first remove the old certs, as done for foreman under [Get a shiny new Certificate].
Edit /etc/puppetlabs/puppet/puppet.conf
and properly set ca_server
to the AIO CA. You can also set server
to the name of the Puppet Master you are creaing
Now generate a new certificate. Be sure to include the generic puppet Master name as a dns_alt_name.
puppet agent --test --dns_alt_names "puppet" --server $puppetca_name
Because of the --dns_alt_names
, you can’t sign this certificate from the AIO CA GUI. You need to ssh to the AIO CA and run
puppet cert --allow-dns-alt-names sign $puppet_node_name
The puppet master needs both the CA (which it has) and Foreman (ditto).
Confirm params.sh
and run
cd foreman-scripts
./install-foreman.sh
At this point, run puppet agent -t --server $puppetca_name
until you get a clean run or something addressable manifests.
Now run plain puppet agent -t
with no qualifications so the PM can talk to itself. Success here means it mostly works.
Go to the new foreman node, and
- Confirm that we can see the logs on the smart proxy
- Import the environments and classes from the new PM
- Classify the nodes.
Copy the production /etc/puppetlabs/puppetserver/secure-eyaml/keys
to the new
server so it can decrypt the hiera files.
scp $production_pm_host:/etc/puppetlabs/puppetserver/secure-eyaml/keys\* \
/etc/puppetlabs/puppetserver/secure-eyaml/keys
chown puppet.puppet /etc/puppetlabs/puppetserver/secure-eyaml/keys/*
Run r10k to prime the environment pump. 1st run can take some time.
/opt/puppetlabs/puppet/bin/r10k deploy environment -pv
Fixup the Foreman
Import the puppet environments now that r10k has imported them on the PM.
Run puppet agent -t
to report to the PM, and create a host entry on foreman for itself. Classify it.
Fixup the AIO CA
The autosign password has been created on the foreman server.
-
Confirm that the password is properly set in
/etc/puppetlabs/puppet/autosign.foreman.py
file. -
Confirm that
/etc/puppetlabs/puppet/puppet.conf
contains[master] autosign = /etc/puppetlabs/puppet/autosign.foreman.py { mode = 0550 }
You can test autosign by running
/etc/puppetlabs/puppet/autosign.foreman.py $hostname
and it will display if the host is in build mode or not.
Note that it returns a cagey does not exist
error if there is a problem with the foreman password. For a host that exists but is not in build mode, you should get an explicit is not in build mode
response.
When you establish the PuppetCA smart proxy, on the AIO CA, edit the configs under /etc/foreman-proxy/settings.d and disable puppet.yml and puppetca.yml
The smart proxy / tftp always comes last
Cause it ain’t really important in most cases
Edit me