This document describes how to build the infrastructure from scratch, as well manage it in general. 'SCRIPTS' refers to the scripts repo, found at https://github.com/vtluug/scripts 'ANSIBLE' refers to the ansible repo, found at https://github.com/vtluug/ansible For ansible: - Install it - Install any additional packages in the ansible readme Table of Contents 0. Summary 1. Installation 2. Networking 3. Configuration Summary ====== NFS Configuration: - All hosts: - 755 root:root /home 10.98.0.7:/cistern/nfs/home - 755 root:root /nfs - 755 root:root /nfs/tank - 775 nobody:luugusers /nfs/tank/share 10.98.0.6:/tank/nfs/share - 777 root:root /nfs/tank/scratch 10.98.0.6:/tank/nfs/scratch - 755 root:root /nfs/cistern - 775 nobody:luugusers /nfs/cistern/share 10.98.0.7:/cistern/nfs/share - 775,g+s nobody:officers /nfs/cistern/files 10.98.0.7:/cistern/nfs/files - 755 root:root /nfs/keg - 775 nobody:luugusers /nfs/keg/share 10.98.0.8:/keg/nfs/share - Libvirt hosts only (meltdown, spectre): - 755 root:root /nfs/cistern/libvirt 10.98.0.7:/cistern/nfs/libvirt - 755 root:root /nfs/cistern/libvirt/images - 755 root:root /nfs/cistern/libvirt/images/meltdown (For meltdown vms) - 755 root:root /nfs/cistern/libvirt/images/spectre (For spectre vms) - 755 root:root /nfs/cistern/libvirt/isos - 775 root:root /var/lib/docker /dev/$hostname-vg/docker-lv # For restricting docker container sizes - Docker Hosts only: - 770 root:root /nfs/cistern/docker - 755 root:root /nfs/cistern/docker/data 10.98.0.7:/cistern/nfs/docker/data - Sczi Hosts only: - 755 root:root /nfs/cistern/srv 10.98.0.7:/cistern/nfs/srv - ZFS: - All of the paths listed above should be ZFS filesystems on their respective hosts uexcept the images and isos directory within libvirt, which should be manually created Installation (Baremetal) ======================== All hosts - Create the papatux user with the password in the vtluug-admin repo - This is used as a general admin account - Aside from meltdown and spectre, use 2 boot drives in raid if possible - Open ssh port 2222 (and 22) since this is externally blocked unless the IPs have exemptions. - Ensure ssh is enabled and python is installed - Ensure /etc/hostname contains the host's fqdn Meltdown & Spectre - Install using LVM: - Name the Volume Group 'meltdown-vg'. - Create a 50G root Logical Volume called 'root-lv'. - Create a Logical Volume taking up 90% of free space called 'docker-lv'. This is for setting size limits on containers. - Format 'docker-lv' as xfs. Ansible will take care of mounting it. Router - Disable root password login over ssh. More configuration will be done in the 'Configuration' section after Networking Networking ========== - IPs and MACs of hosts are listed in scripts/router/lan/local_hosts - Hosts get IPv4 address assigned using DHCP based on MAC, and IPv6 are assigned using SLAAC - Set up physical boxes based on architecture_diagram.png TODO Router (Debian) --------------- Configure IP addresses - Copy SCRIPTS/router/ip-config/interfaces into /etc/network/interfaces Configure ARP Proxying - Copy SCRIPTS/router/proxy/arp_proxy.sh into /usr/local/bin/ - Copy SCRIPTS/router/proxy/arp_proxy.service into /etc/systemd/system - Start and enable arp_proxy.service Configure dnsmasq (DHCP/SLAAC/DNS) & resolv.conf - Install dnsmasq - Copy SCRIPTS/router/lan/dnsmasq.conf into /etc/dnsmasq.conf - Copy SCRIPTS/router/lan/local_hosts into /usr/local/bin - Copy SCRIPTS/router/lan/resolv.conf into /etc/resolv.conf - Start and enable dnsmasq.service Configure iptables (NAT/Firewall) - Install iptables - Copy SCRIPTS/router/lan/vtluug_iptables.sh into /usr/local/bin - Copy SCRIPTS/router/lan/vtluug_iptables.service into /etc/systemd/system - Start and enable vtluug_iptables.service Libvirt hosts (Ubuntu via netplan) ---------------------------------- - Configure the bridge interface - Copy SCRIPTS/libvirt-hosts/ip-config/01-netcfg.yaml to /etc/netplan/01-netcfg.yaml - Change $INTERFACE to the interface being bridged - Change $MAC to the mac address specified in $SCRIPTS/router/lan/local_hosts - Run 'netplan apply' - Update dnsmasq on joey with the correct MAC address & apply changes (Only if new host is added w/o any records in dnsmasq) - Restart networking on the libvirt hosts Configuration ============= These steps MUST be done in order. YMMV otherwise. At this point, all bare metal hosts should have an admin user created, so root ssh will be disabled in the following step NFS --- To build each zfs pool, first look up the device ids in /dev/disk/by-id since these are guaranteed to not change (unlike /dev/sdX) - Install 'zfsutils-linux'. If the following commands produce errors reboot first. - Cyberdelia - 7 1TB drives in a raidz2 zfs pool named 'tank' - Similarly, run 'zpool create -f -o ashift=12 keg raidz2 ... - Dirtycow - 8 2TB drives in a raidz2 pool named 'cistern' - Similarly, run 'zpool create -f -o ashift=12 cistern raidz2 ... - Shellshock - 3 500GB drives in a raidz2 zfs pool named 'cistern' - Run 'zpool create -f -o ashift=12 keg raidz1 - Run 'zfs create ' accoring to the 'NFS Configuration' section in the beginning of the document. Baremetal Hosts --------------- - Make sure you've copied your ssh key to all the baremetal servers so ansible works - Run 'ansible-playbook main.yml -i hosts.cfg -K -u papatux --limit=.vtluug.org --skip-tags needs_freeipa' for each host in nfs - NFS needs to work first before we can do anything and freeipa needs to work before proper permissions can be set - You could do the nfs instead of each host individually, but doing 1 host a time makes things easier to debug - Create a homedir for papatux on nfs with the correct umask settings since it won't be a user in freeipa - Run 'ansible-playbook main.yml -i hosts.cfg -K -u papatux --limit=.vtluug.org --skip-tags needs_freeipa' for each host in baremetal and not in nfs group - You could do the nfs instead of each host individually, but doing 1 host a time makes things easier to debug Create the VMs -------------- - Run 'ansible-playbook deploy.yml -i hosts.cfg -u papatux -k -K -@/path/to/accounts/file'' - Configuring FreeIPA Server - Copy 'SCRIPTS/freeipa/ipa-config/ifcfg-eth0:0' to '/etc/sysconfig/network-scripts/eth0:0'. - The configuration is not automated because it is only done once, and you should be aware of how it's configured - Install packages freeipa-server, ipa-server-dns, and rng-tools - In /etc/hosts, add chimera's v4 & v6 address, fqdn, and hostname to the bottom (on 2 separate lines) - In /etc/NetworkManager/NetworkManager.conf, add 'dns=none' under [main] - Reboot - In /etc/resolv.conf, replace the content with a v4 & v6 address (preferably VTs) - Reboot again - Run 'ipa-server-install' (as root) - Use these options: - Do you want to configure integrated DNS (BIND)? [no]: yes - Server host name: chimera.vtluug.org - Domain name: krb.vtluug.org - Realm name: KRB.VTLUUG.ORG - Directory Manager PW: See accounts file in gitolite-admin - Admin PW: See accounts file in gitolite-admin - Configure DNS forwarders: yes - Search for missing reverse zones: yes - (Press Enter until you get to the next step) - Continue to configure system with these values: yes - Run 'kinit admin' to authenticate as admin - Create groups, set shell/home/email, and disable private groups - 'ipa group-add luugusers --desc "VTLUUG Members"' - 'ipa group-add officers --desc "VTLUUG Officers"' - 'ipa config-mod --defaultgroup luugusers' - 'ipa config-mod --defaultshell=/bin/bash' - 'ipa config-mod --homedirectory=/nfs/cistern/home' - 'ipa config-mod --emaildomain=vtluug.org' - 'ipa-managed-entries -e "UPG Definition" -p $DIRECTORY_MANAGER_PW disable' - Reboot one last time - Do NOT try to use letsencrypt. FreeIPA can be very fragile sometimes so just don't do it. - Run 'ansible-playbook main.yml -i hosts.cfg -u papatux -k -K --limit chimera.vtluug.org' to configure the rest of the system Configure baremetal hosts for freeipa login & correct folder permissions ------------------------------------------------------------------------ - Run 'ipa-client-install --domain=krb.vtluug.org --server=chimera.vtluug.org --principal=admin' - For NFS hosts, set the proper permissions for nfs folders in the 'NFS Configuration' now that freeipa is working - Run the ansible scripts from 2 sections ago, but with '--tags' instead of '--skip-tags' - Run 'mount -a' on nfs hosts to mount dirs with correct permissions Configure the new VMs (mostly) ------------------------------ - Run 'ansible-playbook main.yml -i hosts.cfg -u papatux -k -K --limit=vms' - This mostly configures all the new hosts Docker Services Config -------------------------------------- - Ansible should've pulled https://github.com/vtluug/apps.git into /nfs/cistern/docker/apps on meltdown, spectre, & sczi (+ any other docker hosts) - Follow directions in the README for each of the services Acidburn specific config ???