Deploying OpenStack using TripleO dynamic templates

Staring with the OpenStack Queens release, TripleO dynamically generates templates using jinja2 templates. This blog post demonstrates how to deploy your overcloud using templates that get dynamically generated at deploy-time.

The OpenStack version I will be using for this demo is Red Hat OpenStack Platform 14 (Rocky). Documentation for this can be found at docs.redhat.com as well as the upstream documentation found at docs.openstack.org. Templates used for this blog can be found on github.com. This blog assumes that you have deployed a successful undercloud (also known as Director).

For this deploy, I will be using the following systems:

  • Director – RHEL 7.6 VM with 24GB RAM and 8 vCPUs
  • 7 x HP DL360 G7 (3 Controllers and 4 Nova Nodes running Ceph (aka HCI or Hyper-Converged Infrastructure)

Step 1 – Introspection of your hardware.

The first thing in any TripleO deployment is to tell TripleO about your hardware. This is achieved by having TripleO PXE-boot your hardware and inspect it. Using an instackenv.json file containing information about your overcloud hardware, run the following command:


1
openstack overcloud node import ~/instackenv.json --introspect --provide

If all runs correctly, your output should look like this:


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
(undercloud) [stack@director14 ~]$ openstack overcloud node import ~/instackenv.json --introspect --provide
Waiting for messages on queue 'tripleo' with no timeout.
8 node(s) successfully moved to the "manageable" state.
Successfully registered node UUID a0e0d152-21c6-40f3-b19d-fbd92aac59e6
Successfully registered node UUID 7887510b-0d09-45cb-a826-ad1748fc00fa
Successfully registered node UUID 16be5f5f-43b2-4e57-ae1a-aed529afe94c
Successfully registered node UUID 94cea84a-c4a8-4094-9553-96cb6ed86a6e
Successfully registered node UUID 7917c925-c887-4639-8a11-5c66739f7412
Successfully registered node UUID af8262fa-1dbd-4b20-964a-ea1c6ec8a958
Successfully registered node UUID ba2ca7f9-3787-4add-8404-b9588264a26c
Successfully registered node UUID 8d0068cd-dfb2-4d6e-8a3a-6931b42e8b3a
Waiting for introspection to finish...
Waiting for messages on queue 'tripleo' with no timeout.
Introspection of node 8d0068cd-dfb2-4d6e-8a3a-6931b42e8b3a completed. Status:SUCCESS. Errors:None
Introspection of node 7887510b-0d09-45cb-a826-ad1748fc00fa completed. Status:SUCCESS. Errors:None
Introspection of node ba2ca7f9-3787-4add-8404-b9588264a26c completed. Status:SUCCESS. Errors:None
Introspection of node 16be5f5f-43b2-4e57-ae1a-aed529afe94c completed. Status:SUCCESS. Errors:None
Introspection of node af8262fa-1dbd-4b20-964a-ea1c6ec8a958 completed. Status:SUCCESS. Errors:None
Introspection of node 94cea84a-c4a8-4094-9553-96cb6ed86a6e completed. Status:SUCCESS. Errors:None
Introspection of node 7917c925-c887-4639-8a11-5c66739f7412 completed. Status:SUCCESS. Errors:None
Introspection of node a0e0d152-21c6-40f3-b19d-fbd92aac59e6 completed. Status:SUCCESS. Errors:None
Successfully introspected 8 node(s).
Waiting for messages on queue 'tripleo' with no timeout.
8 node(s) successfully moved to the "available" state.

Step 2 – Defining Network

Now that we have our hardware introspected, we need to define our networking environment for our overcloud. To do this, we fill out the template file named “network_data.yaml” and place it into our ~/templates directory


1
2
3
4
5
6
7
8
9
10
11
cp /usr/share/openstack-tripleo-heat-templates/network_data.yaml ~/templates
cat ~/templates/network_data.yaml
- name: Storage
  vip: true
  vlan: 30
  name_lower: storage
  ip_subnet: '172.16.1.0/24'
  allocation_pools: [{'start': '172.16.1.4', 'end': '172.16.1.250'}]
  ipv6_subnet: 'fd00:fd00:fd00:3000::/64'
  ipv6_allocation_pools: [{'start': 'fd00:fd00:fd00:3000::10', 'end': 'fd00:fd00:fd00:3000:ffff:ffff:ffff:fffe'}]
----> Truncated <----

I have modified my network_data.yaml file to suit my needs by reusing the existing sample’s naming convention, but providing my specific network information.


1
2
3
4
5
6
7
- name: Storage
  vip: true
  vlan: 6
  name_lower: storage
  ip_subnet: '172.16.6.0/24'
  allocation_pools: [{'start': '172.16.6.10', 'end': '172.16.6.18'}]
----> Truncated <----

Step 3 – Define overcloud roles

In this next step, we define the roles that we will be using to deploy our overcloud. We will be using the most common roles (Controller and Compute), but we will also be creating an additional role for our hyper-converged nodes. Fortunately, this process has been simplified with the inclusion of a list and a create tool. Use the following command to list all of the available roles in TripleO


1
2
3
4
5
6
7
8
9
openstack overcloud roles list
+-----------------------------+
| Role Name                   |
+-----------------------------+
| BlockStorage                |
| CephAll                     |
| CephFile                    |
| CephObject                  |
----> Truncated <----

Once you have determined the specific roles needed, we will use another command to create a roles_data.yaml file that defines everything we need to define our roles for our overcloud.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
openstack overcloud roles generate Controller ComputeHCI Compute -o ~/templates/roles_data.yaml

cat ~/templates/roles_data.yaml
###############################################################################
# File generated by TripleO
###############################################################################
###############################################################################
# Role: Controller                                                            #
###############################################################################
- name: Controller
  description: |
    Controller role that has all the controler services loaded and handles
    Database, Messaging and Network functions.
----> Truncated <----

Step 4 – Using our network and roles customizations to generate our templates

In this step, we will run a python script included in TripleO to read our network_data.yaml and roles_data.yaml file to create a temporary dump of the TripleO templates. This is useful to see what will be dynamically generated at deploy-time as well as allow us to grab some of the templates to make further customizations for our overcloud. Note that this python script must be run from the openstack-tripleo-heat-templates directory for it to work.


1
2
3
4
5
6
7
8
9
cd /usr/share/openstack-tripleo-heat-templates/
tools/process-templates.py -n ~/templates/network_data.yaml -r ~/templates/roles_data.yaml -o /tmp/templates
jinja2 rendering normal template net-config-bond.j2.yaml
rendering j2 template to file: /tmp/templates/./net-config-bond.yaml
jinja2 rendering normal template net-config-bridge.j2.yaml
rendering j2 template to file: /tmp/templates/./net-config-bridge.yaml
jinja2 rendering normal template net-config-linux-bridge.j2.yaml
rendering j2 template to file: /tmp/templates/./net-config-linux-bridge.yaml
----> Truncated <----

Using the “-o” switch instructs the python script to create an output directory to generate the templates to a directory specified. If you examine the /tmp/templates directory, you will see that a clone of the openstack-tripleo-heat-templates directory has been created with actual yaml files generated based on what was in the network_data.yaml and roles_data.yaml files. The content in this newly created directory is usable to deploy an overcloud, however, in my case I need to modify a few other items in order for my deploy to complete.

Step 5 – Nic Configs

TripleO templates do a great job in creating nic-config templates for commonly used deployments, however my setup is a bit different. Instead of having a dedicated NIC (eth0) for provisioning and then an OVS-Bond of 2 NICs (eth1 and eth2) for the OpenStack overcloud controlplane networks, I want to use 4 NICs in an LACP Linux Bond for provisioning and controlplane networks. To achieve this, I must modify the generated nic-config templates to suit my needs. I will start with the generated nic-configs for “bond-with-vlans” and modify them. I do this by copying the bond-with-vlans directory from the temporarily generated templates to my custom templates directory. You can diff the generated templates with my customized templates to see what changes I made.


1
2
3
4
5
mkdir ~/templates/nic-configs
cp -R /tmp/templates/network/config/bond-with-vlans ~/templates/nic-configs/
vi ~/templates/nic-configs/controller.yaml
vi ~/templates/nic-configs/compute.yaml
vi ~/templates/nic-configs/computehci.yaml

Step 6 – Network Customizations

We need to instruct TripleO during the deploy to make customizations such as custom nic-configs, bridge mapping, LACP configurations, MTU sizes, etc. This is achieved by making a yaml file defining those customizations and sourcing it during deployment time. Changes in this file will override any defaults in the generated templates. I have created a file named network.yaml which contains those customizations.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
cat templates/network.yaml
resource_registry:
  OS::TripleO::Controller::Net::SoftwareConfig: /home/stack/templates/nic-configs/controller.yaml
  OS::TripleO::ComputeHCI::Net::SoftwareConfig: /home/stack/templates/nic-configs/computehci.yaml
  OS::TripleO::Compute::Net::SoftwareConfig: /home/stack/templates/nic-configs/compute.yaml

parameter_defaults:
  NeutronBridgeMappings: 'datacentre:br-ex'
  NeutronFlatNetworks: 'datacentre'
  NeutronNetworkVLANRanges: 'datacentre:1:100'
  NeutronNetworkType: 'vxlan,vlan,flat'
  NeutronTunnelType: 'vxlan'
  NeutronExternalNetworkBridge: "''"

  # enable isolated metadata agent on controllers
  # https://access.redhat.com/solutions/2292841
  # Enable isolated Neutron metadata (allow metadata server in provider networks)
  NeutronEnableIsolatedMetadata: true

  # Set Jumbo MTU for tenant networks
  NeutronGlobalPhysnetMtu: 8896

  # DNS
  DnsServers: ['192.168.1.249', '192.168.0.250']
  CloudName: overcloud.lab.lan
  CloudDomain: lab.lan

  # Bonding options
  BondInterfaceOvsOptions: 'mode=802.3ad lacp_rate=1 updelay=1000 miimon=100'

  # Global DNS name for instances
  NeutronDnsDomain: lab.lan
  NeutronPluginExtensions: "qos,port_security,dns"
  ControllerExtraConfig:
    neutron::agents::dhcp::dnsmasq_local_resolv: true
    neutron::agents::dhcp::enable_isolated_metadata: true

Step 7 – Ceph Customization

You need to configure how TripleO will configure the ceph nodes by defining what disks will be used for OSDs, journal drive mappings, and pool sizing. Following the Ceph documentation for TripleO deployed Ceph, I created a file called ceph-custom-config.yaml and placed it in my ~/templates directory. This file will get called during deployment to override any defaults from the TripleO generated templates.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
parameter_defaults:
  CephConfigOverrides:
    mon_max_pg_per_osd: 3072
    journal_size: 5120
    osd_pool_default_size: 3
    osd_pool_default_min_size: 2
    osd_pool_default_pg_num: 128
    osd_pool_default_pgp_num: 128
  CephAnsibleDisksConfig:
    osd_scenario: lvm
    osd_objectstore: bluestore
    devices:
      - /dev/sdb
      - /dev/sdc
      - /dev/sdd
      - /dev/sde
      - /dev/sdf
  CephPools:
    - {"name": .rgw.root, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.control, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.meta, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": default.rgw.log, "pg_num": 16, "pgp_num": 16, "application": rados}
    - {"name": images, "pg_num": 128, "pgp_num": 128, "application": rbd}
    - {"name": metrics, "pg_num": 16, "pgp_num": 16, "application":openstack_gnocchi}
    - {"name": backups, "pg_num": 16, "pgp_num": 16, "application": rbd}
    - {"name": vms, "pg_num": 512, "pgp_num": 512, "application": rbd}
    - {"name": volumes, "pg_num": 256, "pgp_num": 256, "application": rbd}
  CephPoolDefaultPgNum: 128

Step 8 – Other overcloud customizations

I am statically assigning hardware nodes to each role by defining a scheduler hint. This ensures that a physical server is statically chosen for a specific server name (i.e. server in rack-u 1 will become controller1). I also define static hostnames, domain names, IPs, VIPs and SSL certificates for all nodes by copying their corresponding templates from /tmp/templates into my custom ~/templates directory and modifying them to suit my needs. Reference the following files for my customizations:

Step 9 – Deployment

Now we will use all of the customizations from previous steps to deploy our overcloud. To do so, I create a deploy-overcloud.sh script to include the deployment command as well as to source all of my custom templates.


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
openstack overcloud deploy --templates \
-r ~/templates/roles_data.yaml \
-n ~/templates/network_data.yaml \
-e ~/templates/containers-prepare-parameter.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-environment.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/network-isolation.yaml \
-e ~/templates/network.yaml \
-e ~/templates/scheduler_hints_env.yaml \
-e ~/templates/node-info.yaml \
-e ~/templates/ips-from-pool-all.yaml \
-e ~/templates/fixed-ip-vips.yaml \
-e ~/templates/inject-trust-anchor-hiera.yaml \
-e ~/templates/enable-tls.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ssl/tls-endpoints-public-ip.yaml \
-e ~/templates/ceph-custom-config.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-ansible.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/ceph-ansible/ceph-rgw.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/services/octavia.yaml \
-e /usr/share/openstack-tripleo-heat-templates/environments/disable-telemetry.yaml \
-e ~/templates/misc-settings.yaml

If all goes well, you should have a fully deployed overcloud. To modify your overcloud later after the initial deploy, you will use the deploy-overcloud.sh script with any modifications you need to apply.

Keystone Optimization

Issue:

The default configuration of Keystone is not necessarily tuned for anything specific.  As with other components of OpenStack, Keystone tuning based on use case and size are required in order for it to perform well.  When integrating Keystone with large LDAP environments (10k users+), if not tuned properly, you can suffer slow logons, API lag and an incredibly slow Horizon Dashboard.

Objective:

To Improve OpenStack Keystone performance when integrating with LDAP / Active Directory.

Software and Hardware Used for testing:

  • Red Hat OpenStack Platform Version 12
  • Ceph 2.4
  • 8 HP DL360 G7
    • 3 controllers (2 x Intel x5620, 24GB RAM, SSDs for OS)
    • 4 Hyper-converged nodes (2 x Intel x5620, 24GB RAM, 7 x 72GB 15KSAS)
    • 1 Utility server (dual-x5620, 72 GBRAM, and 4 x SSD RAID-10)
      • Red Hat OpenStack Platform Director version 12 running as a VM
        • 8 vCPU, 24 GBs RAM, 100GB disk)
  • Microsoft Windows Server 2008 Enterprise Active Directory

Environment tested

Windows 2k8 Server Enterprise with 10000+ users and nested groups.  Red Hat OpenStack Platform Version 12 Hyper-converged Reference architecture with 3 x Active Controller cluster, 4 x node Ceph/Nova nodes, with full network isolation, SSL certificates for OpenStack Endpoints, Horizon GUI and LDAPs.   Also, I am using Fernet Tokens for keystone instead of UUID Tokens.  This is strongly recommended as it alleviates the burden of token persistence in a database amongst other reasons.

NOTE:  Red Hat OSP 12 uses a containerized control plane so instead of editing conf files and restarting the services, I am editing the template files uses to create the containers and then restarting the containers.   If you are not running a containerized based control plane, edit the conf files and restart the respective service.

Results

These results demonstrate an OSP 12 Director-based deployment of an OSP 12 Overcloud with the default Active Directory configurations as documented.  I then enable and tune user & auth pooling as well as implement user & group filtering.  Lastly, I enable memcached to be used by Keystone for caching tokens, catalogs, and roles.

Download the token_issue_script used for testing.

With defaults (pooling disabled, no LDAP filtering, and no caching layer)
time ./token_issue.sh 100
getting 100 tokens
real 2m43.052s
user 0m5.627s
sys 0m1.642s
With Keystone pooling configured and LDAP user & group filtering
time ./token_issue.sh 100
getting 100 tokens
real 0m27.178s
user 0m5.698s
sys 0m1.470s
With Keystone pooling configured, LDAP user & group filtering, and Keystone Caching
time ./token_issue.sh 100
getting 100 tokens
real 0m10.569s
user 0m5.559s
sys 0m1.465s

Custom Settings

Keystone Pooling

Check current Keystone Pooling configuration

# Check settings
cat << EOF > check_keystone_pooling.sh
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_pool
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_size
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_max
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_delay
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_timeout
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_lifetime
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_auth_pool
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_size
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_connection_lifetime
EOF
chmod +x check_keystone_pooling.sh
./check_keystone_pooling.sh

Configure and enable Keystone Pooling # THIS WILL RESTART KEYSTONE

# Configure Keystone Pooling
cat << EOF > set_keystone_pooling.sh
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_pool True
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_size 200
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_max 20
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_retry_delay 0.1
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_timeout -1
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap pool_connection_lifetime 600
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap use_auth_pool True
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_size 1000
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap auth_pool_connection_lifetime 60
docker restart keystone
EOF
chmod +x set_keystone_pooling.sh
./set_keystone_pooling.sh

Results Pagination

Pagination is important in large LDAP environments as only a certain number of records will be returned by default.   This option defines the maximum number of results per page that keystone should request from the LDAP server when listing objects.  Add this to your keystone/domains/keystone.DOMAINNAME.conf file and restart Keystone

crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/domains/keystone.LAB.conf ldap page_size 2

To test the page_size, you can use ldapsearch with the following syntax.  Note that without specifying a page size, not all results will be returned.

$ ldapsearch -LLL -H ldap://ldapserver.domain.com -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' -w PASSWORD |grep sAMAccountName |wc -l
Size limit exceeded (4)
825

$ ldapsearch -LLL -H ldap://ldapserver.domain.com -E pr=1/noprompt -b 'dc=domain,dc=com' -D 'DOMAIN\USERNAME' -w PASSWORD |grep sAMAccountName |wc -l
10052

User and Group Filtering

Here are examples of user and group filters that can be used in the [ldap] section of /etc/keystone/domains/keystone.YOUR_DOMAIN_NAME.yaml.

For the user_filter example, I am filtering out all users excepts for those who are either a member of OpenStack-Admins OR OpenStack-Users.

user_filter=(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))

For the group_filter example, I am filtering out all groups  except for those with the ObjectClass of Group AND with a group named OpenStack, OpenStack-Admins, OR OpenStack-Users

group_filter=(&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))

To test LDAP user and group filtering, use the following ldapsearch syntax

Members of group1 AND group2
ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))'|grep sAMAccountName
Members of group1 OR group2
$ ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(|(memberOf=CN=OpenStack-Admins,OU=People,DC=lab,DC=lan)(memberOf=CN=OpenStack-Users,OU=People,DC=lab,DC=lan)))'|grep sAMAccountName
sAMAccountName: user1
sAMAccountName: user2
sAMAccountName: user3
sAMAccountName: user4
sAMAccountName: user5
List groups with names
$ ldapsearch -LLL -H ldap://192.168.1.249 -E pr=10000/noprompt -b 'dc=lab,dc=lan' -D 'LAB\USERNAME' -w PASSWORD '(&(objectClass=Group)(&(|(cn=OpenStack)(cn=OpenStack-Admins)(cn=OpenStack-Users))))'|grep distinguishedName
distinguishedName: CN=OpenStack,OU=People,DC=lab,DC=lan
distinguishedName: CN=OpenStack-Admins,OU=People,DC=lab,DC=lan
distinguishedName: CN=OpenStack-Users,OU=People,DC=lab,DC=lan

Keystone Caching

Check if caching is enabled

# Run on each controller as root
cat << EOF > ~/check_keystone_cache.sh
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache enabled
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend_argument
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf catalog caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf domain_config caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf federation caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf revoke caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf role caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token cache_on_issue
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity caching
crudini --get /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity cache_time
EOF
chmod +x ~/check_keystone_cache.sh
~/check_keystone_cache.sh

Set caching per on controllers # THIS WILL RESTART KEYSTONE

# Run on each controller as root
cat << EOF > ~/enable_keystone_cache.sh
API_IP=$(grep $HOSTNAME.internalapi /etc/hosts|awk '{print $1}')
echo $API_IP
systemctl enable memcached
systemctl start memcached
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache enabled true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend dogpile.cache.memcached
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf cache backend_argument url:$API_IP:11211
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf catalog caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf domain_config caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf federation caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf revoke caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf role caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf token cache_on_issue true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity caching true
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf identity cache_time 600
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf eventlet_server admin_workers 72
crudini --set /var/lib/config-data/puppet-generated/keystone/etc/keystone/keystone.conf eventlet_server public_workers 72
docker restart keystone
EOF
chmod +x ~/enable_keystone_cache.sh
~/enable_keystone_cache.sh

Verify memcached hits

You can run the following command from your controllers to watch cache hits and misses.  The hits should be increasing which means caching is working.

# Run on each controller
API_IP=$(grep $HOSTNAME.internalapi /etc/hosts|awk '{print $1}')
watch "memcached-tool $API_IP:11211 stats |grep -A2 get_hits"

Ceph RADOS Gateway Multi-Site replication with Active Directory accounts

Objective:

Active Directory users in the east coast region will be able to create Containers and Objects in object storage which will be replicated and consumed by Active Directory users in the west coast region and vice a versa.

Prerequisites:

The Following is assumed for this deployment:

  1. Two OpenStack deployments: East Coast and West Coast Datacenters respectively
  2. Active Directory Domain Controllers replicated in each site
  3. Ceph clusters configured for each site
  4. Keystone deployments independent from each other
  5. Keystone using LDAP (Active Directory) for authentication
  6. Keystone utilizing Fernet tokens

Software used for demonstration

  1. Red Hat OpenStack Platform (RH OSP) 11 (based on OpenStack Ocata release)
  2. Red Hat Enterprise Linux (RHEL) 7.4
  3. 2 RH OSP Director 11 virtual machines (east and west respectively)
  4. Ceph 2.2

Deployment:

To deploy the OpenStack environments for each site, I deployed RH OSP 11 Director at each site (director11east.lab.lan & director11west.lab.lan) on virtual machines consisting of 24 GB RAM, 8 vCPUs, 60GB Disk and 2 NICs (public and provisioning) each.

I used the following triple-o templates and custom templates to create the two overcloud
deployment changing only the hostnames, IPs, subnets and VLANs between east and west
overclouds: east templates & west templates

Once you have deployed both OpenStack sites (east and west respectively), reference my ad-post.sh script in the templates above to create the Keystone Domain which will point to Active Directory for authentication.  Within this newly created KeystoneV3 Domain, create a new tenant called “east” in both the East and West OpenStack environments.

IMPORTANT:  Even though we have created a tenant in both OpenStack environment with the same name, the Unique ID associated with that tenant name is different for each cluster.  We need to make the West Coast OpenStack Cluster have the same ID for the “east” tenant to match what the East Coast OpenStack Cluster has. 

On one of the OpenStack controllers in the East Coast OpenStack Environment, use the following command to obtain the ID associated with the tenant named “east”

1
source overcloudrc.v3
1
openstack project show list --domain LAB

On one controller in the West Coast OpenStack Environment, run the following commands as root. RUN THIS ONLY ONCE AND FROM ONLY ONE CONTROLLER. IT WILL REPLICATE TO THE OTHERS

1
mysql keystone

# View the field you are about to change

1
select * from project where name='east';

# change the existing ID to the ID from the East Coast OpenStack Environment
# update id in DB with project ID from east overcloud

1
update project set id='ID_GATHERED_FROM_EAST_OPENSTACK_ENV' where name='east';

East Coast Cluster

Once each OpenStack site has been deployed, we need to configure the Ceph RADOS Gateway REALMs.   We will start with the East Coast OpenStack Environment and make it the Master REALM.  This script is intended for new Ceph Clusters with absolutely ZERO data on it.  THIS WILL DESTROY EXISTING CEPH POOLS AND CREATE NEW ONES!!!!   The PG numbers I used are based on my environment, but you can generate you own specific settings by using the Red Hat Ceph Placement Groups calculator: https://access.redhat.com/labsinfo/cephpgc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
east_pub_endpoint=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
realm_name=redhat

# Create a name for your zonegroup. This is global
zonegroup=us

# Create a name for your zone. This is local to each ceph deployment
rgw_zone=us-east

# AD Test User
ad_test_user=kholden

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm create --rgw-realm=$realm_name --default
radosgw-admin zonegroup create --rgw-zonegroup=$zonegroup --endpoints=$east_pub_endpoint --rgw-realm=$realm_name --master --default
radosgw-admin zone create --rgw-zonegroup=$zonegroup --rgw-zone=$rgw_zone --master --default --endpoints=$east_pub_endpoint
radosgw-admin zonegroup remove --rgw-zonegroup=default --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

# create new pools from ceph us.east pools

ceph osd pool create us-east.rgw.intent-log 16
ceph osd pool set us-east.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.log 16
ceph osd pool set us-east.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.data 128
ceph osd pool set us-east.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.extra 16
ceph osd pool set us-east.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.buckets.index 16
ceph osd pool set us-east.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.control 16
ceph osd pool set us-east.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.gc 16
ceph osd pool set us-east.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.data.root 16
ceph osd pool set us-east.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.usage 16
ceph osd pool set us-east.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users 16
ceph osd pool set us-east.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.email 16
ceph osd pool set us-east.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.swift 16
ceph osd pool set us-east.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.users.uid 16
ceph osd pool set us-east.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-east.rgw.meta 16
ceph osd pool set us-east.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

radosgw-admin period update --commit
radosgw-admin period update --commit
radosgw-admin zone delete --rgw-zone=default
radosgw-admin period update --commit
radosgw-admin zonegroup delete --rgw-zonegroup=default
radosgw-admin period update --commit
radosgw-admin user create --uid="synchronization-user" --display-name="Synchronization User" --system
access_key=$(radosgw-admin user info --uid="synchronization-user"|grep access_key|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')
secret_key=$(radosgw-admin user info --uid="synchronization-user"|grep secret|awk '{print $2}'|sed 's/^"\(.*\)".*/\1/')

radosgw-admin zone modify --rgw-zone=us-east --access-key=$access_key --secret=$secret_key
radosgw-admin period update --commit
cp /etc/ceph/ceph.conf ~/ceph.conf_backup
echo "rgw_zone=us-east" &gt;&gt; /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring

systemctl stop ceph-radosgw.target
sleep 10
systemctl start ceph-radosgw.target

# Check RGW sync status
radosgw-admin sync status

# Create an openstack credentials file with your AD, Domain, and Tenant information

cp eastrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’
source ken
openstack container list

Next we need to configure the West OpenStack environment to be part of the Ceph RADOS Gateway REALM (I called the REALM “redhat”) that we created in the previous step.

IMPORTANT:  For the master realm setup, I created both an Access Key and Secret Key to be used for authentication of the REALM.  YOU MUST USE THE ACCESS AND SECRET KEYS USED IN THE PREVIOUS STEP FOR THE SCRIPT BELOW.  DO NOT USE THE KEY’S I HARDCODED BELOW

West Coast Ceph Cluster:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
#!/bin/bash
# East Coast Ceph RADOS Gateway (Master REALM) configuration script
# To be run only once on east coast ceph rados gateway controller
# To be run only as root

if ! [ $(id -u) = 0 ]; then
echo "I am not root!"
exit 1
fi

if ! [[ -f ~/eastrc.v3 ]] ; then
echo "No RC file exists in /root/"
exit
fi

unset OS_PASSWORD OS_AUTH_URL OS_USERNAME OS_TENANT_NAME OS_NO_CACHE OS_IDENTITY_API_VERSION OS_PROJECT_DOMAIN_NAME OS_USER_DOMAIN_NAME
source eastrc.v3

## Variables ##

# gather ceph rados gateway public endpoint URL
EAST_PUB_ENDPOINT=$(crudini --get /etc/ceph/ceph.conf client.radosgw.gateway rgw_keystone_url)

# Create a name for your realm. This is global
REALM_NAME=redhat

# Create a name for your zonegroup. This is global
ZONE_GROUP=us

# Create a name for your zone. This is local to each ceph deployment
RGW_ZONE=us-east

# AD Test User
AD_TEST_USER=kholden

# AD Test User Password
AD_TEST_USER_PASSWORD

# AD Domain Name
AD_DOMAIN=LAB

# OpenStack Project Name
PROJECT_NAME=east

### Script ###

tar cvf ~/ceph.backup.tar /etc/ceph

radosgw-admin realm pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin realm default --rgw-realm=redhat
radosgw-admin period pull --url=$east_pub_endpoint --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr
radosgw-admin zone create --rgw-zonegroup=us --rgw-zone=us-west --access-key=1HMKB05PQQ78YV5US3KY --secret=OuWfjeqO7Z15hUr5FLf37uUph8XWNb3Sylctrvpr --endpoints=http://192.168.15.120:8080
radosgw-admin zone delete --rgw-zone=default
ceph osd dump | grep pool | grep default | awk -F\' '{print $2}' | xargs -P8 -I{} rados rmpool {} {} --yes-i-really-really-mean-it

ceph osd pool create us-west.rgw.intent-log 16
ceph osd pool set us-west.rgw.intent-log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.log 16
ceph osd pool set us-west.rgw.log size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.data 128
ceph osd pool set us-west.rgw.buckets.data size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.extra 16
ceph osd pool set us-west.rgw.buckets.extra size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.buckets.index 16
ceph osd pool set us-west.rgw.buckets.index size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.control 16
ceph osd pool set us-west.rgw.control size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.gc 16
ceph osd pool set us-west.rgw.gc size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.data.root 16
ceph osd pool set us-west.rgw.data.root size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.usage 16
ceph osd pool set us-west.rgw.usage size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users 16
ceph osd pool set us-west.rgw.users size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.email 16
ceph osd pool set us-west.rgw.users.email size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.swift 16
ceph osd pool set us-west.rgw.users.swift size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.users.uid 16
ceph osd pool set us-west.rgw.users.uid size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

ceph osd pool create us-west.rgw.meta 16
ceph osd pool set us-west.rgw.meta size 2
while [ $(ceph -s | grep creating -c) -gt 0 ]; do echo -n .;sleep 1; done

cp /etc/ceph/ceph.conf ~/
echo "rgw_zone=us-west" &gt;&gt; /etc/ceph/ceph.conf
chown ceph /etc/ceph/*.keyring
radosgw-admin period update --commit

 

Run 

1
systemctl stop ceph-radosgw.target

# Wait until you see the following message
# Broadcast message from systemd-journald@east-controller1.lab.lan (Fri 2017-08-25 11:57:53 UTC):
#
# haproxy[35919]: proxy ceph_rgw has no server available!

Then run 

1
systemctl start ceph-radosgw.target

# Create openstack credentials file with AD user, Domain, and Tenant information:
cp westrc.v3 ken
change username from ‘admin’ to  ‘kholden’ (MY AD USERNAME)
change domain settings from ‘default’ to ‘lab’
change project from ‘admin’ to ‘east’
change password to AD password for user ‘kholden’

logout and back in to clear ENV variables
source ken
openstack container list

You should now be able to create containers and objects in the east coast datacenter and then see them replicate to the west coast datacenter.  You should be able to also create containers and objects in the west data center and see them replicate to the east datacenter