GitLab

remove Ansible.set_inventories method

See merge request main/infrastructure!342

rationale: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments

This method was used only by the unit tests.

Avoid timeout from SSH client: don't restart SSH service

See merge request main/infrastructure!341

- use reload instead of restart
- reload service as soon as possible
- reload only when the default SSH port isn't used

Workaround 'Connection timed out' from SSH client. These timeouts occur
while SSH service is restarted:

    # journalctl -o short-precise -u sshd
    18:22:57.447216 copy-to-host systemd[1]: Started OpenBSD Secure Shell server.
    18:22:57.878128 copy-to-host sshd[601]: Did not receive identification string from 82.123.240.182
    18:23:20.370278 copy-to-host sshd[580]: Received signal 15; terminating.
    18:23:20.371926 copy-to-host systemd[1]: Stopping OpenBSD Secure Shell server...
    18:23:20.373162 copy-to-host systemd[1]: Stopped OpenBSD Secure Shell server.
    18:23:20.375161 copy-to-host systemd[1]: Starting OpenBSD Secure Shell server...
    18:23:20.392807 copy-to-host sshd[885]: Server listening on :: port 22.
    18:23:20.393059 copy-to-host systemd[1]: Started OpenBSD Secure Shell server.

    # journalctl -o short-precise -u cloud-final.service
    18:22:58.139066 copy-to-host systemd[1]: Starting Execute cloud user/final scripts...
    18:22:58.724808 copy-to-host cloud-init[606]: Provision began: 2020-11-09 18:22:58.724631
    [...]
    18:23:20.502234 copy-to-host cloud-init[606]: Cloud-init v. 20.2 running 'modules:final' at Mon, 09 Nov 2020 18:22:58 +0000. Up 14.85 seconds.
    18:23:20.502234 copy-to-host cloud-init[606]: Cloud-init v. 20.2 finished at Mon, 09 Nov 2020 18:23:20 +0000. Datasource DataSourceOpenStackLocal [net,ver=2].  Up 36.
    Nov 09 18:23:20.544560 copy-to-host systemd[1]: Started Execute cloud user/final scripts.

'log-cli-level' pytest switch enable log output too

See merge request main/infrastructure!340

It doesn't only control the test log level.

test_clone_clobber don't use OpenStack API

See merge request main/infrastructure!339

Remove extraneous line

See merge request main/infrastructure!338

Fix "'dict' object has no attribute 'strip'" exception which has been
introduced by 252ce4e0.

Destroy ressources created by test_openstack_create_or_update

See merge request main/infrastructure!337

Call o.destroy_everything(None) instead of using openstack_name fixture.

'openstack_name' fixture can not be used to destroy the created
ressources. Indeed the prefix can not be reused (there is no way to
use the prefix with ServiceOpenStack.create_or_update).

This issue was encountered while using the following command:

    $ tests/run-tests.sh tox -e py3 -- -k "test_openstack_create_or_update or test_clone_and_destroy" tests/enough/
    [...]
    tests/enough/common/test_common_service.py::test_openstack_create_or_update PASSED
    tests/enough/common/test_init.py::test_clone_and_destroy FAILED

    =================================== FAILURES ===================================
    ____________________________ test_clone_and_destroy ____________________________

    @pytest.mark.skipif('SKIP_OPENSTACK_INTEGRATION_TESTS' in os.environ,
                        reason='skip integration test')
    def test_clone_and_destroy(tmpdir):
	[...]
        original = Enough(original_config_dir, settings.SHARE_DIR,
                          domain=original_domain,
                          driver='openstack')
    >   assert original.openstack.o.server.list().strip() == ''
    E   AssertionError: assert '+-----------...---+--------+' == ''
    E     - +-----------+--------+---------------------+-----------+--------+
    E     - | Name      | Status | Networks            | Image     | Flavor |
    E     - +-----------+--------+---------------------+-----------+--------+
    E     - | bind-host | ACTIVE | Ext-Net=51.68.95.206| Debian 10 | s1-2   |
    E     - +-----------+--------+---------------------+-----------+--------+

    tests/enough/common/test_init.py:38: AssertionError

ansible_utils: use only one bake method

See merge request main/infrastructure!332

use unchanged 'json' format instead of 'value'

See merge request main/infrastructure!335

Fix this issue:

    $ tests/run-tests.sh tox -e py3 -- -k test_create_missings tests/enough/common/test_init.py
    [...]
    Traceback (most recent call last):
      File "/srv/share/src/ansible/enough.community/infrastructure.git/tests/enough/common/test_init.py", line 290, in test_create_missings
	assert bind_internal_ip == internal_dns
    AssertionError: assert '10.30.20.120' == "['10.30.20.120']"
      - 10.30.20.120
      + ['10.30.20.120']
      ? ++            ++

    During handling of the above exception, another exception occurred:

    Traceback (most recent call last):
      File "/srv/share/src/ansible/enough.community/infrastructure.git/enough/common/retry.py", line 19, in f_retry
	return f(*args, **kwargs)
      File "/srv/share/src/ansible/enough.community/infrastructure.git/enough/common/openstack.py", line 399, in destroy_everything
	raise OpenStackLeftovers('scheduled removal of ' + ' '.join(leftovers))
    enough.common.openstack.OpenStackLeftovers: scheduled removal of stack(bind-host) network(internal)
    openstack.py               341 INFO     destroy_everything acting
    openstack.py                34 INFO     []

Destroy volumes and snapshots using ID instead of Name

See merge request main/infrastructure!334

names might not be unique.

Fix this issue:

    sh.py        628 INFO <Command 'py3/bin/openstack --os-cloud production volume list --format=value -c Name'>: starting process
    sh.py        628 INFO <Command 'py3/bin/openstack --os-cloud production volume list --format=value -c Name', pid 120>: process started
    openstack.py  34 INFO enough_test_1604579181
    openstack.py  34 INFO enough_test_1604579181
    sh.py        628 INFO <Command 'py3/bin/openstack --os-cloud production volume delete enough_test_1604579181'>: starting process
    sh.py        628 INFO <Command 'py3/bin/openstack --os-cloud production volume delete enough_test_1604579181', pid 124>: process started
    openstack.py  35 INFO Failed to delete volume with name or ID 'enough_test_1604579181-test-destroy-volume':
			  More than one volume exists with the name 'enough_test_1604579181-test-destroy-volume'.
    openstack.py  35 INFO 1 of 1 volumes failed to delete.

Fetch volume id before two volumes of the same name exist

See merge request main/infrastructure!333

Fix regression introduced by 436b297b:

    $ run-tests.sh tox -e py3 -- -s -k test_restore_local tests/enough/common/test_init.py
    [...]
    retry.py                    21 INFO     OpenStack.destroy_everything: scheduled removal of snapshot(2020-11-04-sample-volume) stack(sample-host) volume(2020-11-04-sample-volume) network(internal), Retrying in 1 seconds...
    retry.py                    23 DEBUG
    Traceback (most recent call last):
      File "infrastructure/tests/enough/common/test_init.py", line 267, in test_restore_local
	assert e == e.restore_local(snapshot)
      File "infrastructure/enough/common/__init__.py", line 249, in restore_local
	self.openstack.replace_volume(host, volume, delete_volume=False)
      File "infrastructure/enough/common/openstack.py", line 565, in replace_volume
	volume_id = self.o.volume.show('--format=value', '-c', 'id', volume).stdout.strip()
      File "infrastructure/.tox/py3/lib/python3.7/site-packages/sh.py", line 1427, in __call__
	return RunningCommand(cmd, call_args, stdin, stdout, stderr)
      File "infrastructure/.tox/py3/lib/python3.7/site-packages/sh.py", line 774, in __init__
	self.wait()
      File "infrastructure/.tox/py3/lib/python3.7/site-packages/sh.py", line 792, in wait
	self.handle_command_exit_code(exit_code)
      File "infrastructure/.tox/py3/lib/python3.7/site-packages/sh.py", line 815, in handle_command_exit_code
	raise exc
    sh.ErrorReturnCode_1:

      RAN: infrastructure/.tox/py3/bin/openstack --os-cloud production volume show --format=value -c id 2020-11-04-sample-volume

rationales:
- Ansible.bake_ansible_playbook & Playbook.bake were both baking
  ansible-playbook command
- Ansible.bake_ansible_playbook ignores environment variables
  Playbook.bake doesn't (same fix as 24760bb7)
- Playbook.get_variable_hosts was using inventories twice

handle python-openstackclient new output format

See merge request main/infrastructure!331

python-openstackclient==5.4.0 is used since bcc7df47.
This version changes the output of some commands (another example:
50694679).

    $ openstack --version
    openstack 5.3.1
    $ openstack server show --format=value -c volumes_attached test
    id='aae6e78e-ef6c-46a8-9469-7b3789eaf92e'

    $ openstack --version
    openstack 5.4.0
    $ openstack server show --format=value -c volumes_attached test
    [{'id': 'aae6e78e-ef6c-46a8-9469-7b3789eaf92e'}]

psono: do *not* override settings.yaml

Closes #285

See merge request main/infrastructure!330

Otherwise it will loose the server keys

Fixes: main/infrastructure#285

icinga: doc: fix incorrect file for email documentation

See merge request main/infrastructure!329

psono first implementation

Closes #283, #277, #278, #284, #279, #280, #282, and #262

See merge request main/infrastructure!326