Commit 9c95ad87 authored by Loïc Dachary's avatar Loïc Dachary

Merge branch 'wip-key' into 'master'

remove support for infrastructure_key at the root of the repo

See merge request !394
parents 13ab12f6 2ee0d34f
Pipeline #1577 passed with stage
in 20 minutes and 26 seconds
......@@ -6,7 +6,6 @@ pytestdebug.log
docs/_build/
inventories/common/01-hosts.yml
*.pyc
/infrastructure_key*
secret
tests/domain.yml
tests/clouds.yml
......
......@@ -8,15 +8,12 @@ look at the `list of bugs and features
<https://lab.enough.community/main/infrastructure/issues>`__,
otherwise keep reading.
.. note:: If you want to contribute to the Enough code base, take
a look at `the repository <https://lab.Enough.community/main/app>`__.
Resources
---------
* Repository and issue tracking: http://lab.enough.community/main/infrastructure
* Forum: https://forum.enough.community/
* Instant messenging: https://chat.enough.community/enough/
* Chat: https://chat.enough.community/enough/
* License: `AGPLv3 <https://lab.enough.community/main/infrastructure/blob/master/LICENSE>`__
* :doc:`Who's who <team>`
......@@ -62,8 +59,6 @@ The working directory, in the container, is the root of the
Running tests that do not require OpenStack
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
There is no need to have OpenStack credentials to run these tests.
* ``PYTEST_ADDOPTS='-m "not openstack_integration"' tests/run-tests.sh``
......@@ -75,9 +70,9 @@ Introduction
The tests running without OpenStack only cover a fraction of what
Enough does. To verify that a playbook actually works, it needs to be
run on a live host and tests must check that is working. For instance
the tests for weblate request that the weblate server sends a mail and
verify it is relayed by the postfix server.
run on a live host and tests must check that is working. For instance
the tests for `Weblate` request that the `Weblate` server sends a mail and
verify it reaches the postfix server.
When modifying a role or a playbook in the directory `playbooks/ABC`
one is expected to add a test for the new behavior and verify it runs
......@@ -97,10 +92,6 @@ The private key is named `infrastructure_key`, the public key is named
`infrastructure_key.pub` and both are located in
`.tox/ABC/.pytest_cache/d/dotenough/ABC.test/` directory.
If `infrastructure_key` and `infrastructure_key.pub` files exist at the
top level directory of the Git repository, then these files are copied to
`.tox/ABC/.pytest_cache/d/dotenough/ABC.test/` instead of being generated.
Obtain an API token
+++++++++++++++++++
......@@ -144,6 +135,38 @@ or
...
It must define two cloud environment: `production` and `clone` (for backup
restoration testing purposes). Here is a complete example:
.. code::
---
openstack_provider: fuga
clouds:
production:
auth:
auth_url: "https://identity.api.ams.fuga.cloud:443/v3"
user_id: "6a79dfb7410c4884fceb23031189b"
password: "qecOSdBAH6ZjE4M2UnZbnnWdsZihe"
user_domain_id: "99009ec244eebb85827488bb2aed4"
project_domain_id: "9900e2c244eebb85827488bb2aed4"
project_id: "203e72ec8a85b9dc808719e452902"
region_name: "ams"
interface: "public"
identity_api_version: 3
clone:
auth:
auth_url: "https://identity.api.ams.fuga.cloud:443/v3"
user_id: "3b40cf2cb71b4bdc95c009347445f"
password: "RBX0S2BdXWlBztUKkPWcAfnNFSNNj"
user_domain_id: "de844dabe43948cb87ed24e2d5c438a9"
project_domain_id: "de8abe43948cb87ed24e2d5c438a9"
project_id: "82cb2f62a70f5928e3a4686622e39"
region_name: "ams"
interface: "public"
identity_api_version: 3
Running
+++++++
......@@ -167,7 +190,7 @@ to skip some steps to speed up test debugging:
--enough-no-tests Do not run the tests step
--enough-no-destroy Do not run the destroy step
...
$ tests/run-tests.sh tox -e openvpn -- --enough-no-destroy playbooks/openvpn/tests
$ tests/run-tests.sh tox -e authorized_keys -- --enough-no-destroy playbooks/authorized_keys/tests
The domain name used for testing is in
`.pytest_cache/d/dotenough/bind.test/inventory/group_vars/all/domain.yml`,
......@@ -175,40 +198,6 @@ where `bind` must be replaced by the name of the service. It is handy
for debugging (i.e. browsing the web interface of a service, ssh to a
machine that failed to run properly, etc.)
If a test fails, it will **not** destroy the resources provisioned
for the test, they must be destroyed explicitly with something like:
* ``tests/run-tests.sh tox -e openvpn -- --enough-no-create --enough-no-tests playbooks/openvpn/tests``
OpenStack providers
+++++++++++++++++++
OVH is the OpenStack provider used by default. There is one another OpenStack
provider supported: Fuga. The OpenStack provider can be chosen with the pytest
``--provider`` switch.
In order to execute linter (``flake8``), documentation (``doc``) and python tests
(``py3``), using Fuga provider for the python tests:
.. code-block:: sh
$ PYTEST_ADDOPTS='--provider=fuga' tests/run-tests.sh
The following command allows to execute the OpenStack integration tests only
(using Fuga):
.. code-block:: sh
$ tests/run-tests.sh tox -e py3 -- --enough-no-destroy \
--provider=fuga playbooks/icinga/tests
This command run the icinga service tests using Fuga provider:
.. code-block:: sh
$ tests/run-tests.sh tox -e icinga -- --enough-no-destroy \
--provider=fuga playbooks/icinga/tests
Upgrade testing
---------------
......@@ -221,9 +210,9 @@ a given Enough version (`2.0.7` for instance), use:
$ tests/run-upgrade-tests.sh 2.0.7 icinga
...
`run-tests.sh` performs the following steps:
`run-upgrade-tests.sh` performs the following steps:
* checkout the ``2.0.7`` tag into ``../infrastructure-versions/1.0.7/infrastructure``
* checkout the ``2.0.7`` tag into ``../infrastructure-versions/2.0.7/infrastructure``
* run ``tox -e icinga`` from the ``2.0.7`` directory and keep the hosts
* run ``tox -e icinga`` from the current version, re-using the hosts with the icinga version installed from ``2.0.7``
......@@ -231,8 +220,8 @@ ssh to a host under test
------------------------
If `tests/run-tests.sh tox -e chat` was run and the hosts have not
been destroyed because the tests failed, the following can be used to
ssh on a host:
been destroyed because the `--enough-no-destroy` option was set,
the following can be used to ssh on a host:
::
......@@ -251,7 +240,7 @@ To run the tests manually within the test container:
$ tests/run-tests.sh bash
user@6642e3759c43:~/infrastructure$ tox -e flake8
Use the ``--log-cli-level`` switch in order:
Use the ``--log-cli-level`` switch in order to:
* enable log display during test run (live logging)
* control the test log level
......@@ -262,21 +251,19 @@ For example:
$ tests/run-tests.sh tox -e py3 -- --log-cli-level=INFO -s -x tests/enough/common/test_openstack.py
`--log-cli-level <https://docs.pytest.org/en/stable/logging.html#live-logs>`_ and following switches are `pytest ones <https://docs.pytest.org/en/stable/contents.html>`_.
`--log-cli-level <https://docs.pytest.org/en/stable/logging.html#live-logs>`_ and following switches are from `pytest <https://docs.pytest.org/en/stable/contents.html>`_.
To execute only one test:
* ``tests/run-tests.sh tox -e py3 -- tests/enough/common/test_openstack.py::test_heat_definition``
Control-C won't work if you're trying to stop the tests, ``docker kill enough-tox`` should be used instead.
There should not be any leftover after a test involving OpenStack
fails, because the fixtures are supposed to thoroughly cleanup. But
bugs are to be expected in a test environment and it may be necessary
to manually remove leftovers, using the ``openstack`` command like so:
* ``tests/run-tests.sh env OS_CLIENT_CONFIG_FILE=~/.enough/dev/inventory/clouds.yml openstack --os-cloud production stack list``
* ``tests/run-tests.sh env OS_CLIENT_CONFIG_FILE=~/.enough/dev/clone-clouds.yml openstack --os-cloud production stack list``
* ``tests/run-tests.sh env OS_CLIENT_CONFIG_FILE=tests/clouds.yml openstack --os-cloud production stack list``
* ``tests/run-tests.sh env OS_CLIENT_CONFIG_FILE=tests/clouds.yml openstack --os-cloud clone stack list``
In case leftover are manually deleted using ``stack delete`` command, the
following directory must be manually removed: ``.tox/<test environment>/.pytest_cache/``,
......@@ -302,10 +289,10 @@ Check the value of an ansible variable:
-i .tox/icinga/.pytest_cache/d/dotenough/icinga.test/inventory \
-m debug -avar=ansible_host
Ansible repository layout
-------------------------
Repository layout
-----------------
The `ansible repository
The `ansible part of the repository
<http://lab.enough.community/main/infrastructure/>`_ groups playbooks
and roles in separate directories to reduce the number of files to
consider when working on improving a playbook or a service.
......@@ -313,7 +300,6 @@ consider when working on improving a playbook or a service.
* ``playbooks/authorized_keys``: distribute SSH public keys
* ``playbooks/backup``: daily VMs snapshots
* ``playbooks/bind``: DNS server and client
* ``playbooks/letsencrypt-nginx``: nginx reverse proxy with letsencrypt integration
* ``playbooks/icinga``: resources monitoring
* ``playbooks/infrastructure``: VMs creation and firewalling
* ``playbooks/postfix``: outgoing mail relay for all VMs
......
......@@ -9,7 +9,6 @@ RUN curl -L "https://github.com/docker/compose/releases/download/1.24.0/docker-c
ENV REQUESTS_CA_BUNDLE /etc/ssl/certs
RUN useradd --shell /bin/bash debian && echo "debian ALL=(ALL) NOPASSWD:ALL" >> /etc/sudoers
COPY infrastructure_key.pub /home/debian/.ssh/authorized_keys
RUN chown -R debian:debian /home/debian
# allow deb-systemd-invoke to start systemd services
......
......@@ -14,15 +14,10 @@ def make_config_dir(domain, enough_dot_dir):
def prepare_config_dir(domain, enough_dot_dir):
os.environ['ENOUGH_DOMAIN'] = domain
config_dir = make_config_dir(domain, enough_dot_dir)
# Reuse existent SSH key. If it doesn't exist, then a key will be
# generated by DotEnough.ensure_ssh_key.
if os.path.isfile('infrastructure_key'):
shutil.copy('infrastructure_key', f'{config_dir}/infrastructure_key')
shutil.copy('infrastructure_key.pub', f'{config_dir}/infrastructure_key.pub')
all_dir = f'{config_dir}/inventory/group_vars/all'
if not os.path.exists(all_dir):
os.makedirs(all_dir)
shutil.copyfile('inventory/group_vars/all/clouds.yml', f'{all_dir}/clouds.yml')
shutil.copyfile('tests/clouds.yml', f'{all_dir}/clouds.yml')
shutil.copyfile('inventory/group_vars/all/provision.yml', f'{all_dir}/provision.yml')
open(f'{all_dir}/certificate.yml', 'w').write(textwrap.dedent(f"""\
---
......
......@@ -12,7 +12,7 @@ from enough.common.openstack import OpenStack
from tests import prepare_config_dir
@pytest.fixture(autouse=True, scope='session')
@pytest.fixture(autouse=True)
def debug_enough():
logging.getLogger('enough').setLevel(logging.DEBUG)
......@@ -108,13 +108,9 @@ def pytest_runtest_teardown(item, nextitem):
check_generated_files = [
'inventory/group_vars/all/private-key.yml',
'infrastructure_key',
'infrastructure_key.pub',
]
if os.environ.get('GITLAB_CI'):
# contributors are allowed to define such files, CI isn't
check_generated_files.extend((
'infrastructure_key',
'infrastructure_key.pub',
))
# first remove all these files in order to not interfere with the next
# tests then fail
found = []
......
......@@ -20,9 +20,6 @@ function prepare_environment() {
function prepare_repository() {
git submodule --quiet sync
git submodule --quiet update --init --recursive
if test -f infrastructure_key ; then
chmod 600 infrastructure_key
fi
}
function prepare_inventory() {
......
Markdown is supported
0% or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment