Initial installation
Obtaining the source code
When developing, the first step is always to obtain the source code of Kadi4Mat using git, which might have to be installed first using:
sudo apt install git
Before actually obtaining the code, it is recommended to create a fork of the main
repository. Afterwards, the code can be cloned into
a local directory via SSH (recommended) or HTTPS, the latter being shown in the
following command. Note that the <username>
placeholder needs to be substituted with
the correct username/namespace that the new fork resides in:
git clone https://gitlab.com/<username>/kadi.git ${HOME}/workspace/kadi
This will copy the code into the workspace/kadi
directory in the current user’s home
directory. This directory can of course be changed freely, however, the rest of this
documentation assumes that the source code resides there and that all commands are run
within this directory:
cd ${HOME}/workspace/kadi
To be able to update the code from the central repository, it can be added as an
additional remote, often called upstream
(note that the default remote after
cloning, pointing to the new fork, is always called origin
):
git remote add upstream https://gitlab.com/iam-cms/kadi.git
Installing the dependencies
Tip
Some of the dependent services mentioned in this section, namely PostgreSQL, Redis and (optionally) Elasticsearch, can also be run via Docker containers instead of installing and running them separately.
Python and Virtualenv
As the backend code of the application is based on the Flask web framework and multiple other Python libraries, Python 3 needs to be installed. This should generally be the case already, otherwise it can be installed using:
sudo apt install python3
Note
Note that a Python version >=3.9 and <3.14 is required. The currently installed version can be checked using:
python3 --version
If the currently installed Python version is not suitable, please see the instructions on how to install and use a different Python version for production or development installations.
To create an isolated environment for the application and its dependencies, Virtualenv is used, which can be installed like this:
sudo apt install virtualenv
Libraries
Some external libraries and tools are required as additional dependencies, which can be installed using:
sudo apt install libmagic1 build-essential python3-dev python3-venv libpq-dev libpcre3-dev
PostgreSQL
The RDBMS used in the application is PostgreSQL. Any up to date version >=13 should work, which can be installed like this:
sudo apt install postgresql
Redis
Redis is an in-memory data structure that can be used for different purposes. Currently it is used as cache for request rate limiting and as a message broker for running asynchronous tasks with Celery, a distributed task queue. Any up to date version >=6 should work, which can be installed like this:
sudo apt install redis-server
Node.js
Node.js and npm are used for managing the frontend dependencies and building/compiling the asset bundles. The current LTS version can be installed using:
sudo apt install ca-certificates curl gnupg
sudo install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg
echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_22.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list
sudo apt update && sudo apt install nodejs
For more information, see also assets in the project overview.
Elasticsearch
Note
As Elasticsearch requires quite a few system resources, especially memory, installation is optional for development. If you opt not to install it at the moment, this whole section may be skipped completely. Note that in this case, the full-text resource search functionality will not work.
If Elasticsearch is required, especially when developing search-related functionality, it can also be installed later on using the instructions below this box. Remember to initialize the search indices after the installation using the Kadi CLI, as explained later in the initialization section.
Alternatively, even if installed now, Elasticsearch can always be stopped until needed again using:
sudo systemctl disable elasticsearch # Disable Elasticsearch starting automatically
sudo systemctl stop elasticsearch # Stop Elasticsearch
sudo systemctl start elasticsearch # When needed, start it again
Once Elasticsearch is running again, using the Kadi CLI may be required to reindex existing data if the search indices went out of sync with the database, which happens when creating or updating resources while Elasticsearch is not running:
kadi search reindex
Elasticsearch is the full-text search engine used in the application. Currently, only version 8 is supported, which can be installed like this:
sudo apt install wget apt-transport-https gnupg
wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo gpg --dearmor -o /usr/share/keyrings/elasticsearch-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/elasticsearch-keyring.gpg] https://artifacts.elastic.co/packages/8.x/apt stable main" | sudo tee /etc/apt/sources.list.d/elastic-8.x.list
sudo apt update && sudo apt install elasticsearch
Some settings have to be changed after installation by using the following command, which will overwrite the default configuration file. Among others, these settings configure Elasticsearch to use a single-node cluster and disable the basic security features included in free Elasticsearch installations, which are not needed in this simple setup.
echo -e "path.data: /var/lib/elasticsearch\npath.logs: /var/log/elasticsearch\ndiscovery.type: single-node\nxpack.security.enabled: false\nxpack.security.transport.ssl.enabled: false\nxpack.security.http.ssl.enabled: false" | sudo tee /etc/elasticsearch/elasticsearch.yml
To start Elasticsearch and also configure it to start automatically when the system boots, the following commands can be used:
sudo systemctl enable elasticsearch.service
sudo systemctl start elasticsearch.service
Installing Kadi4Mat
To create and activate a new virtual environment for the application, the following commands can be used:
virtualenv -p python3 ${HOME}/.venvs/kadi
source ${HOME}/.venvs/kadi/bin/activate
Note
If not using the default Python version installed via APT, virtual environments need
to be created in a slightly different way, since the virtualenv version installed
via APT may not work properly with the newly installed Python version. Instead, the
built-in venv
module can be used:
python3.x -m venv <my_venv> # Instead of "virtualenv -p python3 <my_venv>"
source <my_venv>/bin/activate
pip install wheel
pip install -U pip
Note that the wheel
package needs to be installed separately after activating
the virtual environment for the first time, while updating pip ensures that its most
recent version is used, both of which virtualenv normally would have done
automatically.
Tip
When using the virtualenvwrapper tool,
new environments created like explained in the note above can still be managed using
this tool after the initial creation, as long as the new virtual environment was
created in the directory specified by the WORKON_HOME
environment variable.
This will create and activate a new virtual environment named kadi using Python 3 as
interpreter. The environment is stored inside the .venvs
directory in the current
user’s home directory. This directory can of course be changed freely. For an easier way
to manage virtual environments, a tool like virtualenvwrapper can also be helpful. For all following steps,
the virtual environment is assumed to always be active (by sourcing the activate
script).
Afterwards, the application can be installed via pip using the previously checked out source code:
pip install -e .[dev]
This will install the application in editable mode (-e
), which simply creates a link
to the sources so all changes are reflected in the installed package immediately. By
specifying the [dev]
modifier, all development dependencies will be installed as
well.
At this point, it is also recommended to install the pre-commit hooks already by running:
pre-commit install
Configuration
PostgreSQL
To set up PostgreSQL, a user and a database belonging to that user have to be created.
When prompted for a password, it is recommended to use kadi
. This way, the
default development configuration of the application does not need to be changed later
on.
sudo -Hiu postgres createuser -P kadi
sudo -Hiu postgres createdb -O kadi -E utf-8 -T template0 kadi
Kadi4Mat
In order for the application (or more specifically, its command line interface, as
explained in the next section) to know which environment it should run in, the
KADI_ENV
environment variable needs to be set accordingly. It is highly recommended
to set this variable via a .env
file. This file has to be created first, and should
reside in the project’s root directory, i.e. ${HOME}/workspace/kadi/.env
, when
following along with the example directory structure. After creating it, the following
content needs to be added:
KADI_ENV=development
For the development
environment, all configuration values, e.g. regarding the
database, have usable default values set. These correspond to the configuration
described in this documentation, so no further changes should be necessary.
If necessary, customizing any configuration values is best done using the
KADI_CONFIG_FILE
environment variable. This variable needs to point to a valid
configuration file, which also needs to be created first, in which the desired values
can be specified. As before, the .env
file should be used, e.g. assuming the config
file resides in config/config.py
relative to the project’s example root directory,
the following content can be added:
KADI_CONFIG_FILE=${HOME}/workspace/kadi/config/config.py
Please see Configuration for more information about
the configuration file itself and its values. Also, check out the manual
production
installation for an example and kadi/config.py
to see all default configuration
values for the different environments, grouped via different classes.
Initializing the application
Before the application can be used, some initialization steps have to be done using the
Kadi command line interface (CLI). As the Kadi CLI
needs to run in the context of the application, it needs access to the .env
file, so
the correct environment is used. Therefore, commands should always be run inside the
directory that contains this file.
kadi assets dev # Install the frontend dependencies and build the assets
kadi db init # Initialize the database
kadi i18n compile # Compile the backend translations
If Elasticsearch was installed previously, the following command has to be run additionally to initialize the search indices:
kadi search init
Another useful command when setting up the application for the first time is the following one, which can be used to set up some sample users and resources:
kadi db sample-data
The command creates the following sample users:
Username |
Password |
System role |
Sysadmin |
Notes |
---|---|---|---|---|
user |
user |
Member |
Yes |
Main user for testing purposes. |
admin |
admin |
Admin |
No |
Admin user. Can manage all resources. |
member |
member |
Member |
No |
Regular user. Can create resources. |
guest |
guest |
Guest |
No |
Guest user. Read only access. |
Running the application
Flask includes a lightweight development server, which can be run using the Kadi CLI:
kadi run
Afterwards, the application should be reachable locally at http://localhost:5000.
To be able to run asynchronous background tasks with Celery (needed for example when processing chunked file uploads, purging records or sending emails), the following command needs to be run in a separate terminal:
kadi celery worker -B --loglevel=INFO
This will start the normal Celery worker as well as Celery Beat in a single process for convenience when developing. The latter is used for periodic tasks (e.g. for removing expired, deleted resources).