Probable memory leak

[nstwfdev]

Hello there.
Finded already existing issue #488 (comment), but it closed, reopen.

Started from 180mb. But after ~10-12h, docker stats said:
20d8cd58d036 linkding 0.01% 232MiB / 31.29GiB 0.72% 867kB / 2.16MB 0B / 0B 13

docker-compose.yml

  linkding:
    container_name: "${LD_CONTAINER_NAME:-linkding}"
    image: sissbruecker/linkding:latest
    ports:
      - "${LD_HOST_PORT:-9090}:9090"
    volumes:
      - "${LD_HOST_DATA_DIR:-./data}:/etc/linkding/data"
    env_file:
      - .env
    restart: unless-stopped
    depends_on:
      - postgres

.env

LD_CONTAINER_NAME=linkding
LD_HOST_PORT=9090
LD_HOST_DATA_DIR=./data

LD_CONTEXT_PATH=
LD_SUPERUSER_NAME=postgres
LD_SUPERUSER_PASSWORD=postgres
LD_DISABLE_BACKGROUND_TASKS=False
LD_DISABLE_URL_VALIDATION=True
LD_ENABLE_AUTH_PROXY=False
LD_AUTH_PROXY_USERNAME_HEADER=
LD_AUTH_PROXY_LOGOUT_URL=
LD_CSRF_TRUSTED_ORIGINS=

LD_DB_ENGINE=postgre
LD_DB_DATABASE=linkding
LD_DB_USER=postgres
LD_DB_PASSWORD=postgres
LD_DB_HOST=postgres
LD_DB_PORT=5432
LD_DB_OPTIONS=

[sissbruecker]

I've looked into this a bit, and an increase in memory usage, up to multiple tens of megabytes is unfortunately unavoidable at the moment. One cause for this is the scraper that loads website titles and descriptions when adding bookmarks. It uses the charset_normalizer library for detecting an appropriate encoding for the website, and the library creates extensive caches for what it needs to do. I could not find an alternative that would reserve less memory in my short investigation.

Another thing that I ran into when testing locally is that the process memory never decreases, even though objects seem to be dereferenced. Not sure if that is how Python is supposed to behave. Anyway, if someone is up for debugging this in more detail that would be appreciated.

[zygimantus]

Not sure if related, but similar problem for me with docker image latest version. I start a container on my laptop and everything just freezes.

[sissbruecker]

Sounds more like #453

[sissbruecker]

I've been doing some more testing related to memory usage lately and could not observe a memory leak. The app will take up some more memory at runtime, but it does not look like it keeps growing.

Reasons why linkding takes up more memory after running for some time:

• Caches being built by charset_normalizer as explained above
• HTML templates used for rendering different views are cached in memory, as those views are being accessed

Another reason why memory usage for Docker containers can grow is due to the Linux system caching accessed files. As long as the host system provides more memory, and the Docker container does not have an explicit memory limit, the system in the container will keep using that available memory from the host for caching. In general linkding does not access any huge files, but this can become noticable when creating or viewing HTML snapshots, as those can take up several megabytes.

For an installation of the base image, memory usage should hover around what is shown in the OP, so somewhere around 230 MB. For the plus image you should plan with around 1 GB of available memory, in order to be able to run Chromium when creating HTML snapshots.