Do we need a proper Search engine?

[joschahenningsen]

With the growing factors contributing to searching through videos (Description, Slide transcriptions, Video Transcriptions, Chat messages,...) we might consider using a proper backend for it rather than a sql full-text search.

Software we could consider include

• Meilisearch
• Elasticsearch
• Bleve
• Zinc
• ...?

Some of these are written in go and could be integrated into our binary without the need of another service.

[joschahenningsen]

@CommanderStorm you might have an opinion/expertise on this as well :)

[CommanderStorm]

Given the scale of the project, I can recommend meilisearch.

Reasoning:

• Scale: "we" don't need something as scalable (and therefore difficult to set up) as elastic
• Configurability: meili is very configurable. We (NavigaTUM) have been very happy with the features
• Fast: meili is very fast (top "normal" query time is around 10ms, which includes some network deamons waking up. For "normal" queries avg is around 3ms) => search as you type is possible
• meili does have an Go integration: https://github.com/meilisearch/meilisearch-go you may choose to not use these integrations for reasons, but normally this is not nessesary.
• meili has a instantsearch integration for the frontend:https://docs.meilisearch.com/learn/getting_started/quick_start.html#front-end-integration which may be advantageous, but considering that "we" already have a search bar implemented this is probably not nessesary.

Reasoning not to:

• not written in go => not directly integratable into the framework you've build. This would have to be it's own microservice with the associated difficulties.
• no direct integrationinto your database => it would be recommendable to have a way to export all the data meili needs in one internal API route.
  This could help in Desaster recovery, as I think this is an easier route than maintaining hard state in two places.
  I don't know if doing this via a Cron job or putting in the extra work to mirror DB changes to meili is worth it.
  For NaviaTUM we have even made this stateless in a similar route (state is kept in git, not in a DB for us, because the problem is rigid).
• no direct support for Google Dorking except "" and +. Most noticeable: - is not directly supported and other dorks would have to be implemented in the wrapper.

[joschahenningsen]

Thanks for your insights! I just tried a simple meili deployment and it looks very straightforward and promising!