Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

.Net: Add Sql Server Support for memory in .NET SDK #2339

Closed
wants to merge 2 commits into from

Conversation

kbeaugrand
Copy link
Contributor

@kbeaugrand kbeaugrand commented Aug 7, 2023

Motivation and Context

Fix #2312

As discussed in #2265, the objective is to add support of MSSQL Server (and Azure SQL) for vector database of the project.

Description

I created the SQL server support based on what is presented in the following article : https://learn.microsoft.com/en-us/samples/azure-samples/azure-sql-db-openai/azure-sql-db-openai/

The code structure corresponds to that already in place with PostgreSQL or SQLite. A DB client performs actions towards the database, while the MemoryStore uses this client to make the necessary calls.

The DB Client creates 2 basic tables:

  • SKMemoryCollections (list of collections)
  • SKMemories (contains all memory entries, by collection)

In addition, when requesting the creation of a new collection, the client creates a new table SKEmbedding_{collectionName} for indexing all vector data.
The clustered column store index for vector search is therefore present in the last table.

When deleting collections, the client deletes the corresponding embedding table and deletes the entry in the collections table.

Note that the embedding is store twice (1 in the Memories tables + in the SKEmbedding_* table). This is necessary since, we have to provide also get feature withEmbeddings. It's faster to read the embedding directly in the column of Memory instead of constructing it from the vectors table.

As far as unit and integration tests are concerned, I've reproduced the tests available for PostgreSQL and adapted them to SqlServer.

Contribution Checklist

Sorry, something went wrong.

@kbeaugrand kbeaugrand requested a review from a team as a code owner August 7, 2023 08:19
@shawncal shawncal added .NET Issue or Pull requests regarding .NET code kernel Issues or pull requests impacting the core kernel memory connector labels Aug 7, 2023
@shawncal shawncal changed the title Add Sql Server Support for memory .Net: Add Sql Server Support for memory Aug 7, 2023
@kbeaugrand kbeaugrand force-pushed the feature/sql-server branch 2 times, most recently from e062806 to b9e2bb4 Compare August 7, 2023 08:23
@kbeaugrand kbeaugrand changed the title .Net: Add Sql Server Support for memory .Net: Add Sql Server Support for memory in .NET SDK Aug 7, 2023
@kbeaugrand kbeaugrand force-pushed the feature/sql-server branch 4 times, most recently from a3589c8 to 0ffeeb2 Compare August 7, 2023 08:29
@nacharya1
Copy link
Contributor

@kbeaugrand FYI: We recently updated our contributing guidelines [here](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#adding-plugins-and-memory-connectors , stating we are not directly adding plugin and connectors directly into the SDK repository. We are encouraging the community to host their own plugins and connectors separately in their own repositories. At the time we are exploring how we can provide a way to discover plugins and connectors and should have an update soon. I am going to close this issue given our change in contribution guidelines.

@nacharya1 nacharya1 closed this Sep 7, 2023
@kbeaugrand kbeaugrand deleted the feature/sql-server branch September 8, 2023 06:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kernel Issues or pull requests impacting the core kernel memory connector .NET Issue or Pull requests regarding .NET code
Projects
Archived in project
Development

Successfully merging this pull request may close these issues.

.Net: Implement SQLServer MemoryStore
4 participants