Increase database format version

[katzyn]

There are various random incompatibilities in persisted databases between H2 2.1.214 and current H2 in both MVStore and database metadata layers.

It is very dangerous to open database created by H2 2.0.X or 2.1.X with H2 compiled from current sources and vice versa. A database format version is increased here to distinguish database files from H2 2.0/2.1 from database files for upcoming 2.2.220.

I also added existing 2.0 and 2.1 releases to Upgrade utility, maybe it will be useful for someone for migration to 2.2.

[rPraml]

Hello @katzyn,
this change could be a showstopper for some developers.
(My thoughts, which I have written down below, should not be taken as negative criticism, but rather to show the problems and possible solutions that have arisen as a result of the conversion.)

What do you recommend to automatically update existing files?

In our case we have a web application, that uses H2 to do some internal computations.

We will have the problem when upgrading, that existing files (on the customer's Tomcat application server) can no longer be opened. So if the customer changes the app.war, we need to do either:

1. Providing the customer with a complex upgrade description
2. Include an automatic upgrade task in our application code
3. Implement a SCRIPT export (e.g. export the DB on shutdown before installing the next application version)
4. Delete and recreate the database

I see that the upgrade utility is trying to download the artifact via Maven - there will be no mvn command available on the tomcat production server, also a firewall is preventing internet access. So option1 & 2 is out.

Fortunately, in our case, the H2 databases do not contain any business-critical data that cannot be reconstructed, so at least WE have the option 4 to delete the database.

Implementing option 3 would be technically possible, but will either require user interaction or we need to export the db on every shutdown.

Technically the best solution would be, if every H2 version can read the previous version and can do an in-place update on startup (yes, will increase startup time)

I also checked the upgrade utility. It does some classloader-magic and registers an old driver. This might be a problem in web-applications, where multiple threads might open H2 databases.
I have the idea, to provide the old H2 driver under a different namespace (e.g. jdbc:h2_214:...), so that an application can open both versions.

Roland

[grandinj]

You need to do a backup and restore of the database files.
How you do that is up to you, we (H2) don't provide anything other than the backup and restore commands.

In my company we include the "old" jar in a secondary location, then we launch H2 as an external process to do the dump, then we do the restore using the new jar, which is already in the classpath, then we proceed as normal.

[julesjacobsen]

Hi H2 team - firstly thanks for the amazing utility. We've been using it for about a decade now with zero problems and have been able to use it for storing and accessing over 550 million KV pairs in our largest MVStore. However, the changes from 1.4x to 2.0.x and now to 2.2.x are extremely frustrating to have to deal with. Do you have an idea on how often this database format change is going to occur? Is there going to be a link to the H2 version - so far a semantic versioning goes 2.2.220 might as well be 3.0.0 given it won't read data produced with earlier versions. In an ideal world the database file format would remain stable.

Anyway, my question is this: Is there a recommended way to upgrade an MVStore file directly using something like MVStoreTool or Upgrade? Would this approach be a disaster waiting to happen if used on production data - https://github.com/katzyn/h2database/blob/72f6e983789b0ff413ba110d218c18ded50cb7b8/h2/src/test/org/h2/test/store/TestMVStore.java#L868-L886 ? (I presume yes)

Thanks for your help,

Jules

[katzyn]

H2 never had long-term storage formats and almost never had automatic upgrades (excluding upgrade helper for migration from archaic format to PageStore). Dump and restore were always required, newer versions have a helper class (it requires Internet access or pre-downloaded jar with old version in local Maven repository) and there is also a third-party upgrade utility.

Semantic versioning scheme was introduced only with H2 2.0.202, in older versions big releases with significant and sometimes incompatible changes and minor bugfix releases were mixed together (for example, 1.4.197, 1.4.198, and 1.4.200 were big releases, but 1.4.199 was a bugfix release for 1.4.198). In modern versions situation is clear: all x.y.* releases are more or less compatible with each other and it is usually recommended to use a release with latest build number (2.0.206 instead of 2.0.202 and 2.0.204, 2.1.214 instead of 2.1.210 and 2.1.212) unless you hit some regression critical for your project. Newer releases may have more features, however. But if first or second number in version is different, this version is not expected to be fully compatible with older or newer series of releases.

Also all newer versions don't try to open database files in incompatible formats to avoid data loss or corruption. Outdated 1.*.* releases were able to open not fully compatible files from different versions and corrupt them, this lesson was learned and we don't allow these dangerous operations any more.

---

Situation with MVStore is more complicated. It mostly receives changes, improvements, and new features needed for H2 itself and it doesn't have any fully stable API for direct usage. It's documentation is also outdated, for example, you need to call registerVersionUsage() / deregisterVersionUsage() around long read operations since the version 1.4.198.

Unfortunately, there are no documented ways to upgrade a MVStore file. MVStoreTool can dump its content, but I think it cannot create a new store from this dump. MVStore needs some own data import and export format, for now only H2 has such format (SQL dump).

[julesjacobsen]

Thanks for the clarification. I guess we were lucky and never had an issue as we used later versions of H2 only to read earlier databases and we were able to use data created back to 2017.

In modern versions situation is clear: all x.y.* releases are more or less compatible with each other and it is usually recommended to use a release with latest build number (2.0.206 instead of 2.0.202 and 2.0.204, 2.1.214 instead of 2.1.210 and 2.1.212) unless you hit some regression critical for your project. Newer releases may have more features, however. But if first or second number in version is different, this version is not expected to be fully compatible with older or newer series of releases.

This would be valuable to document as it would help people to expect a change when this is the case rather than finding out the hard way. Apologies if this is documented already, I obviously missed it.

Yes, I tried the MVStoreTool dump command earlier and found there was no import command to mirror it, so I guess that's more for development debugging than migration use. For our purposes it'll be easier to manually export the data and then re-import it into a new MVStore instance each time there is a format change. Hopefully, the 2.2.x series lasts a good few years though.