Repository navigation
[h2-1.4.199 MVStore] Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' #2118
Description
Iterating through a map with a cursor sporadically throws a 'IllegalStateException: Chunk {number} not found' when keys are inserted concurrently. To provoke the issue having a short retention time makes the issue to appear more likely.
The database looks intact after the exception. I looks like the cursort references a page which has been re-used / deleted by another thread.
Exception:
java.lang.IllegalStateException: Chunk 13 not found [1.4.199/9]
at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:883)
at org.h2.mvstore.MVStore.getChunk(MVStore.java:1090)
at org.h2.mvstore.MVStore.readBufferForPage(MVStore.java:1025)
at org.h2.mvstore.MVStore.readPage(MVStore.java:2186)
at org.h2.mvstore.MVMap.readPage(MVMap.java:554)
at org.h2.mvstore.Page$NonLeaf.getChildPage(Page.java:1086)
at org.h2.mvstore.Cursor.hasNext(Cursor.java:53)
at info.gamlor.testing.Main.lambda$main$0(Main.java:29)
at info.gamlor.testing.Main.lambda$runInThread$2(Main.java:53)
at java.base/java.lang.Thread.run(Thread.java:835)
Reproduction app:
package info.gamlor.testing;
import org.h2.mvstore.MVMap;
import org.h2.mvstore.MVStore;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.Random;
import java.util.concurrent.ForkJoinPool;
public class Main {
public static void main(String[] args) throws Exception {
String DB_FILE_NAME = "iterate-keys.mvstore";
Path dbFile = Paths.get(DB_FILE_NAME);
if(Files.exists(dbFile))
Files.delete(dbFile);
try (MVStore store = MVStore.open(DB_FILE_NAME)) {
// Shorten retention time to make bug appear more likely
store.setRetentionTime(1000);
MVMap<String, String> testMap = store.openMap("test-map");
// Iterate over all keys over an over. This will fail sporadically if keys are inserted concurrently
Thread iteratorThread = runInThread("Iterate", () -> {
for (int i = 0; i < 20000; i++) {
long keyCount = 0;
for (String k : testMap.keySet()) {
keyCount++;
}
System.out.println("Run no " + i + " iterated over " + keyCount + " keys.");
}
});
// Insert new entries concurrently
Thread insertThread = runInThread("Insert", () -> {
long i = 0L;
while (iteratorThread.isAlive()) {
i++;
testMap.put("key:" + i, "data: " + i);
}
});
iteratorThread.join();
insertThread.join();
}
}
private static Thread runInThread(String name, Runnable block) {
Thread t = new Thread(() -> {
try {
block.run();
System.out.println(name + " ended");
} catch (Exception e) {
e.printStackTrace();
System.err.println("Program failed, exit");
System.exit(0);
}
}, name);
t.start();
return t;
}
}
Activity
[-]MVStore: Moving through keys with cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/-][+]MVStore: Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/+][-]MVStore: Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/-][+][h2 1.4.199 MVStore] Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/+][-][h2 1.4.199 MVStore] Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/-][+][h2-1.4.199 MVStore] Cursor sporadically fail with 'IllegalStateException: Chunk {number} not found' [/+]andreitokar commentedon Sep 18, 2019
This is expected. The reason for this failure is the fact that store is multi-versioned and key iterator iterate over the latest version at the time, when it was created. In a meantime, the other thread keeps inserting new entries and those new changes are periodically stored due to auto-commit. This creates new versions of the store. Older versions are dropped, otherwise file size will grow dramatically. Now, how many recent versions should be kept? Since store has no knowledge of which versions are still in use, it keeps N recent versions (apart from current). N is set by setVersionsToKeep(N) (default is 5). Also all versions younger than specified retention time are kept as "in-use". If that is not sufficient or right values are hard to determine, user can (should?) explicitly declare version usage. This can be done via de/registerVersionUsage(). This will maintain usage counter for each version and prevent version (and all subsequent versions as well) from being "garbage collected". If deregisterVersionUsage() is not called in a timely manner, store size will keep growing.
In your case version, upon which you are iterating, goes out range and is removed, hence this exception.
This can be fixed as:
After such change issue is not reproducible anymore, even if retention time and number of versions to keep are set to 0.
gamlerhart commentedon Sep 21, 2019
@andreitokar Thanks for the explanation. Did I miss obvious documentation about this behavior. Might be worth to add the Javadoc or general documentation otherwise.
andreitokar commentedon Sep 21, 2019
@gamlerhart, you are welcome. It is hard to miss something, which does not exists. 😸 This is a relatively new feature and documentation on MVStore is sparse and outdated. I will update it soon.
garneke commentedon Nov 14, 2019
@andreitokar I read your reply and potential fix. I have the same issue, however, I am not sure how I would implement your solution. The test that produced this error was iterating over all of the keys. I assume that you are using that to monitor the versions that are included and set the max to the maximum versions in use.
I on the other hand, I am simply referencing specific keys in my application. This particular exception happened when my MVStore was collecting a lot of records during a high volume run and I crashed on this exception. I cannot have this.
I recently updated versions to 1.4.199, I never experienced this with 1.4.195.
Is this exception a function of the newer version?
Should I revert back to the older version or is there a way that I can ignore version registration and get any version of the key that still exists?
I should mention that I am not using the versioning functionality as such. I am just using it for a fast persistent message store.
Your help would be hugely appreciated!
gamlerhart commentedon Nov 15, 2019
First I would check that the database file isn't corrupted and you have the same issue. A curropted database may create the same exception. Can you reproduce reliably.
One ugly solution would be to use registerVersionUsage()/deregisterVersionUsage() while getting the keys.
Another work around would be increase .setVersionsToKeep() or increase .getRetentionTime().
Long term, I think it's unexpected that a simple MVMap.get() crashes with a
Chunk {number} not found. It seems it should not drop any chuck it's reading during an atomic operation.andreitokar commentedon Nov 16, 2019
@garneke,
While most of what @gamlerhart just said here is true, I disagree with his last statement.
Even seemingly "atomic" operations like get() / put() may cause that exception, but only in extremely corner cases. I only able to reproduce it with both setRetentionTime(0) and setVersionsToKeep(0), and even then presence of a cache helps to avoid crash most of the time, and only with Builder.cacheSize(0) it fails consistently. Setting retention time even to 10ms prevents failures completely (at least for in my setting).
I wonder, what is your configuration and use case, so you experience this exception on a regular basis.
Of course, use of registerVersionUsage()/deregisterVersionUsage() eliminates the whole issue for regardless of all those settings.
Version 1.4.195 is more or less single-threaded (or rather fully synchronized), and starting from 1.4.197 reads are completely and writes are mostly non-blocking. Issue is more profound in 1.4.199 (and even more so in 1.4.200), because we do much more aggressive file space garbage collection.
garneke commentedon Nov 18, 2019
@andreitokar
I am setting my builder cacheSize to 512MB and the file to compress, but I have been leaving the rest of the settings as their default value.
My use case...
I am reading database records and caching them in the MVStore. The process requires that I do not lose any data so all records were previously cached to the physical disk as individual files. In order to reduce File IO we chose to store them in the MVStore allowing its built in persistence to protect the records.
We can push millions of records through the system in a day. The records will occasionally be read,written,renamed and eventually deleted as they propagate throughout the process.I can not guarantee that the builder cache size is greater than all the messages that are currently active ( not deleted ) in the process.The individual record names are unique so there should not be too many "versions" of a record at any one time. Although I do not use the mulit-version functionality so I do not totally understand how it is implemented.
In the last few weeks I have had this exception mulitple times on different MVStore files. Each time I believe the MVStore file has become corrupted.
I have returned to version 1.4.195 because I am not comfortable with the results I have had with 1.4.199
I appreciate your help and if you have a suggestion for me to move forward with a more recent version I would be interested but currently I can't afford to.
shollander commentedon Nov 27, 2019
@andreitokar, I too have a similar issue. We iterate through potentially large amounts
MVStoredata in our code. After upgrading to 1.4.200, we started noticing:Based on the information in this thread, it appears that we just need to start using registerVersionUsage()/deregisterVersionUsage(). However, I am not clear on when this is needed. It sounds like it might even be needed for every get()/put() (i.e. every access) to be completely foolproof.
Has the documentation been updated yet? I did not see it on the MVStore web page.
Can you clearly document this feature, when it is needed, and its implications?
garneke commentedon Nov 27, 2019
@shollander , @andreitokar
This was my concern - I can't image that would be viable to scan all entries for every get()/put() in order to update the version usage.
Perhaps clarification on the use of these functions, as you said, would be useful.
I second that question.
andreitokar commentedon Nov 28, 2019
Yes, this is true. But it does not mean that every call has to be wrapped individually. You can do it around some group of calls / loop, as long as you are ok with the fact that pinned version (and all subsequent ones) will not be garbage collected until it released, so file may temporarily grow larger.
We can wrap every call internally, within get/put etc., but that might negatively effect performance.
It might be easier than you think. You can write your own wrapper and/or use good IDE to do this in semi-automated fashion.
With right design you should not have too many scattered calls. H2 database, for example has only three registerVersion() and four deregisterVersion() call sites.
As far as documentation goes - guilty as charged, it has to be updated.
yuriy-ln commentedon May 12, 2020
I'm also stumbled upon that error after upgrading to 1.4.199 version. It happens sporadically.
Caused by: java.lang.IllegalStateException: Chunk 1830156 not found [1.4.199/9]The case is following.... we have table with some 'kind of index data' which has SQL index/constraint on it.
Periodically we are doing 'bulk data insert' on that table by adding new data in one thread. We have SQL connection without 'autocommit' during those operations and commit is done periodically (and inside bulk insert every 100 records), and after finished step.
By performance reasons we periodically do insert by steps below:
Looks like that code works fine, but fails sometimes and not easy reproducible.
DB file does not look as corrupted, because other parts works further, the db works after that error. But failed 'bulk insert' process makes data inconsistent from 'business view/reason'.
Is there some workaround exist?