Chunk ... no longer exists

I'm using Nitrite database to store a million documents and then perform different read operations on them (for benchmarking).
After storing the documents, while trying to read all documents via collection.find().iterator(), I get the following exception:

java.lang.IllegalStateException: Chunk 162 no longer exists [1.4.196/9]
	at org.h2.mvstore.DataUtils.newIllegalStateException(DataUtils.java:765)
	at org.h2.mvstore.MVStore.getChunkIfFound(MVStore.java:953)
	at org.h2.mvstore.MVStore.getChunk(MVStore.java:935)
	at org.h2.mvstore.MVStore.readPage(MVStore.java:1943)
	at org.h2.mvstore.MVMap.readPage(MVMap.java:741)
	at org.h2.mvstore.Page.getChildPage(Page.java:217)
	at org.h2.mvstore.Cursor.fetchNext(Cursor.java:150)
	at org.h2.mvstore.Cursor.next(Cursor.java:50)
	at org.dizitart.no2.internals.DocumentCursor$DocumentCursorIterator.nextMatch(DocumentCursor.java:115)
	at org.dizitart.no2.internals.DocumentIterator.next(DocumentIterator.java:54)
	at org.dizitart.no2.internals.DocumentIterator.next(DocumentIterator.java:32)
        ...

~~Then, the next smaller read operations return more values than they should.~~(Not related to this problem)
I create the database with the default options and three indexes, and it occurs with and without enabled compressing. The used version is 2.1.0.

Is a million documents too much for Nitrite? The same data and test cases work fine with pure MVStore. Inserting only ~340 entries works fine too.

Could you please post the complete code here for both nitrite and mvstore alone to reproduce this error?

I've tested it with another dataset with ~270k entries that worked fine as well.
The project is really big and many classes are involved, so I hope the adapter classes are enough for you:

NitriteDB:
Constructing DB:

Nitrite db = Nitrite.builder().compressed()
			.filePath(storageDir + File.separatorChar + DB_FILE).openOrCreate();
NitriteCollection collection = db.getCollection(COLLECTION_NAME);
this.collectionAccessor = new CollectionAccessor(db, collection);

CollectionAccessor.java: https://pastebin.com/N63681vg
IteratorWrapper.java: https://pastebin.com/vpeXUMcE

MVStore:
H2MVStoreMultiMap.java: https://pastebin.com/V8wsUXP2
IteratorWrapper is basically the same, without the document converting.

MVStore example is not giving this exception is because you are creating only one map there. But in nitrite, it creates 6 maps - 1 for collection, 3 for indices, 2 for meta information.

Can you please do me a favor and create 4 mvmaps in mvstore example and insert data in all of them simultaneously in the same fashion? I just need to confirm if it is the volume of data that is creating problem or not. I might take this case to mvstore developers, so it will be better if I give as much info as possible.

I forgot to mention that H2MVStoreMutliMap class is instantiated three times.
But why do you want me to create only 4 mvmaps? What are the other two for then, because indexing should be enabled there too.

However, instantiating my MVStore multimap both 4 and 6 times works without problems.

I just want to make sure, you are creating 4 mvmaps in the same store and inserting 1 million of records in each of them parallely right?

I asked for 4 mvmaps, as other 2 are meta maps, it contains only few records, so does not contribute to this problem at all.

I just want to make sure, you are creating 4 mvmaps in the same store and inserting 1 million of records in each of them parallely right?

Yes, exactly.

Thank you. Let me re-create the scenario and see what can be done about this.

I'm currently trying to reduce the code for reproducing the error as well, and it looks like it depends on an outer layer that I didn't include here. I'll keep you updated if there maybe is a mistake on my side.

Nevermind that, I just didn't get the exception sometimes randomly.
So here is the reduced code that definitely threw one for me:
https://pastebin.com/kk9P06sG

Thanks a lot for the time. I am looking into it.

fix for #41

Fix is ready in 2.1.1-SNAPSHOT. Please test and confirm

I'm using Nitrite via Maven, but it's not available there yet?

https://oss.sonatype.org/content/repositories/snapshots/org/dizitart/nitrite/2.1.1-SNAPSHOT/

You need to add snapshot repo https://oss.sonatype.org/content/repositories/snapshots in maven pom

Thanks. Yes, now it works.

fix(gh-1284): MVStore backend can grow uncontrollably (#1288)

Navigation Menu

Repository navigation

Chunk ... no longer exists #41

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Chunk ... no longer exists #41

Description

Activity

anidotnet commented on Dec 22, 2017

ptoews commented on Dec 22, 2017

anidotnet commented on Dec 22, 2017

ptoews commented on Dec 22, 2017

anidotnet commented on Dec 22, 2017

ptoews commented on Dec 22, 2017

anidotnet commented on Dec 22, 2017

ptoews commented on Dec 22, 2017

ptoews commented on Dec 22, 2017

anidotnet commented on Dec 22, 2017

anidotnet commented on Dec 28, 2017

ptoews commented on Dec 28, 2017

anidotnet commented on Dec 28, 2017

ptoews commented on Dec 28, 2017

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions