IllegalArgumentException in OffHeapStore #3051

Activity

1. andreitokar

   added

   MVStore

   on Mar 5, 2021
2. andreitokar commented on Mar 5, 2021

   

   andreitokar

   on Mar 5, 2021

   Contributor

   More actions

   @manaldush

   I would help a lot, if you will post here a complete self-contained test case.
   My gut feeling is that your application uses MVMap as a drop-in replacement of the java Map, and it's missing MVStore.registerVersionUsage() / deregisterVersionUsage() calls.
3. manaldush commented on Mar 6, 2021

   

   manaldush

   on Mar 6, 2021

   Last edited by manaldush

   Author

   More actions

   @andreitokar
   Thank you, I am going to understand, can we use h2 as in-memory offheap map and compare its performance with another storages. We have few processing threads and we can use storage in two modes:

   1. As singleton shared over all threads
   2. Every single thread has its own thread local storage, and data is sharded over all storages.

   Now we are talking about second mode. In my test I've created 50 threads and in every thread I execute operation

   private void load(int characterId) {
       Character character = cache.get(characterId);
       if (character == null) {
           cache.put(characterId, generator.createNewCharacter());
       } else {
           generator.changeCharactersClan(character);
           cache.put(characterId, character);
       }
   }
   

   Get Character(object used as Entry in storages), if object was not found, create it, but if it was found, update it

   I've attached two files:

   H2NoTxCharacterCache.java is decorator over h2 storage, here you can find details of using h2 for test

   public class H2NoTxCharacterCache implements CharacterCache {
   private MVMap<Integer, String> cache;
   private MVStore store;
   private final Serializer serializer = new Serializer();

   public H2NoTxCharacterCache() {
       store = new MVStore.Builder()
               .cacheConcurrency(1)
               .fileStore(new OffHeapStore())
               .open();
       cache = store.openMap("data");
   }
   
   @Override
   public Character get(int id) {
       String value = cache.get(id);
       if (value == null) {
           return null;
       }
       return serializer.deserialize(value, Character.class);
   }
   
   @Override
   public void put(int id, Character character) {
       String value = serializer.serialize(character);
       cache.put(id, value);
   }
   
   @Override
   public void inTx(int key, Runnable runnable) {
       runnable.run();
   }
   
   @Override
   public void close() {
       store.close();
   }
   
   @Override
   public void dumpOffHeapMemory() {
   
   }
   

   }

   LoadGeneratingCachePerThreadBenchmark.java - test file, use JMH benchmark for load generation

   @State(Scope.Thread)
   @WarmUp(iterations = 30, time = 1, timeUnit = TimeUnit.SECONDS)
   @measurement(iterations = 50, time = 1, timeUnit = TimeUnit.SECONDS)
   @fork(value = 1, jvmArgsAppend = "-XX:MaxDirectMemorySize=10G")
   @threads(50)
   @BenchmarkMode({Mode.Throughput, Mode.AverageTime})
   @OutputTimeUnit(TimeUnit.MILLISECONDS)
   public class LoadGeneratingCachePerThreadBenchmark {

   private static AtomicInteger seedController = new AtomicInteger(1);
   @Param({"100000"})
   private volatile int size;
   private final CharacterGenerator generator = new CharacterGenerator(seedController.getAndIncrement());
   @Param({"h2"})
   private volatile String cacheType;
   private CharacterCache cache;
   
   @Setup
   public void prepare() {
      if (cacheType.equalsIgnoreCase("h2")) {
           cache = new H2NoTxCharacterCache();
       } else {
           throw new IllegalArgumentException();
       }
   }
   
   @TearDown
   public void release() {
       if (cache != null) {
           cache.close();
       }
   }
   
   @Benchmark
   public void run() {
       int characterId = generator.nextInt(size);
       cache.inTx(characterId, () -> {
           load(characterId);
       });
   }
   
   private void load(int characterId) {
       Character character = cache.get(characterId);
       if (character == null) {
           cache.put(characterId, generator.createNewCharacter());
       } else {
           generator.changeCharactersClan(character);
           cache.put(characterId, character);
       }
   }
   

   }

   Thanks for help
4. andreitokar commented on Mar 7, 2021

   

   andreitokar

   on Mar 7, 2021

   Contributor

   More actions

   It is still far from a self-contained test case. But my gut feeling was correct. You do need to wrap access to MVMap into MVStore.registerVersionUsage() / deregisterVersionUsage() calls. You can wrap individual get()/put() or whole group of operations, like iteration. It's like transaction demarcation, and it may affect performance.