Project Description

In this release a technique is implemented typical of the database to access the records. A table of the references in memory is used for accessing the objects on memory mapped files (MMF).

 Introduction

Working with large structures of data as List, ArrayList and Dictonary (kept in memory for speed) I have encountered out of memory exceptions, especially on the systems to 32bit. Then I have used the structures that use the memory mapped files (MMF). Read about Memory Mapped Files, http://www.codeproject.com/KB/recipes/MemoryMappedGenericArray.aspx and  http://mmf.codeplex.com/.
These structures result to be notably slow above all on systems for the DataWarehouse!

Then the purpose of this project is to sensitively improve the access on the MMF.
In this first release I will apply to a typical technique of the database: the table of the indexes or the references.

Such table for speed (read/write) is maintained in memory and the repeated interrogations are avoided on the presence or less of the object on MMF.

The further improvement is given during the removal of the object, since this operation doesn't happen on the MMF but only on the table of the references.

 

What's in the project?

IntelliSearch.MemoryMapped - Thread safe generic data types to use in your code

  • GenericMemoryMappedArray<T> - with autogrowing capabilities with new methods: RemoveAt(index), Delete(value), Exist(index) and Contains(value)!
  • RemoveAt :


Serializer - Implements several options for serializing your object http://mmf.codeplex.com/ 

  • Factory method which benchmarks the different approaches and picks the fastest one for your data type
  • Unsafe pointer based serializer - extremely fast for size defined structures/classes
  • Marshal Serializing


Winterdom.IO.Filemap - Modified version http://mmf.codeplex.com/ 

  • Library for using memory mapped files
  • Removed dynamic paging for optimization
  • Removed unused code

 

Benchmark on Intel Atom @1.60Ghz, 2GB RAM, HD 4200rpm

Test Time
Insert (Ideal) 00:00:00.14
Insert (MMF) 00:00:00.56
Exist (Ideal) 00:00:00.015
Exist (MMF) 00:00:00.00
Fetch (Ideal) 00:00:00.015
Fetch (MMF) 00:00:00.031
RemoveAt (Ideal) 00:00:00.484
RemoveAt (MMF) 00:00:00.00

 

The results of the tests have been performed for the insert on 100000 objects and 1000 on the test exist and remove.
The insertion and fetch in MMF result slower in the case of List in memory.
You have an improvement of speed on the methods Exist and RemoveAt in the case of MMF.

The future

  • GenericMemoryMappedDictionary<TKey,TValue>
    • Implement more methods on the classes and improvement performances

     

     

  • Last edited Oct 29, 2010 at 9:16 AM by Isterop, version 6