Use Dictionary to implement batch insertion of log data

background

Recently, I have made another requirement, which is to bury some events on the site. To put it bluntly, it is to record the user's access behavior. So how do you save these data? Click to save it? Obviously it is not suitable, and it must be batch storage to improve efficiency.

Peeping into the problem

First of all, what I think of is Dictionary. I believe everyone is familiar with the Dictionary class in C#. This is a Collection type, which can store data in the form of Key/Value (key value pairs). The biggest advantage of this class is that the time complexity of the search elements is close to O(1). It is often used in actual projects to make local caches of some data to improve overall efficiency. Dictionary is a non-thread-safe type, which can be added to memory first and saved into the database in batches.

Main code implementation

1. Define a Dictionary.

private readonly Dictionary<string, Tuple<ObjectInfo, object>> _storage = new Dictionary<string, Tuple<ObjectInfo, object>>();

2. Add elements, and they need to be thread-safely processed during operation. The easiest way is to lock.

public bool SaveObject<T>(string path, T value) where T : class {
      if ((path))
        throw new ArgumentNullException("path");

      lock (_lock) {
        _storage[path] = (new ObjectInfo {
          Created = ,
          Modified = ,
          Path = path
        }, (object)value);

        if (_storage.Count > MaxObjects)
          _storage.Remove(_storage.OrderByDescending(kvp => .).First().Key);
      }

      return true;
    }

3. Define a queue and consume logs regularly.

public DefaultEventQueue(ExceptionlessConfiguration config, IExceptionlessLog log, ISubmissionClient client, IObjectStorage objectStorage, IJsonSerializer serializer, TimeSpan? processQueueInterval, TimeSpan? queueStartDelay) {
      _log = log;
      _config = config;
      _client = client;
      _storage = objectStorage;
      _serializer = serializer;
      if ()
        _processQueueInterval = ;

      _queueTimer = new Timer(OnProcessQueue, null, queueStartDelay ?? (2), _processQueueInterval);
    }

When deleting here, lock operation is also required.

public bool DeleteObject(string path) {
      if ((path))
        throw new ArgumentNullException("path");

      lock (_lock) {
        if (!_storage.ContainsKey(path))
          return false;

        _storage.Remove(path);
      }

      return true;
    }

public IEnumerable<ObjectInfo> GetObjectList(string searchPattern = null, int? limit = null, DateTime? maxCreatedDate = null) {
      if (searchPattern == null)
        searchPattern = "*";
      if (!)
        maxCreatedDate = ;

      var regex = new Regex("^" + (searchPattern).Replace("\\*", ".*?") + "$");
      lock (_lock)
        return _storage.(k => (k)).Select(k => _storage[k].Item1).Where(f =>  <= maxCreatedDate).Take(limit ?? ).ToList();
    }

Summarize

1. Use Dictionary. Multi-threaded data to memory;

2. When a certain amount is reached, save the data in batches.

3. Use lock to ensure the safety of Dictionary operations.

This is the article about using Dictionary to implement batch insertion of log data. For more related contents of Dictionary to implement batch insertion of log data, please search for my previous articles or continue browsing the related articles below. I hope everyone will support me in the future!