Partitions can end up out of order

[dpgil]

Let's say we have a storage instance with partition size = 2, and we insert rows at timestamp 1 and timestamp 3.

We then have a full partition with minTimestamp=1, maxTimestamp=3. On the next insert, a new partition will be created. When a new partition is created, the first time a row is inserted the minimum timestamp is set (code ref), so let's say we now insert a row at timestamp 2.

We'll now have two partitions with data points out of order. This breaks the query functionality as well.

[dpgil]

One possible solution is by setting the min timestamp in the memory partition constructor, and letting the creator enforce the min timestamp to never be less than the max timestamp of the previous partition.

@nakabonne let me know if I'm missing anything with this. Also, do you think this is a real issue and is it worth working on?

[nakabonne]

@dpgil You're right. This is the issue that I've put off for a while. Basically most data points get timestamped on other programs, where timestamps are not always the same as the process tstorage runs on. Because of that, I presumed we don't determine the min timestamp until the first data point reached.

However, I was totally wrong. We can determine the next min timestamp by just incrementing to the previous max timestamp, a kind of like:

func (s *storage) ensureActiveHead() {
	head := s.partitionList.getHead()
	if head.active() {
		return
	}
	nextMin := head.maxTimestamp + 1

	// All partitions seems to be inactive so add a new partition to the list.
	p := newMemoryPartition(nextMin, s.wal, s.partitionDuration, s.timestampPrecision)