Use TFileMerger to speed up podio-merge-files

[jmcarcell]

Currently podio-merge-files reads every Frame and writes it back. This is very slow (between 10x and 100x) compared to using hadd; I would say unusable. For TTrees and RNTuples, our files can be merged with hadd, producing readable files with the only caveat that there are some categories that may be repeated a few times, like metadata. The new implementation is equivalent to what we had before, but much faster.

BEGINRELEASENOTES

• Use TFileMerger with ROOT files in podio-merge-files to speed it up and add code to make sure the metadata is set correctly. Keep the old behaviour for the other backends (SIO only).

ENDRELEASENOTES

@meleneemil

[tmadlener]

Looks good to me (minor comments below). I suppose we are not yet at a stage where we have less code if we use the root file merger (that hadd uses internally) directly instead of going through hadd and then "fixing things" afterwards?

[jmcarcell]

Looks good to me (minor comments below). I suppose we are not yet at a stage where we have less code if we use the root file merger (that hadd uses internally) directly instead of going through hadd and then "fixing things" afterwards?

In the end I tried both with two passes of hadd and using TFileMerger; for merging files during several minutes it takes the same time. I'll leave the implementation with TFileMerger that doesn't have to run subprocesses. And it is quite close in time to running hadd directly, plus some constant overhead due to opening files, python and creating the metadata. I ran with thousands of files from EOS, RNTuples and it seems to work fine (and as quickly as hadd). hadd is also quite fast because it uses a low compression, which I have enabled as a default here but it is configurable. This should be strictly an improvement over what was there before.
Please @meleneemil, have a try again, it will appear in the nightlies tomorrow.