In some cases performance issues may only become apparent with large data sets, it would be good to be able to scale benchmarks on demand so that CI pipelines are not held up by f.e. loading 1 million records into Redis?

In a similar way it would be good to have a way to have stateful setups, so you can insert 1 million records and not insert 1 million records again when you next runt he test.