Could you please use
explain(analyze, verbose, buffers, settings)
to get the whole picture? To me, it looks like the first query plan is not the best plan. And in this case, that is slower than the index scan on a larger set. It all looks like the wrong database configuration and a rather small test set.