You can pipeline commands into Redis to avoid the round-trip times like this:

{ for ((i=0;i<10000000;i++)) ; do printf "set key$i $i\r\n"; done ; sleep 1; } | nc localhost 6379

That takes 80 seconds to set 10,000,000 keys.

Or, if you want to avoid creating all those processes for printf, generate the data in a single awk process:

awk 'BEGIN{for(i=0;i<10000000;i++){printf("set key%d %d\r\n",i,i)}}'; sleep 1; } | nc localhost 6379

That now takes 17 seconds to set 10,000,000 keys.

The auto-increment key allows a unique number to be generated when a new record is inserted into a table/redis.

There is other way using UUID.

But I think auto-increment is far better due to reason like it need four time more space, ordering cannot be done based on key,etc

I'm doing exactly the same thing. here is an simple example. if you have a better one, welcome to discuss :)

1. connect to redis

import redis
pool = redis.ConnectionPool(host=your_host, port=your_port)
r = redis.Redis(connection_pool=pool)

2.define a function to incr, use pipe

def my_incr(pipe):
    next_value = pipe.hlen('myhash')
    pipe.multi()
    pipe.hsetnx(
        name='myhash',
        key=newkey, value=next_value
    )

3.make the function become a transaction

pipe = r.pipeline()
newkey = 'key1'
r.transaction(my_incr, 'myhash')

In order to be more memory efficient, you can use HASH to store these key-value pairs. Redis has special encoding for small HASH. It can save you lots of memory.

In you case, you can shard your keys into many small HASHs, each HASH has less than hash-max-ziplist-entries entries. See the doc for details.

B.T.W, with the INCR command, you can use Redis to create auto-incremented numbers.

I would like to answer my own question.

If you have sorted key values, the most efficient way to bulk insert and then read them is using a B-Tree based database.

For instance, with MapDB I am able to insert it very quickly and it takes up less memory.