Each Kinesis shard has 1 MiB of data per second or 1000 records in write capacity and 5 read transactions in read capacity. Attempting to exceed these limits results in throttling and your producers and consumers will not be able to carry out their operations.

Causes of throttling include:

  • Hot shards: throttles are caused by a few shards in the stream that receive more requests than the average shard. This is usually caused by uneven partition key distribution
  • Not enough capacity: throttles are caused by not having enough shards to service the number of requests your application requires

Hot Shards


Kinesis assigns your records to a shard by taking the MD5 of your partition key. To find a hot shard, you can log the MD5 sums of your partition keys and then check them against the range each shard returns through the describeStream API. If you find that a disproportionate number of partition keys fall into a range, that shard is a hot shard and should be split. Conversely, shards that have disproportionately less records can be merged to consolidate the capacity needed for those records.

 

Not Enough Capacity


When your application grows, you will send more data to your Kinesis stream and eventually need more capacity. In this case, you will have to reshard your Kinesis stream to add more shards. Most of the time, a separate application will monitor Kinesis and automatically reshard as necessary. In fact, AWS provides an open source application for this purpose.

 

Resources