Each Kinesis shard has 1 MiB of data per second or 1000 records in write capacity and 5 read transactions in read capacity. Attempting to exceed these limits results in throttling and your producers and consumers will not be able to carry out their operations.
Causes of throttling include:
Streams have two dimensions: stream level and shard level. While stream-level metrics offer an overall view, shard-level metrics provide detailed insights specific to individual shards.
Relying solely on stream-level metrics may lead to inaccuracies, especially in scenarios where data distribution across shards is uneven. By focusing only on shard level metrics, Blue Matador can pinpoint specific issues within individual shards, such as capacity constraints or uneven data distribution.
This granular level of insight enables proactive identification and resolution of potential bottlenecks, ultimately optimizing the performance and efficiency of Kinesis Streams data processing workflows.
Kinesis assigns your records to a shard by taking the MD5 of your partition key. To find a hot shard, you can log the MD5 sums of your partition keys and then check them against the range each shard returns through the describeStream API. If you find that a disproportionate number of partition keys fall into a range, that shard is a hot shard and should be split. Conversely, shards that have disproportionately less records can be merged to consolidate the capacity needed for those records.
When your application grows, you will send more data to your Kinesis stream and eventually need more capacity. In this case, you will have to reshard your Kinesis stream to add more shards. Most of the time, a separate application will monitor Kinesis and automatically reshard as necessary. In fact, AWS provides an open source application for this purpose.