Currently working with DynamoDB and a DAX cluster associated with the database, I need to conduct some scans solely on the underlying database at this moment. Due to the private VPC setup for DAX, I can't access it right now, which is a separate issue to address.
The goal is to execute a scan to retrieve all records from a table containing approximately 4k rows. However, due to DynamoDB's limitation of serving only 1M per page, I'm encountering a challenge where my loop seems to be caught in pagination and repeating from the beginning.
Below is a snippet of the code:
(Code snippet goes here)An output example is provided showing how the last primary key matches the first one, indicating a looping behavior in the scan process. Despite generating various keys using `ExclusiveStartKey` / `LastEvaluatedKey`, the loop fails to exit as expected - a puzzling outcome.
Considering other approaches like Query instead of Scan may offer a solution. Although performance is not a critical factor, making Scan work would be preferable.
Update
Exploring the Query option reveals an error message stating that KeyConditions or KeyConditionExpression parameter must be specified. While configuring an expression incorporating all records should fix this, the focus remains on refining the Scan operation for efficiency.
Update 2
A community insight emphasizes checking the AWS SDK version being used. Inconsistencies in dependency constraints prompt a suggestion to install the AWS JS SDK separately to ensure compatibility with the script.
Update 3
Suggestions lead to testing the `aws dynamodb scan` command in the console, exhibiting successful responses with LastEvaluatedKey included, reinforcing the iterative use of --exclusive-start-key when needed. The console approach yields consistent outcomes compared to the original script implementation, providing insights for potential adjustments.