Problem description
DR Restore job fail with with error code: DR32779
Cause
AWS Lambda function’s concurrency quota exceed.
Traceback
[2024-12-24 02:46:37,476] [ERROR] [140040906704448] Error <class 'botocore.errorfactory.TooManyRequestsException'>:An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.. Traceback -Traceback (most recent call last):
File "/code/src/phoenix_client_lib/boto3/lambdaclient.py", line 50, in invoke
File "/usr/local/pyenv/versions/3.9.1/lib/python3.9/site-packages/botocore/client.py", line 535, in _api_call
File "/usr/local/pyenv/versions/3.9.1/lib/python3.9/site-packages/botocore/client.py", line 980, in _make_api_call
botocore.errorfactory.TooManyRequestsException: An error occurred (TooManyRequestsException) when calling the Invoke operation (reached max retries: 4): Rate Exceeded.
Resolution
Verify Concurrent Executions for Lambda:
Login and Go to AWS > Service Quotas > AWS Services > Lambda > Concurrent Executions.
Make sure it should be 1000.
Verification
Once Quota is increased, Trigger a DR Restore job and it should complete successfully.