AWS Lambda timeout when another (long) Lambda is invoked

Question

I am using Step Functions that link together multiple Lambdas. In one of these lambdas I invoke another lambda. Both lambdas have 15 min timeout. However, after some mins the primary lambda that invokes the secondary lambda times out (the secondary lambda happily continues its work).

Error

com.amazonaws.SdkClientException
Cause

{"errorMessage": "Unable to execute HTTP request: Read timed out",
  "errorType": "com.amazonaws.SdkClientException",
  "stackTrace": [
    cause": {
      "errorMessage": "Read timed out",
   "errorType": "java.net.SocketTimeoutException",
   "stackTrace": [
  "java.net.SocketInputStream.socketRead0(Native Method)",
  "java.net.SocketInputStream.socketRead(SocketInputStream.java:116)",
  "java.net.SocketInputStream.read(SocketInputStream.java:171)",
  "java.net.SocketInputStream.read(SocketInputStream.java:141)",
  "sun.security.ssl.InputRecord.readFully(InputRecord.java:465)",
  "sun.security.ssl.InputRecord.read(InputRecord.java:503)",
  "sun.security.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:983)",
  "sun.security.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:940)",
  "sun.security.ssl.AppInputStream.read(AppInputStream.java:105)",

This is the code, as you can see I try to raise the timeout using 4 different commands.

  // (2) Instantiate AWSLambdaClientBuilder to build the Lambda client
        AWSLambdaClientBuilder builder = 
 AWSLambdaClientBuilder.standard().withRegion(region);
        // (3) Build the client, which will ultimately invoke 
   the function
        AWSLambda client = builder.build();
        
        // (4) Create an InvokeRequest with required parameters
        InvokeRequest req = new 
   InvokeRequest().withFunctionName(random_arn).withPayload(jsonString); 
        
        // (5) Invoke the function and capture response
        int timeout = (15*60*1000);
        req.setSdkClientExecutionTimeout(timeout);
        req.setSdkRequestTimeout(timeout);
        req.withSdkClientExecutionTimeout(timeout);
        req.withSdkRequestTimeout(timeout);
        
        InvokeResult result = client.invoke(req);

Any idea how to tackle this timeout?

score 7 · Accepted Answer · 2018-12-22 23:07:07Z

7

There is more than one timeout you might be interested when working with AWS Lambda. The one you are referring to is the function execution timeout - the timeout people usually mean while talking about Lambda. However, there are also other timeouts you want to configure. Take a look at AWS CLI docs:

https://docs.aws.amazon.com/cli/latest/reference/index.html

Here you have --cli-read-timeout and --cli-connect-timeout. If I'd create a Lambda function with say 5 mins function execution timeout, and invoke it using AWS CLI using RequestResponse (synchronous) invocation type, if it happens it takes more than 1 min to execute (default value for --cli-read-timeout), AWS Lambda will fail this execution and trigger automatic re-try, up to a max number of retries. From what you are describing, I believe you are facing that kind of trouble. Set all the timeouts and try again.

A few more links you may find helpful:

AWS Lambda using Java - setting the configuration connection/socket timeouts

https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/section-client-configuration.html

https://docs.aws.amazon.com/AWSJavaSDK/latest/javadoc/com/amazonaws/ClientConfiguration.html#setSocketTimeout-int-

edited Dec 22, 2018 at 23:07

answered Dec 22, 2018 at 22:33

user4925383

Sign up to request clarification or add additional context in comments.

1 Comment

helloworld Over a year ago

Yes, you are right. After some googling I could find the appropriate code for this: ClientConfiguration clientConfiguration = new ClientConfiguration(); clientConfiguration.setSocketTimeout(0); AWSLambdaClientBuilder builder = AWSLambdaClientBuilder.standard() .withClientConfiguration(clientConfiguration) .withRegion(region); // (3) Build the client, which will ultimately invoke the function AWSLambda client = builder.build();

Dunedan · Accepted Answer · 2018-12-22 20:44:52Z

2

As you don't explicitly specify the invocation type, you invoke your second AWS Lambda function from your first AWS Lambda synchronously. So while your second AWS Lambda function is running, the first one will keep running as well, waiting for the response of the second function. That's why your first AWS Lambda function will time out if the sum of the run time of the first and the second AWS Lambda function is higher than the configured maximum timeout of the first AWS Lambda function (15 minutes in your case).

Invoking an AWS Lambda function synchronously from another AWS Lambda function is bad practice as you might run in timeout issues as you experience right now and spend money for running the first AWS Lambda function while the second AWS Lambda function is running.

What you could do is to invoke the second AWS Lambda function asynchronously, by setting the proper invocation type for your InvokeRequest:

InvokeRequest req = new InvokeRequest().withFunctionName(random_arn)
                                       .withPayload(jsonString)
                                       .withInvocationType(InvocationType.Event);

That of course only works if you don't depend on the output of the second AWS Lambda function to finish your first function.

The probably better option, as you're already using AWS Step functions, is to use a Step Functions state machine to orchestrate the invoking of the second AWS Lambda function, as described in the following tutorial: https://aws.amazon.com/de/getting-started/tutorials/create-a-serverless-workflow-step-functions-lambda/

Depending on how your AWS Lambda functions it might make sense to split the first AWS Lambda function, which is currently invoking the second AWS Lambda function into two parts running before and after the invoked Lambda function.

answered Dec 22, 2018 at 20:44

Dunedan

8,5416 gold badges48 silver badges55 bronze badges

4 Comments

helloworld Over a year ago

Hi, thanks for your detailed reply. I am aware of the downside of running these 2 lambdas synchronously. I should have mentioned that I am invoking the 2nd lambda in another region, something that cannot be accomplished within a step function pipeline which requires that all lambdas run in the same region (please correct me if I am wrong, I'd love to invoke the lambdas in different regions! from step function). And yes, the first lambda should be waiting patiently for the 2nd lambda to finish. Luckily, I can run the first lambda with fairly low amounts of memory

helloworld Over a year ago

while the second lambda performs the heavy work (and uses more memory). So I have some overhead, but it's not enormous. So I think I still need to wait for my 2nd lambda to finish and continue with the json result from that run. Do I miss something? I hope there is something I miss :-)

Dunedan Over a year ago

I still suggest splitting your invoking Lambda function into two parts. One part until you trigger the second Lambda function and another part for processing the result. As you can't use Step Functions for orchestration you could use something else, like SQS to persist the result. So you would invoke the function asynchronously and have the result-processing Lambda function being triggered by new messages in a SQS queue, which get written to it by the invoked Lambda function in the other region.

helloworld Over a year ago

Hi Dunedan, that's also a very interesting and elegant solution. I was thinking of posting something to s3 but using SQS is better. However, I have found a timeout setting that also works, which is much easier.

Collectives™ on Stack Overflow

AWS Lambda timeout when another (long) Lambda is invoked

2 Answers 2

1 Comment

4 Comments

Your Answer

Linked

Hot Network Questions

Collectives™ on Stack Overflow

2 Answers 2

1 Comment

4 Comments

Your Answer

Sign up or log in

Post as a guest

Linked

Related