pandas - Reducing cold start in a python AWS Lambda Function - Stack Overflow

admin管理员组

文章数量:1024630

I have an AWS lambda function written in python that needs a way of manipulating data stored in tables. My solution is to use pandas to read the tables in as parquet files. While this works, the cold start of this lambda function goes from ~400ms to 2000ms as soon as I add a pandas layer (even without any computation). I am wondering if there's any options out there that will get my cold start time to less than 1000ms? The total computation time of this function is <100ms, so it's a shame to have it so inflated.

I have an AWS lambda function written in python that needs a way of manipulating data stored in tables. My solution is to use pandas to read the tables in as parquet files. While this works, the cold start of this lambda function goes from ~400ms to 2000ms as soon as I add a pandas layer (even without any computation). I am wondering if there's any options out there that will get my cold start time to less than 1000ms? The total computation time of this function is <100ms, so it's a shame to have it so inflated.

• python
• pandas
• amazon-web-services
• aws-lambda

Share Improve this question Follow edited Nov 18, 2024 at 22:12 John Rotenstein 271k2828 gold badges448448 silver badges532532 bronze badges asked Nov 18, 2024 at 16:15 Jesse McMullen-CrummeyJesse McMullen-Crummey 3,87566 gold badges1212 silver badges2323 bronze badges 3

• Do you know how that time breaks down in terms of time spent importing pandas vs time spent loading your parquet files? – Nick ODell Commented Nov 18, 2024 at 16:24
• 1 Presume you have researched the common options such as provisioned concurrency, minimal pandas packaging (if that's a thing), increase RAM size, profile execution with Lambda power tuning, use arm64, find an alternative to pandas. Pandas et al are not your friends when it comes to low-latency FaaS. – jarmod Commented Nov 18, 2024 at 16:51
• 1 Also, literally announced yesterday, AWS Lambda now supports SnapStart for Python runtimes. – jarmod Commented Nov 19, 2024 at 13:27

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Pandas is a powerful but heavy library. Since cold start time depends on the size of the deployment package (including the layer size), consider using a Lighter alternative, like:

Polar which is a much faster and lighter DataFrame library with similar functionality to Pandas

PyArrow a lightweight and optimized for working with Parquet files and Arrow tables.

If Pandas is necessary, Try to reduce the Lambda layer and exclude any unnecessary files like 'tests, documentations'

Also cold start is primarily caused because of the initialisation time, so you can use AWS Lambda provisioned concurrency, which will keep some instances of your function warm and ready to serve traffic without a cold start penalty, but you need to put in your consideration that this feature is paid, and not for free.

I have an AWS lambda function written in python that needs a way of manipulating data stored in tables. My solution is to use pandas to read the tables in as parquet files. While this works, the cold start of this lambda function goes from ~400ms to 2000ms as soon as I add a pandas layer (even without any computation). I am wondering if there's any options out there that will get my cold start time to less than 1000ms? The total computation time of this function is <100ms, so it's a shame to have it so inflated.

I have an AWS lambda function written in python that needs a way of manipulating data stored in tables. My solution is to use pandas to read the tables in as parquet files. While this works, the cold start of this lambda function goes from ~400ms to 2000ms as soon as I add a pandas layer (even without any computation). I am wondering if there's any options out there that will get my cold start time to less than 1000ms? The total computation time of this function is <100ms, so it's a shame to have it so inflated.

• python
• pandas
• amazon-web-services
• aws-lambda

Share Improve this question Follow edited Nov 18, 2024 at 22:12 John Rotenstein 271k2828 gold badges448448 silver badges532532 bronze badges asked Nov 18, 2024 at 16:15 Jesse McMullen-CrummeyJesse McMullen-Crummey 3,87566 gold badges1212 silver badges2323 bronze badges 3

• Do you know how that time breaks down in terms of time spent importing pandas vs time spent loading your parquet files? – Nick ODell Commented Nov 18, 2024 at 16:24
• 1 Presume you have researched the common options such as provisioned concurrency, minimal pandas packaging (if that's a thing), increase RAM size, profile execution with Lambda power tuning, use arm64, find an alternative to pandas. Pandas et al are not your friends when it comes to low-latency FaaS. – jarmod Commented Nov 18, 2024 at 16:51
• 1 Also, literally announced yesterday, AWS Lambda now supports SnapStart for Python runtimes. – jarmod Commented Nov 19, 2024 at 13:27

Add a comment  | 

1 Answer 1

Sorted by: Reset to default 1

Pandas is a powerful but heavy library. Since cold start time depends on the size of the deployment package (including the layer size), consider using a Lighter alternative, like:

Polar which is a much faster and lighter DataFrame library with similar functionality to Pandas

PyArrow a lightweight and optimized for working with Parquet files and Arrow tables.

If Pandas is necessary, Try to reduce the Lambda layer and exclude any unnecessary files like 'tests, documentations'

Also cold start is primarily caused because of the initialisation time, so you can use AWS Lambda provisioned concurrency, which will keep some instances of your function warm and ready to serve traffic without a cold start penalty, but you need to put in your consideration that this feature is paid, and not for free.

本文标签： pandasReducing cold start in a python AWS Lambda FunctionStack Overflow