Have you ever wondered how your Lambda function code is triggered?
Each runtime has to conform to the AWS Lambda runtime API.
In managed runtimes, you don't need to take care of that; that's the whole point of using a managed runtime! AWS takes care of that for you.
But how?
It's hard to say because it's not clearly documented. However, while I was conducting some unrelated experiments, I made a very cool discovery!
I think you'll find it interesting too!
If you're working with Go and care about performance, you might want to look at some important metrics, such as:
- the number of allocations
- the time taken for each init() functions
To do so, you need to setup a environment variable:
GODEBUG to inittrace=1.
And tada 🎉!
In addition to information about the binary I wanted to test, I'm now able to see traces from another binary, which seems to be the one from AWS that conforms to the runtime API.
Here is an example of the output 👀👀👀
init internal/bytealg @0.008 ms, 0 ms clock, 0 bytes, 0 allocs
...
init go.amzn.com/lambda/fatalerror @3.7 ms, 0.031 ms clock, 336 bytes, 2 allocs
...
init go.amzn.com/lambda/interop @6.5 ms, 0.023 ms clock, 1128 bytes, 32 allocs
init go.amzn.com/lambda/telemetry @6.5 ms, 0.067 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/core @6.6 ms, 0.007 ms clock, 224 bytes, 12 allocs
init go.amzn.com/lambda/rapi/rendering @6.7 ms, 0.086 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/rapi/handler @6.8 ms, 0 ms clock, 256 bytes, 2 allocs
init go.amzn.com/lambda/rapid @6.8 ms, 0.099 ms clock, 16 bytes, 1 allocs
...
INIT_START Runtime Version: nodejs:18.v9 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:7d5f06b69c951da8a48b926ce280a9daf2e8bb1a74fc4a2672580c787d608206
Here, we can clearly see some explicit package names coming from AWS.
Another important thing to note is that those logs are output BEFORE the INIT_START log line, which provides further evidence that it might be how AWS Lambda is managing your code!
Voila! 🎉
What do you think?
Top comments (20)
Hey, that’s a very cool catch. Way to stay alert and think outside the box and figure things out. That’s a very cool mindset.
Thanks for writing this one up. Very interesting.
If you get a chance, would you mind taking a look at my latest article here on dev.to? Software Developer, Are You Just A Hammer?
I think the kind of mindset it took you to discover this about AWS is a similar mindset to what I’m talking about in my article.
Thanks @raddevus! Glad you liked it!
Updated: Unfortunately the traces do not correspond to the runtime but to another library. I'm sorry.
Hi,Something similar just happened to me with the AWS Lambda Runtime for Python 3.11, but in this case it seems to be written in Rust and uses pyo3 to embed Python into a Rust binary. Also settingRUST_BACKTRACE=1I'm able to see internal Rust traces.I guess it doesn't have much relevance, but just in case I have already notified AWS.Updated: According to AWS response, this does not present a security concern, rather, it is expected behavior.wow this is amazing! Thanks a lot for sharing!
however I cannot reproduce it, you're selecting python3.11 and deploy a hello world function in eu-central-1 + set the RUST_BACKTRACE env variable to 1 right?
Updated: Unfortunately the traces do not correspond to the runtime but to another library. I'm sorry.
Not quite. This information was into logged into CloudWatch after upgrading a Lambda function runtime from Python 3.10 to Python 3.11, while the source code was compiled for Python 3.10.In fact, my Lambda function code is not even executed. An error occurs before, I suppose due to an incompatibility between the versions. Subsequently I have configuredRUST_BACKTRACE=fullas an environment variable, and the full stack backtrace has been printed.That is a very remarkable finding. I had to test it right away. You're right.
So all Lambda functions rely on Rust as a base for the micro-VMS and on top of that, go as a Lambda starter.
I love that this shows optimal use cases for Rust and GO, IMHO
Thanks for your kind words!
Hi Maxime,
First of all, it's a good and brief read
Just for clarity, the logs you attached above, are those from a Node JS based lambda function?
Yes!
If it is easy enough, can we prove that runtime for other languages is also Go? for python or Java?
I've just done the test with java17 go1.x and python3.11, all are printing those init info as well
Congrats, that's a nice finding. But I hope you realized before making it public it also raises a security risk for AWS as by identifying what libraries are used attackers can identify to which vulnerabilities are exposed. Basically this gives access to a large part of source code.
IMO you should have report this to AWS first, get a bounty reward and then make it public.
Thanks, I'm not sure since the package name seems to align with what is already open source here: github.com/aws/aws-lambda-runtime-...
Good read, but are you sure of these?
It's worthy to note that AWS Lambda uses Firecracker for provisioning and running secure sandboxes to execute functions. aws.amazon.com/about-aws/whats-new....
Firecracker, in turn, is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services. firecracker-microvm.github.io/, built by AWS and written in Rust.
Firecraker is indeed the underlying VM used by Lambda Functions, but we still need code to conform to the Lambda Runtime API (docs.aws.amazon.com/lambda/latest/...)
I think we're talking about two different topics, WDYT?
I need to push them to use Rust.
Why? The current approach seem to work quite well!
It may be possible to go even faster with a runtime api implementation written in Rust (smaller binary, faster to start)
<3 please do that!