DEV Community

Cover image for How I Discovered That the AWS Lambda Runtime Might Be Written in Go
Maxime David for AWS Community Builders

Posted on • Originally published at maxday.dev

How I Discovered That the AWS Lambda Runtime Might Be Written in Go

Have you ever wondered how your Lambda function code is triggered?

Each runtime has to conform to the AWS Lambda runtime API.

In managed runtimes, you don't need to take care of that; that's the whole point of using a managed runtime! AWS takes care of that for you.

But how?

It's hard to say because it's not clearly documented. However, while I was conducting some unrelated experiments, I made a very cool discovery!
I think you'll find it interesting too!

If you're working with Go and care about performance, you might want to look at some important metrics, such as:

  • the number of allocations
  • the time taken for each init() functions

To do so, you need to setup a environment variable:

GODEBUG to inittrace=1.

And tada πŸŽ‰!

In addition to information about the binary I wanted to test, I'm now able to see traces from another binary, which seems to be the one from AWS that conforms to the runtime API.

Here is an example of the output πŸ‘€πŸ‘€πŸ‘€

init internal/bytealg @0.008 ms, 0 ms clock, 0 bytes, 0 allocs
...
init go.amzn.com/lambda/fatalerror @3.7 ms, 0.031 ms clock, 336 bytes, 2 allocs
...
init go.amzn.com/lambda/interop @6.5 ms, 0.023 ms clock, 1128 bytes, 32 allocs
init go.amzn.com/lambda/telemetry @6.5 ms, 0.067 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/core @6.6 ms, 0.007 ms clock, 224 bytes, 12 allocs
init go.amzn.com/lambda/rapi/rendering @6.7 ms, 0.086 ms clock, 16 bytes, 1 allocs
init go.amzn.com/lambda/rapi/handler @6.8 ms, 0 ms clock, 256 bytes, 2 allocs
init go.amzn.com/lambda/rapid @6.8 ms, 0.099 ms clock, 16 bytes, 1 allocs
...
INIT_START Runtime Version: nodejs:18.v9 Runtime Version ARN: arn:aws:lambda:us-east-1::runtime:7d5f06b69c951da8a48b926ce280a9daf2e8bb1a74fc4a2672580c787d608206
Enter fullscreen mode Exit fullscreen mode

Here, we can clearly see some explicit package names coming from AWS.

Another important thing to note is that those logs are output BEFORE the INIT_START log line, which provides further evidence that it might be how AWS Lambda is managing your code!

Voila! πŸŽ‰
What do you think?

You can find me on LinkedIn and Twitter!

Top comments (20)

Collapse
 
raddevus profile image
raddevus

Hey, that’s a very cool catch. Way to stay alert and think outside the box and figure things out. That’s a very cool mindset.

Thanks for writing this one up. Very interesting.
If you get a chance, would you mind taking a look at my latest article here on dev.to? Software Developer, Are You Just A Hammer?

I think the kind of mindset it took you to discover this about AWS is a similar mindset to what I’m talking about in my article.

Collapse
 
maxday profile image
Maxime David

Thanks @raddevus! Glad you liked it!

Collapse
 
mikeldeltio profile image
Mikel Del TΓ­o • Edited

Updated: Unfortunately the traces do not correspond to the runtime but to another library. I'm sorry.

Hi,

Something similar just happened to me with the AWS Lambda Runtime for Python 3.11, but in this case it seems to be written in Rust and uses pyo3 to embed Python into a Rust binary. Also setting RUST_BACKTRACE=1 I'm able to see internal Rust traces.

I guess it doesn't have much relevance, but just in case I have already notified AWS.

Updated: According to AWS response, this does not present a security concern, rather, it is expected behavior.

Collapse
 
maxday profile image
Maxime David

wow this is amazing! Thanks a lot for sharing!
however I cannot reproduce it, you're selecting python3.11 and deploy a hello world function in eu-central-1 + set the RUST_BACKTRACE env variable to 1 right?

Collapse
 
mikeldeltio profile image
Mikel Del TΓ­o • Edited

Updated: Unfortunately the traces do not correspond to the runtime but to another library. I'm sorry.

Not quite. This information was into logged into CloudWatch after upgrading a Lambda function runtime from Python 3.10 to Python 3.11, while the source code was compiled for Python 3.10.

In fact, my Lambda function code is not even executed. An error occurs before, I suppose due to an incompatibility between the versions. Subsequently I have configured RUST_BACKTRACE=full as an environment variable, and the full stack backtrace has been printed.

Collapse
 
megaproaktiv profile image
Gernot Glawe • Edited

That is a very remarkable finding. I had to test it right away. You're right.
So all Lambda functions rely on Rust as a base for the micro-VMS and on top of that, go as a Lambda starter.
I love that this shows optimal use cases for Rust and GO, IMHO

Collapse
 
maxday profile image
Maxime David

Thanks for your kind words!

Collapse
 
neeldev96 profile image
Neel

Hi Maxime,

First of all, it's a good and brief read

Just for clarity, the logs you attached above, are those from a Node JS based lambda function?

Collapse
 
maxday profile image
Maxime David

Yes!

Collapse
 
rasiksheth profile image
reliability-architect

If it is easy enough, can we prove that runtime for other languages is also Go? for python or Java?

Collapse
 
maxday profile image
Maxime David

I've just done the test with java17 go1.x and python3.11, all are printing those init info as well

Collapse
 
bgadrian profile image
Adrian B.G.

Congrats, that's a nice finding. But I hope you realized before making it public it also raises a security risk for AWS as by identifying what libraries are used attackers can identify to which vulnerabilities are exposed. Basically this gives access to a large part of source code.
IMO you should have report this to AWS first, get a bounty reward and then make it public.

Collapse
 
maxday profile image
Maxime David

Thanks, I'm not sure since the package name seems to align with what is already open source here: github.com/aws/aws-lambda-runtime-...

Collapse
 
manuelinfosec profile image
Manuel • Edited

Good read, but are you sure of these?

It's worthy to note that AWS Lambda uses Firecracker for provisioning and running secure sandboxes to execute functions. aws.amazon.com/about-aws/whats-new....

Firecracker, in turn, is an open source virtualization technology that is purpose-built for creating and managing secure, multi-tenant container and function-based services. firecracker-microvm.github.io/, built by AWS and written in Rust.

Collapse
 
maxday profile image
Maxime David

Firecraker is indeed the underlying VM used by Lambda Functions, but we still need code to conform to the Lambda Runtime API (docs.aws.amazon.com/lambda/latest/...)
I think we're talking about two different topics, WDYT?

Collapse
 
snoopdougiedougie profile image
Doug

I need to push them to use Rust.

Collapse
 
megaproaktiv profile image
Gernot Glawe

Why? The current approach seem to work quite well!

Collapse
 
maxday profile image
Maxime David

It may be possible to go even faster with a runtime api implementation written in Rust (smaller binary, faster to start)

Collapse
 
maxday profile image
Maxime David

<3 please do that!