Some years ago, AWS introduced the SSM (Simple Systems Manager) Agent. It's an agent that can be started on EC2 instances and perform multiple utility functions. Over the years, the SSM agent was added to all the major cloud-enabled Linux distribution AMIs, including Ubuntu, Amazon Linux, and RHEL. And AWS even added an option to automatically add SSM connectivity to all EC2 instances via Default Host Management!

SSM Agent supports a wide range of functionality. It can inventory running processes, apply patches, run shell commands, establish terminal sessions to EC2 instances, and even set up port forwarding.

The most significant advantage of the SSM Agent is its complete independence from the VPC settings. It uses an EC2 service, ssmmessages, and as a result, it can work just fine even in a VPC that doesn't have connectivity with the public Internet.

I was primarily interested in using this to set up port forwarding and, avoid using bastion hosts for SSH or PostgreSQL access.

Unfortunately, AWS's existing tooling around the SSM protocol is very clunky and can't be easily used in a composable standalone library. So I spent some time doing just that. The result of this work is Gimlet.

Looking at how SSM is supposed to be used normally

The normal way to use the SSM port forwarding is by using the AWS CLI with an optional Session Manager Plugin.

For example, to set up port forwarding to the instance i-0e3a964d49f28a5b8 and port 22 we need to run:

aws ssm start-session --target i-0e3a964d49f28a5b8 --document-name AWS-StartPortForwardingSession --parameters '{"portNumber":["22"], "localPortNumber":["56789"]}'

Starting session with SessionId: admin-0b6458385d2cffd35
Port 56789 opened for sessionId admin-0b6458385d2cffd35.
Waiting for connections...

The AWS CLI itself doesn't actually do anything but initiate the session, and the work of port forwarding is handled by spawning a background daemon:

cyberax@CybArm:~$ ps aux | grep session-manager-plugin
cyberax          43380   0,0  0,0 408636096   1488 s003  S+   10:01     0:00.00 grep session-manager-plugin
cyberax          43163   0,0  0,0 35315540  14816 s001  S+   10:00     0:00.27 session-manager-plugin {"SessionId": "admin-0b6458385d2cffd35", "TokenValue": "AAEAA......417bvh4OL", "StreamUrl": "wss://ssmmessages.us-east-1.amazonaws.com/v1/data-channel/admin-0b6458385d2cffd35?role=publish_subscribe&cell-number=AAEAAf2PldzWvh3EDdw8q7A0JB+nBIqkCvU+htAEPX0+D2QYAAAAAGPPc/9A2Ugc4EBbN5Qt9rCZMB9iBuYX6zUdShZndrZ5tvWh3g==", "ResponseMetadata": {"RequestId": "208ec786-7805-4965-91d7-8e6f7e95a603", "HTTPStatusCode": 200, "HTTPHeaders": {"server": "Server", "date": "Tue, 24 Jan 2023 06:00:31 GMT", "content-type": "application/x-amz-json-1.1", "content-length": "947", "connection": "keep-alive", "x-amzn-requestid": "208ec786-7805-4965-91d7-8e6f7e95a603"}, "RetryAttempts": 0}} us-east-1 StartSession pers {"Target": "i-0e3a964d49f28a5b8", "DocumentName": "AWS-StartPortForwardingSession", "Parameters": {"portNumber": ["22"], "localPortNumber": ["56789"]}} https://ssm.us-east-1.amazonaws.com

Yup. The parameters, including connection tokens and request metadata, are passed through the command line. Sigh.

The daemon is also quite a bit... hacky. It's just not written well, and can't really be used as a composable library inside your own application.

“Reverse engineering” the SSM port forwarding protocol

It's clear that the default implementation of session-manager-plugin leaves a lot to be desired. So we should just re-implement it! AWS is known for its pretty good documentation, so it should be simple, right?

The SSM port forwarding API calls are very eloquently documented by Amazon as special operations used by AWS Systems Manager. Which is about the total extent of the available documentation.

Fortunately, we do have the source code for both the server side and the client side. So we just need to read it and untangle its twisted web.

I documented the results of my investigation in Gimlet's README file.

In the next post, I'm going to demonstrate how Gimlet can be used to build a simple SSH proxy to allow passwordless access to EC2 instances.

Discuss...