Securely Restrict Access to Elasticsearch Using Lambda and IAM Roles

Takahiro Iwasa

Apr 14, 2020

3 min read

Elasticsearch IAM Lambda

Managing access to an Amazon Elasticsearch domain can be challenging, especially if the domain is internet-facing. Placing the domain inside a VPC requires additional costs for NAT Gateways or NAT Instances. However, using an AWS Lambda function as a proxy offers a cost-effective and secure solution.

This post outlines the steps to restrict access to Elasticsearch only from a Lambda function configured with an IAM role.

Before starting, ensure the following are installed:

• AWS SAM CLI
• Python 3.x

Here is the structure for the SAM application:

/
|-- es-proxy-lambda/
|   |-- __init__.py
|   |-- lambda_function.py
|   `-- requirements.txt
|-- samconfig.toml
`-- template.yaml

The following AWS SAM template defines resources such as an Elasticsearch domain, a Lambda function, and associated IAM roles. Important sections to note:

1. The AccessPolicies section restricts access to the Elasticsearch domain. (lines 8-16)
2. The Lambda function has permissions to perform specific actions on the domain. (lines 64-70)

AWSTemplateFormatVersion: "2010-09-09"
Transform: AWS::Serverless-2016-10-31

Resources:
  Elasticsearch:
    Type: AWS::Elasticsearch::Domain
    Properties:
      AccessPolicies:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              AWS:
                - !GetAtt IamRole.Arn
            Action: es:*
            Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda/*
      DomainName: es-for-lambda
      EBSOptions:
        EBSEnabled: true
        VolumeSize: 10
        VolumeType: standard
      ElasticsearchClusterConfig:
        DedicatedMasterEnabled: false
        InstanceCount: 1
        InstanceType: t2.small.elasticsearch
      ElasticsearchVersion: 7.4

  Lambda:
    Type: AWS::Serverless::Function
    Properties:
      CodeUri: es-proxy-lambda/
      Environment:
        Variables:
          ES_DOMAIN: !GetAtt Elasticsearch.DomainEndpoint
      FunctionName: es_proxy_lambda
      Handler: lambda_function.lambda_handler
      Role: !GetAtt IamRole.Arn
      Runtime: python3.8

  LogGroup:
    Type: AWS::Logs::LogGroup
    Properties:
      LogGroupName: !Sub
        - /aws/lambda/${name}
        - {name: !Ref Lambda}
      RetentionInDays: 1

  IamRole:
    Type: AWS::IAM::Role
    Properties:
      AssumeRolePolicyDocument:
        Version: '2012-10-17'
        Statement:
          - Effect: Allow
            Principal:
              Service: lambda.amazonaws.com
            Action: sts:AssumeRole
      ManagedPolicyArns:
        - arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole
      Policies:
        - PolicyDocument:
            Version: '2012-10-17'
            Statement:
              - Effect: Allow
                Action:
                  - es:ESHttpHead
                  - es:DescribeElasticsearchDomain
                  - es:ESHttpGet
                  - es:DescribeElasticsearchDomainConfig
                Resource: !Sub arn:aws:es:${AWS::Region}:${AWS::AccountId}:domain/es-for-lambda
          PolicyName: policy
      RoleName: es-proxy-lambda-role

Add the following dependencies. Note that boto3 is pre-installed in the Lambda runtime environment:

certifi==2019.11.28
chardet==3.0.4
elasticsearch==7.5.1
idna==2.9
requests==2.23.0
requests-aws4auth==0.9
urllib3==1.25.8

The following script leverages requests_aws4auth to securely connect to the Elasticsearch domain:

import os
import boto3
from elasticsearch import Elasticsearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth

es_domain = os.environ.get('ES_DOMAIN')
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(
    credentials.access_key, credentials.secret_key, 'ap-northeast-1', 'es', session_token=credentials.token
)

es = Elasticsearch(
    hosts=[{'host': es_domain, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection
)


def lambda_handler(event, context):
    response = es.info()
    print(response)

Replace <YOUR_S3_BUCKET> with your actual value.

version = 0.1
[default]
[default.deploy]
[default.deploy.parameters]
stack_name = "es-proxy-lambda"
s3_bucket = "<YOUR_S3_BUCKET>"
s3_prefix = "es-proxy-lambda"
region = "ap-northeast-1"
capabilities = "CAPABILITY_IAM CAPABILITY_NAMED_IAM"

Use the following commands to build and deploy the application. Note that creating the Elasticsearch domain can take 10-20 minutes:

sam build
sam deploy

Invoke the Lambda function. Successful execution indicates that the function can access the Elasticsearch domain.

Test direct access to the Elasticsearch domain. Unauthorized access should result in an error message:

$ curl https://<elasticsearch-domain-endpoint>/
{"Message":"User: anonymous is not authorized to perform: es:ESHttpGet"}

Remove all resources using the following command:

sam delete --stack-name es-proxy-lambda

By leveraging an IAM role attached to an AWS Lambda function, you can securely restrict access to an Elasticsearch domain. While this approach avoids the cost of additional NAT infrastructure, consider placing your domain inside a VPC for enhanced security if feasible.

Happy Coding! 🚀