README.md

ElasticGraph::IndexerLambda

Adapts elasticgraph-indexer to run in an AWS Lambda.

Dependency Diagram

graph LR;
    classDef targetGemStyle fill:#FADBD8,stroke:#EC7063,color:#000,stroke-width:2px;
    classDef otherEgGemStyle fill:#A9DFBF,stroke:#2ECC71,color:#000;
    classDef externalGemStyle fill:#E0EFFF,stroke:#70A1D7,color:#2980B9;
    elasticgraph-indexer_lambda["elasticgraph-indexer_lambda"];
    class elasticgraph-indexer_lambda targetGemStyle;
    elasticgraph-indexer["elasticgraph-indexer"];
    elasticgraph-indexer_lambda --> elasticgraph-indexer;
    class elasticgraph-indexer otherEgGemStyle;
    elasticgraph-json_ingestion["elasticgraph-json_ingestion"];
    elasticgraph-indexer_lambda --> elasticgraph-json_ingestion;
    class elasticgraph-json_ingestion otherEgGemStyle;
    elasticgraph-lambda_support["elasticgraph-lambda_support"];
    elasticgraph-indexer_lambda --> elasticgraph-lambda_support;
    class elasticgraph-lambda_support otherEgGemStyle;
    aws-sdk-s3["aws-sdk-s3"];
    elasticgraph-indexer_lambda --> aws-sdk-s3;
    class aws-sdk-s3 externalGemStyle;
    elasticgraph-warehouse_lambda["elasticgraph-warehouse_lambda"];
    elasticgraph-warehouse_lambda --> elasticgraph-indexer_lambda;
    class elasticgraph-warehouse_lambda otherEgGemStyle;
    click aws-sdk-s3 href "https://rubygems.org/gems/aws-sdk-s3" "Open on RubyGems.org" _blank;

Loading

SQS Message Payload Format

This gem is designed to run in an AWS lambda that consumes from an SQS queue. It uses ElasticGraph::JSONIngestion::Indexer, so messages in the SQS queue use JSON Lines format to encode indexing events.

JSON lines format contains individual JSON objects delimited by a newline control character (not the \n string sequence), such as:

{"op": "upsert", "__typename": "Widget", "id": "123", "version": 1, "record": {...} }
{"op": "upsert", "__typename": "Widget", "id": "123", "version": 2, record: {...} }

When publishing into SQS, be sure to keep messages under the 256 KiB SQS message limit.