Ingest fraud ip list from integraladscience into somquery redis
npm install ops-ias-fraud-ip-ingestIn order to avoid serving ads to fraudulant traffic we use data of a third party ad verification provider called _Integral Ad Science (IAS)_.
They are sending lists of IPs claimed as fraudulent by them on an hourly base (S3 upload). We will use that data to block the related traffic (or at least monitor the outcome in the first phase).
* IAS uploads a list of ~2.5Mio. IP's to a S3 bucket on hourly basis
* A StepFunction / Lambda ingests these IP's to the Somquery Redis Cache
* in bunches of 100,000 in order to avoid latency in Redis / Timeouts of Lambda
* concurrent ingest of a newer set before switching versions
* a separate Lambda for removing entries of older versions
* Somquery is checking the remote IP of every call that is to return any VAST
* the result in written into the debug output / Cerberus log data
* the log data is monitored in Sherlog in order to get a figure of the ratio that would really get blocked
* After every IAS upload a timestamp is generated (
* Ingest of one entry per IP with schema ip:
* After Ingest set ip:currentTimestamp to
* Another Lambda is constanty removing IP entries with timestamps oder than currentTimestamp
* Somquery has to perform 2 Redis calls:
* const ts = await client.get('ip:currentTimestamp);
* ``const isFraud = await client.get(ip:${ts}:${);``
Redis is a highly available mem cache used in all our Somquery EC2 instances. Operating with bigger chunks could potentially increase any risk of interfere with Redis' life performance or of a timeout of the Lambda function.
Basicly for the same reasons (performance / timeout) plus an additional security layer that not any older data caused by a Ingest Lambda crash would bloat up Redis' size without noticing.
* Bucket: arn:aws:s3:::adtech-ias-data-exchange
* S3 Event Rule: arn:aws:events:eu-central-1:377710842884:rule/ias-data-exchange-event-rule
* StepFunction: arn:aws:states:eu-central-1:377710842884:stateMachine:IasIngestFraudIpsStateMachine
* Ingest Lambda: arn:aws:lambda:eu-central-1:377710842884:function:ias-ingest-fraud-ips
* Cleanup Trigger: arn:aws:events:eu-central-1:377710842884:rule/ias-fraud-data-cleanup-hourly
* Cleanup Lambda: arn:aws:lambda:eu-central-1:377710842884:function:ias-cleanup-fraud-ips