Amazon Open Sources Python Library for AWS Glue
Written by Kay Ewbank   
Tuesday, 04 June 2019

Amazon has open-sourced a Python library known as Athena Glue Service Logs (AGSlogger) that makes it easier to parse log formats into AWS Glue for analysis and is intended for use with AWS service logs.

Organizations that use Amazon Simple Storage Service (S3) for storing logs often want to query the logs using Amazon Athena, a serverless query engine for data on S3. Amazon says that many customers use Athena to query logs for service and application troubleshooting, performance analysis, and security audits.

athena

The newly open-sourced Python library, Athena Glue Service Logs (AGSlogger), has predefined templates for parsing and optimizing a variety of popular log formats. AGSLogger lets you define schemas, manage partitions, and transform data as part of an extract, transform, load (ETL) job in AWS Glue. The idea is that developers will be able to use the library with AWS Glue ETL jobs to give you a common framework for processing log data.

The library is designed to do an initial conversion of AWS Service logs, then keep converting logs as they are delivered to S3. While it is possible to query the logs in place using Athena, for cost and performance reasons it can be better to convert the logs into partitioned Parquet files. The library has Glue Jobs for a number of types of service log that will create the source and destination tables, convert the source data to partitioned Parquet files, and maintain new partitions for the source and destination tables.

The library supports a number of log types:

  • Application Load Balancer
  • Classic Load Balancer
  • AWS CloudTrail
  • Amazon CloudFront
  • S3 Access
  • Amazon VPC Flow

Once converted from row-based log files to columnar-based Parquet, the data can be queried using Athena. Apache Parquet is an open-source column-oriented storage format originally developed for Apache Hadoop, but now more widely used .

 athena

More Information

Athena Glue Service Logs On GitHub

Related Articles

Athena Query Alterer Open Sourced

Databricks Delta Adds Faster Parquet Import

New AWS Services

AWS Improvements For Developers

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Banner


Running PostgreSQL Inside Your Browser With PGLite
18/03/2024

Thanks to WebAssembly we can now enjoy PostgreSQL inside the browser so that we can build reactive, realtime, local-first apps directly on Postgres. PGLite is about to make this even easier.



Couchbase's Coding Assistant Goes GA
11/03/2024

Capella iQ, the AI coding assistant for developers that makes interacting with Couchbase using natural language possible, has gone from private beta to being generally available.


More News

raspberry pi books

 

Comments




or email your comment to: comments@i-programmer.info

Last Updated ( Tuesday, 04 June 2019 )