ACKF logoACKF
Analytics

AWS Glue

View on GitHub

AWS Glue

AWS Glue is a managed, serverless extract-transform-load (ETL) and metadata catalog service for preparing and loading data for analytics. It provides crawlers and a central Data Catalog for schema discovery and metadata, and runs managed ETL jobs (including Apache Spark-based jobs) to transform and move data. It typically sits between data sources (S3, JDBC-accessible stores, streaming sources when configured) and analytics or data warehouse services that query processed data.

🗂 Resource Category

Analytics • Application Integration

🧠 Exam Memory Hook

Think: "Serverless ETL + central data catalog + Spark jobs = AWS Glue"

📖 Ownership

Classification: AWS Managed Service

AWS responsibilities: AWS operates and manages the underlying serverless service platform, control plane, compute infrastructure for managed ETL jobs, the Glue service APIs, and the hosted Data Catalog storage; AWS secures and patches the physical infrastructure, hypervisor, operating system and the managed runtime used to run Glue jobs and crawlers; AWS maintains service availability according to its service model and provides managed integrations with other AWS services.

Customer responsibilities: The customer configures ETL job code, job parameters, IAM permissions, network configuration (VPC endpoints, security groups), data classification and schemas in the Data Catalog, crawler settings, triggers/workflows, and monitors job execution and logs; the customer is responsible for the data stored in and accessed by Glue, access control, data encryption configuration, and validating transformations and query correctness.

Patching responsibilities: AWS patches the physical infrastructure, hypervisor, underlying operating systems, and the managed runtime/platform that executes Glue jobs and crawlers. The customer patches or updates any code they provide, packaged dependencies included with their jobs, and any software on compute resources they manage; if using customer-managed compute (for example, running ETL outside of Glue), the customer is responsible for patching guest operating systems, runtimes, libraries, and applications.

🏗 Typical Architecture

💡 Top 5 Features

  • Central Data Catalog and catalog tables for storing metadata and schema definitions.
  • Crawlers that inspect data stores to infer schemas and populate the Data Catalog.
  • Serverless ETL jobs that run managed Apache Spark-based transformations.
  • Glue Studio for visual job authoring, testing, and iterative development of ETL logic.
  • Triggers and workflows to schedule and orchestrate ETL jobs and dependent tasks.

✅ Top 5 Use Cases

  • Build batch ETL pipelines that transform raw files in Amazon S3 into columnar formats for analytics and load them into data warehouses.
  • Populate and maintain a central metadata catalog so ad-hoc query engines like Amazon Athena can discover schemas and partitions.
  • Author and run managed Spark-based transformations without managing the underlying cluster infrastructure.
  • Orchestrate sequences of data preparation tasks using triggers and workflows to enforce data processing order and dependencies.
  • Perform incremental data ingestion using job bookmarks and connectors to JDBC sources to synchronize transactional stores with analytic datasets.

🏗 Architecture Placement

AWS Glue typically sits as the metadata and ETL layer in a data platform, ingesting or reading raw data from sources such as Amazon S3, JDBC-accessible databases, or streaming sources when configured, and writing transformed datasets to targets like Amazon S3 or data warehouses. It provides the Data Catalog that other analytics services (for example Amazon Athena or Amazon Redshift Spectrum) use to discover and query datasets. Glue is a regional, account-scoped service and is invoked by scheduled triggers, event-based invocations, or on-demand job runs.

🎯 Commonly Used With

- Amazon S3
- Amazon Athena
- Amazon Redshift
- Amazon RDS
- Amazon CloudWatch

🌍 5 Real-World Examples

  • A financial analytics team uses Glue crawlers to discover nightly trade files in S3 and runs Glue ETL jobs to normalize schemas for downstream reporting.
  • A retail analytics platform applies Glue transformations to clickstream logs in S3 to produce parquet datasets for interactive queries with Athena.
  • A healthcare data pipeline uses Glue to extract and transform data from an RDS instance and catalog the resulting tables for analytics teams.
  • A media company orchestrates recurring Glue workflows to combine metadata and usage logs, then loads the results into a Redshift data warehouse for reporting.
  • A logistics provider uses Glue ETL jobs to join IoT telemetry and order records, storing processed datasets in S3 for downstream machine learning feature extraction.

🎓 AWS Exam Clues

  • Choose AWS Glue when a managed, serverless ETL service with a central Data Catalog is required for analytics workflows.
  • Consider AWS Glue if you need schema discovery (crawlers) and a catalog consumable by Athena and other analytics services.
  • Prefer Amazon EMR or self-managed Spark when you need long-running, fine-grained control of cluster configuration and custom Spark dependencies.
  • Assess network requirements: Glue jobs that access databases in a VPC require proper VPC configuration and endpoints.
  • Account for runtime billing and job orchestration when designing scheduled ETL pipelines and large-scale transformations.

📝 Quick Revision

AWS Glue is a serverless ETL service plus a central Data Catalog for schema discovery and managed Spark-based transformations. Use it to prepare data for analytics and querying; plan for IAM/data access controls, VPC connectivity for data sources, job runtime costs, and schema/partition management in the catalog.

🏷 Keywords

Data Catalog • Crawlers • ETL jobs • Glue Studio • Serverless Spark • Job bookmarks • Triggers • Workflows • Connectors • Schema inference • Partitioning • Catalog tables