Providing quality software engineering content in the form of tutorials, applications, services, and commentary suited for developers.
In this article I demonstrate "How To" use Terraform to provision an AWS EMR Cluster along with establish Read Only S3 Bucket access for consuming data from another AWS Account.
In this article I review key characteristics and functionality of Apache Hive and how you can utilize Amazon Elastic MapReduce (EMR) to provision a Apache Hive Cluster for experimentation and big data processing and analytics.
In this How To article I demonstrate how to use the AWS CLI to create an Amazon Elastic Map Reduce (EMR) Cluster along with some common supplementary resources for experimentation and development on an EMR cluster.
Here I present an end-to-end example of a Serverless event driven architecture using Confluent Cloud for stream processing paired with AWS Lambda for event responsive logic using the Serverless Application Model (SAM) framework. Together this architecture will compose a system for fictitious financial stock quote email alerting.
In this How To article I demonstrate setting up a Docker Compose based implementation of the Community Components of the Confluent Platform complete with the kafka-connect-datagen plugin for Kafka Connect to generate test and/or developement data useful for working with Kafka.
In this article I present an example of how one can use Kafka and the Confluent ksqlDB stream processing database to process a simplified dataset of fake stock quotes. The ultimate goal of this excercise will be to user ksqlDB to inspect a stream of stock quotes for individual companies in 1 minute windows and identify when a window has introduced a new daily high or low stock price.
In this How To article I will show a simple example of how to use the explode function from the SparkSQL API to unravel multi-valued fields. I have found this to be a pretty common use case when doing data cleaning using PySpark, particularly when working with nested JSON documents in an Extract Transform and Load workflow.
In this How To article I demonstrate running a simple Flask Python REST API service on a local minikube Kubernetes cluster using the VirtualBox Driver.
In this article I demonstrate using a Python based AWS Lambda SAM project with the AWS Data Wrangler Lambda Layer to perform data format translation from GZipped JSON files into Parquet upon an S3 upload event.
When it comes to quantitative analysis on data in database tables standard SQL provides a set of aggregate functions like SUM(), MAX(), and MIN(). There are two main ways these functions get used in practice: (i) collapsing the table data down to represent the aggregate calculation result set or, (ii) presenting the aggregate calculation per row maintaining the granularity of the complete table. Windowing functions are used to accomplish this second option and will the focus of this article.
By