How To Upload and Download Files in AWS S3 with Python and Boto3
In this How To tutorial I demonstrate how to perform file storage management with AWS S3 using Python's boto3 AWS library.
Posts about AWS S3
By Watermarking Videos with AWS Elastic Transcoder S3 and Lambda
In this article I walk through the process of setting up an automated process utilizing AWS S3, Lambda, and Elastic Transcoder to embed a watermark in to videos initiated by uploading to an S3 bucket.
Creating PostgreSQL Hot Standby Log Shipped Replica using WAL-E
In this article I demonstrate how to setup a log shipped replica of a Postgres v11 master database creating a warm standby replica for High Availability and read scaling utilizing the WAL-E program. WAL-E is a Python based program designed to facilitate performing continuous archiving of PostgreSQL WAL files and base backups.
Streaming Logs to S3 with Kinesis Firehose in a Serverless Project
In this article I demonstrate how to setup a AWS Serverless Application Model (SAM) project for near realtime streaming of CloudWatch logs to S3 using Kinesis Data Firehose. To keep things interesting I'll be using a Python based demo application that exposes two REST APIs, one for scraping and saving quotes from the web to a DyanmoDB table and another for listing the saved quotes.
Introduction to Redshift using Pagila Sample Dataset Including ETL from Postgres using AWS Glue
In this article I give a practical introductory tutorial to using Amazon Redshift as an OLAP Data Warehouse solution for the popular Pagila Movie Rental dataset. I start with a basic overview of the unique architecture Redshift uses to accomplish its scalable and robust use case as an enterprise cloud data warehouse. Then armed with this basic knowledge of Redshift architecture I move on to give a practical example of designing a schema optimal for Redshift based off the Pagila sample dataset.
Managing S3 Data Store Partitions with AWS Glue Crawlers and Glue Partitions API using Boto3 SDK
In this article I dive into partitions for S3 data stores within the context of the AWS Glue Metadata Catalog covering how they can be recorded using Glue Crawlers as well as the the Glue API with the Boto3 SDK.
Serverless Conversions From GZip to Parquet Format with Python AWS Lambda and S3 Uploads
In this article I demonstrate using a Python based AWS Lambda SAM project with the AWS Data Wrangler Lambda Layer to perform data format translation from GZipped JSON files into Parquet upon an S3 upload event.
Exploring AWS VPC Endpoints by Examples with AWS CDK
In this article I present the concepts of AWS Virtual Private Cloud (VPC) Endpoints through simple examples of setting up and consuming AWS services made explicitly accessible through VPC endpoints.
S3 Content Distribution via CloudFront Signed Urls Provisioned with AWS CDK
In this article I demonstrate a Cloud Architecture pattern for distributing content stored in S3 but fronted by AWS CloudFront and accessed through signed URLs viable for a specified amount of time.
Navigation
Favorites
Tags
Refactoring
Spring Boot
Stream Processing
Kafka
Security
AWS CDK
AWS
Java
Redis
PostgreSQL
Databases
Spring Security
AWS-Lambda
DevOps
OAuth
AWS S3
sysadmin
psql
Data Engineering
Gradle
AWS Kinesis
Click CLI Framework
Python
Natural Language Processing
MachineLearning
BeautifulSoup
requests
Apache Flink
AWS EMR
Apache Hive
Linux
ksqlDB
AWS Serverless Application Model
REST
PySpark
Kubernetes
SciKit Learn
AWS Glue
Ubuntu
Redshift
Flask
CentOS
uWSGI
Nginx
AWS Elastic Transcoder
C++
OpenCV
Sorting
Animations
JavaFX
Collections
Java HTTP Client
ChoiceBox
ComboBox
Try-With-Resources
Jsoup
Web Scraping
FXML
JavaScript
NodeJS
Factory Method
OOP
Eclipse
Intro to Java
TextBlob
Build Systems
static assets
cloning
streams
Django Auth
Django
Python Social Auth
Django Guardian
VueJS