Highly Scalable Fault Tolerant Scalable Systems Considertions: Dr Tanenbaum on Fault Tolerant Operating Systems. https://en.wikipedia.org/wiki/Andrew_S._Tanenbaum Murphys Law: https://en.wikipedia.org/wiki/Edward_A._Murphy,_Jr. https://en.wikipedia.org/wiki/Murphy%27s_law If there’s more than one possible outcome of a job or tasks and one of those outcome will result in disaster or an undesirable consequence , …
AWS VPC Primer
AWS VPC : VPC are tied a specific Region. You can’t have your VPC span across multiple Regions. Security Groups: Instant Level Security NACL : Subnet Level Access Control VPC-VPC: Throgh VPC Peering CIDR : /16 – First 16 Bits are for Network CIDR : …
AWS AURORA RDS – Loading Billion Rows
We do online assessment and reporting for K-12 Education Industry. ( http://emetric.net/ ). We had a need to load 3 through K-12 students historical test reports for entire state for past 10 + years , which ended up being close to billion rows for historical data.
Amazon reInvent 2015 Las Vegas
Highlights: Data Migration Tool – We use aurora and data loading is little more than walk in the park. Hope this solves our problem. QuickSight – Explore this product for our needs RedShift – See how it can cutdown our Data warehouse costs. I have never …
Importing and Exporting Data From a MySQL DB Instance
Importing and Exporting Data into Amazon RDS We had a business need to load ~ Billion rows of Data into MySQL or Aurora Database. These are the information I gathered on optimizing load process. Load Data Infile Link to MySQL Amazon Recommendations based on their …
Amazon RDS Information
Aurora RDS: AWS Released encryption for Aurora Instances. Encryption is not available as of now. Amazon RDS Encryption Links Encrypting Amazon RDS Resources You can encrypt your Amazon RDS instances and snapshots at rest by enabling the encryption option for your Amazon RDS DB instance. …
Random Errors
Putty Fatal Error: Network Error: Software caused connection to abort. PuTTY Fatal Error I worked with CentOS servers from Windows PCs, and I had the same problem with PuTTY. A session didn’t last more than 1-5 minutes. I tried to play with PuTTY settings (keepalives, …
MySQL Ramp Up
MYSQL: Add Column to Existing Table and Make it Auto Increment and Add Primary Key Constraint Link : Alter Table and Add Column ALTER TABLE StudentPortalData ADD StudentPortalDataID INT UNSIGNED NOT NULL AUTO_INCREMENT, ADD PRIMARY KEY (StudentPortalDataID); Query Plan and Query Profile for Executed Queries …
SQL Server Hardware Selection
SQL Server Hardware Selection , licensing cost and BenchMark Tools.
Apache Drill Setup and Usuage
Apache Drill Setup and Usuage: Install and Configure Apache Drill http://drill.apache.org/docs/installing-drill-on-windows/ sqlline> !connect “jdbc:drill:zk=local” Enter username for jdbc:drill:zk=local: admin Enter password for jdbc:drill:zk=local: ***** http://localhost:8047/ SELECT derivedtable.ItemID ,AVG( CAST (derivedtable.TimeSpent AS INTEGER)) AvgTimeSpent FROM ( SELECT rawresponsetime.Time10.`id` ItemID ,rawresponsetime.Time10.`value` `TimeSpent` FROM ( SELECT (flatten(`time`)) `Time10`, …