Azure Data Architecture Guide

User Manual: Pdf

Open the PDF directly: View PDF PDF.
Page Count: 62

Azure Data
Architecture Guide
Agenda
Subway Map
Types of Data
Real-Time Streaming Architecture
Lambda Architecture
Hot Path
•View Actionable Data Quickly
•Likely Looking at an Aggregate
•Requires Professional Presentation
•Better if Ad-Hoc Querying is Supported
Cold Path
•Long-term Storage
•Must be cheap to query and store
•May be processed multiple times
•Analysis
•Prediction
•Learning
Hot & Cold Path Analysis
What
happened?
Why did it
happen?
What will
happen?
How can we
make it
happen?
Difficulty
Value
Kappa Architecture
IoT Message Processing
Data Warehousing
Data Warehousing
Data Warehouse Pattern
Data Warehouse Technology Choices
•SMP (small/medium data)
•Azure SQL Database
•SQL Server in a virtual machine
•Azure SQL Database managed
instance
Data Warehouse Technology Choices
•MPP (big data)
•Azure Data Warehouse
•Apache Hive on HDInsight
•Interactive Query (Hive LLAP) on
HDInsight
SQL Data Warehouse
SQL Data Warehouse
Azure Storage
Data Warehousing
No-SQL
Document Data Stores
Key-Value Data Stores
Graph Data Stores
And Then There’s More…
•Columnar
•Object
•Time Series
•External Index
Processing JSON
No-SQL Technology Choices
•Azure Data Factory
•Azure Logic Apps
•Azure Functions
•App Service
•Azure Data Lake Analytics
•Azure HDInsight
No-SQL Technology Choices
•Spark SQL
•HBase
•Hive
•SQL Data Warehouse
•Azure Machine Learning Workbench
•SQL SSIS
Azure Cosmos DB
Key-Value
Column-family
Graph
Documents
SQL
Clickstream Analysis
Storing Relational and No-SQL Data
Processing JSON in Real-Time
Data Lake
Analytics
Storage
WebHDFS
YARN
Unstructured Semi-Structured Structured
U-SQL
HDInsight
Authoring Jobs App Integration
Core Hadoop
Consistent REST API’s
Breadth of Clients (Java, JS, .NET, etc)
Authoring frameworks and languages
End User Tooling (IDE’s, Analyst tools, Command lines)
Connectivity
Programmability
Security
Loosely coupled
Lightweight
Low cost to extend
Scenario oriented
Innovation flows upward
New compute models
Perf enhancements
Extend breadth & depth
Enable new scenarios
Integrate with current tool
chains
HDInsight & Data Lake
Azure Data Lake Store
WebHDFS-compatible REST API
Azure HDInsight
Hadoop WebHDFS Client
On-Demand Big Data Analytics
Natural Language Processing
Processing Free-Form Text
Natural Language Processing (NLP)
“Great to meet you! I
need to extend my
booking next week by one
day. Can you also book
me a car?”
Processing Free-Form Text using NLP
Language Understanding (LUIS)
Create your
own LU model
Train by providing
examples
Deploy to an HTTP
endpoint and
activate on any
device
Maintain model
with ease
Natural Language Processing Technology Choices
•Azure HDInsight
•with Spark and Spark MLlib
•Microsoft Cognitive Services
•LUIS
•Bing Search APIs
Speech-to-Text Translation
can can you
here me
Can you hear
me?
Automatic
Speech
Recognition
Machine
Translation
TrueText
Speech-to-Speech Translation
can can you
here me
Can you hear
me?
Automatic
Speech
Recognition
Machine
Translation
TrueText
Text to
Speech
Intelligent Applications
Advanced Analytics & Deep Learning
Data Pipeline
Extract, transform, and load
Extract, load, and transform
Extract, Transform, Load (ETL)
Semantic Modeling
Semantic Modeling
Online Analytical Processing Pattern (OLAP)
OLAP Technology Choices
•SQL Server with Columnstore indexes
•Azure Analysis Services
•SQL Server Analysis Services (SSAS)
Azure Analysis Services
Data
Modeling
SQL Database
SQL Data Warehouse
Data Lake
HDInsight/Spark
Other
Other
SQL Server/Oracle
Third-Party
Power BI Desktop
Excel
Power BI
Lifecycle
Management
Security
Business Logic
& Metrics
In-Memory
Cache
Azure Analysis Services
Data Mart
•Search
•Browse
•Filter
Discover
•Metadata
•Experts
•Context
Understand
•Data Assets
•Familiar Tools
•Existing
Processes
Consume
•Tag
•Document
•Publish
Contribute
Business Intelligence
Relational Data
Relational Data
Transactional Data
Online Transaction Processing (OLTP) Pattern
OLTP Technology Choices
•Azure SQL Database
•Azure SQL Database Managed Instance
•SQL Server in an Azure Virtual Machine
•Azure Database for MySQL
•Azure Database for PostgreSQL
Thank You!
E-mail: sidney@seesharprun.net
Twitter: @sidney_andrews

Navigation menu