Rocket U2 Databases and the MultiValue model
Dan McGrath, Product Manager Previously – Developer & Systems Architect
Today we will cover…
3
Today we will cover…
4
Rocket Software Rocket Software (www.rocketsoftware.com) is a global software development firm that builds enterprise products and delivers enterprise solutions in the following segments: Business Intelligence and Analytics Storage, Networks, and Compliance Application Development, Integration, and Modernization Database Servers and Tools.
Rocket is engaged in business and technology partnerships with IBM, EMC, Fujitsu, HP Enterprise Services, Avaya, Motorola, Epicor, and many others.
The company is headquartered in Boston, Massachusetts, USA. ©2012 Rocket Software, Inc. All Rights Reserved.
5
Rocket Worldwide Locations
©2012 Rocket Software, Inc. All Rights Reserved.
6
Rocket by the Numbers 22 years
1,000+ employees 600+ engineers 100+ products 15+ R&D labs 10+ OEM partners 1,000+ ISV partners 10,000+ customers ©2012 Rocket Software, Inc. All Rights Reserved.
7
Rocket Brands
©2012 Rocket Software, Inc. All Rights Reserved.
8
Who Uses Rocket U2? Rocket U2 is installed at 30,000 sites around the world. Users include:
40%
• US Higher Education
50%
• Top 400 North America energy product companies
55%
• Fortune 100 companies
80%
• Automobile dealerships in North America
80%
• Mexico’s emergency response systems
100%
• Airlines checking luggage in Australia/New Zealand
©2012 Rocket Software, Inc. All Rights Reserved.
9
Now, the real stuff…
UniData & UniVerse
11
U2 Database - Platforms
Windows
AIX
HP-UX
Solaris
RHEL
SuSE
12
UniData & UniVerse MultiValue (MV) DB Engine • Efficient disk I/O • Table joining not required – nested data • Flexible storage structure Relational DB Engine
• Map MV to Relational Tables • ODBC/JDBC access • Optional Data-type Enforcement • Federated Database System 13
UniData & UniVerse Built-in JSON & XML support Customizable security framework Customizable DB commands Automatic Data Encryption
Federated Database System Replication 14
Database Structure Instance
Accounts Files
Localhost
Production
Staging
VOC
Dev
[Files…]
15
Database Structure
VOC
Keywords Commands Files Scripts 16
File types
Hash Tables
OS Directories
Static Dynamic Locally Distributed
17
File Structures Settings • # Groups • Block size • Hash Algorithm
Hash table
Example file: 3 groups Header • # Groups • Block Size • Hash Algo
Group 0
Group 1
Group 2
Overflow
• IDs/Pointers • Data
• IDs/Pointers • Data
• IDs/Pointers • Data
• IDs/Pointers (2) • Data (1)
Block size
Block size
Block size
Limited by OS 18
File Structures Settings
Dynamic Hash table
• Hash Table’s settings • Split/Merge %
Example file: Initial 3 groups & first split
• • • •
Header
Group 0
Group 1
Group 2
# Groups Block Size Hash Algo Split/Merge
• IDs/Pointers • Data
• IDs/Pointers • Data
• IDs/Pointers • Data
Separate Overflow file • IDs/Pointers (2) • Data (1)
Group 3 • IDs/Pointers • Data
Linear extension; buddy system
19
File Structures Settings
Locally Distributed Hash table
• Hash Table’s settings • Distribution Method
Part 0
Part …
Part n
Header
Header
Header
Groups
Groups
Groups
Overflow
Overflow
Overflow
20
File Structures Directory
Inherits OS & File system limits
ID: OS filename
Data: OS file contents
21
Data Structures - MultiValue Records
•Key/Value pairs •1-n columns • Can be 0-bytes
Column types:
•Single value •List (ordered) •List of lists (ordered)
Features • Dynamic columns & lists • Indexable ‘Virtual’ Columns • ‘Virtual’ Lookups 22
Data Structures – ‘people’
Jon Skeet
a45fe2
Tony the Pony
Unicorns
goo.gl/C2z9f goo.gl/n44vS
goo.gl/0GPMb goo.gl/MimWd
Boat Programming
5bd4e6 Jeff Atwood
jQuery
goo.gl/iWs1u
Unicorns
goo.gl/MimWd
Estimates
goo.gl/PCrfW
23
Data Structures
Jon Skeet
a45fe2
Tony the Pony
Unicorns
goo.gl/C2z9f goo.gl/n44vS
goo.gl/0GPMb goo.gl/MimWd
Boat Programming
5bd4e6 Jeff Atwood
jQuery
goo.gl/iWs1u
Unicorns
goo.gl/MimWd
Estimates
goo.gl/PCrfW
24
Data Structures
Jon Skeet
a45fe2
Tony the Pony
Unicorns
goo.gl/C2z9f goo.gl/n44vS
goo.gl/0GPMb goo.gl/MimWd
Boat Programming
5bd4e6 Jeff Atwood
jQuery
goo.gl/iWs1u
Unicorns
goo.gl/MimWd
Estimates
goo.gl/PCrfW
25
Data Structures
Jon Skeet
a45fe2
Tony the Pony
Unicorns
goo.gl/C2z9f goo.gl/n44vS
goo.gl/0GPMb goo.gl/MimWd
Boat Programming
5bd4e6 Jeff Atwood
jQuery
goo.gl/iWs1u
Unicorns
goo.gl/MimWd
Estimates
goo.gl/PCrfW
26
Metadata - MultiValue
Terminology • @ID = Key • Dictionary ≈ Schema • Dictionary Item = Column Definition • Field/Attribute = Column • File = Table/Collection 27
Metadata - MultiValue Files
Sharing
Schema-less
• Logical file = VOC entry • Physical file = OS entry • Multiple logical per physical • Logical file / primary dictionary • View with other dictionaries
• Dictionaries optional • Not enforced; ‘views’ 28
Metadata - MultiValue Dictionary Item Types
• Direct (indexable) • Virtual - Local (indexable) • Virtual - Lookup (indexable-ish) • Associations 29
Metadata – Direct @ID Type: Direct (Data Descriptor) Location: 0 Column: Single @ID
Jon Skeet
a45fe2
Tony the Pony
Unicorns
goo.gl/C2z9f goo.gl/n44vS goo.gl/0GPMb goo.gl/MimWd
Boat Programming jQuery
goo.gl/iWs1u 30
Metadata – Direct Name
Topics
URLs
Type: Direct
Type: Direct
Type: Direct
Location: 1
Location: 2
Location: 3
Column: Single
Column: List
Column: List of Lists
@ID
Name
Topics Jon Skeet
a45fe2
Tony the Pony
Unicorns
URLs
goo.gl/C2z9f goo.gl/n44vS goo.gl/0GPMb goo.gl/MimWd
Boat Programming jQuery
goo.gl/iWs1u 31
Metadata – Associations Topics
URLs
Location: 2
Location: 3
Column: List
Column: List of Lists
Association: Interests
Association: Interests
@ID
Name
Topics Jon Skeet
a45fe2
Tony the Pony
Unicorns
Interests Type: Association
Items: Topics, URLs
URLs
goo.gl/C2z9f goo.gl/n44vS goo.gl/0GPMb goo.gl/MimWd
Boat Programming jQuery
goo.gl/iWs1u 32
Metadata – Virtual (Local) NumTopics
Soundex
Type: Virtual
Type: Virtual
Location: DCOUNT(Topics, @VM)
Location: SUBR(‘soundex’, Topics)
Column: Single
Column: List
@ID
Name
Topics Jon Skeet
a45fe2
Tony the Pony
Unicorns
URLs goo.gl/C2z9f goo.gl/0GPMb goo.gl/MimWd
Soundex
J523
goo.gl/n44vS
4
U526 B316
Boat Programming jQuery
NumTopics
goo.gl/iWs1u
j260
33
Metadata – Virtual (Lookup) CreditCard Type: Virtual Location: TRANS(‘creditCards’,@ID, 1, ‘X’)
Column: Single @ID
Name
Topics Jon Skeet
a45fe2
Tony the Pony Unicorns
URLs
CreditCard
goo.gl/C2z9f goo.gl/n44vS goo.gl/0GPMb goo.gl/MimWd
Boat Programming
5bd4e6 Jeff Atwood
jQuery
goo.gl/iWs1u
Unicorns
goo.gl/MimWd
Estimates
goo.gl/PCrfW
3787-344936-71000 34
Metadata – Sharing @ID
byteLen
Type: Direct
Type: Virtual
Location: 0
Location: LEN(@RECORD)
Column: Single
Column: Single
JSON Type: Virtual Location: SUBR(‘toJSON’, @RECORD, @FILENAME) Column: Single 35
Locking Read: Shared Group Lock Example file: 3 groups Header • # Groups • Block Size • Hash Algo
Group 0
Group 1
Group 2
Overflow
• IDs/Pointers • Data
• IDs/Pointers • Data
• IDs/Pointers • Data
• IDs/Pointers (2) • Data (1)
Write: Exclusive Group Lock
36
SQL Engine & Nested Tables SQL Engine embedded
• SQL at the command-line • Schema management tools
‘Virtual’ tables for SQL
• Explicitly by schema generation • On-the-fly using ‘UNNEST’ keyword
JDBC & ODBC access
• Able to run native commands too
Data-type enforcement
• Ignore completely • Log violations to disk • Raise errors and prevent action 37
Federated Database System
Store data in an external DB
Access as if it was local
Drivers for: • SQL Server • DB2 • Oracle
Driver API exposed • We use this API to write our drivers
38
Automatic Data Encryption File level encryption
Column level encryption
Record key encryption
Index encryption Multiple password protection
of ‘Master Key’ ‘Wallets’ Grouping / Temporary access 39
Replication
Publisher 1-n Subscribers Account (full environment) replication File replication
Organize into ‘groups’ Can Publish & Subscribe from same machine
©2012 Rocket Software, Inc. All Rights Reserved.
40
Replication
3 modes: Deferred – Sync every ‘x’ time Immediate – Commits once staged
Real-time – Commits once sync’d
©2012 Rocket Software, Inc. All Rights Reserved.
41
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
42
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
43
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
44
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
45
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
46
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
47
Replication Publishing System
Subscribing System
xadmin
uvrepmanager
uvrepmanager
Transaction Control Area (TCA)
uvpub
uvsub sublistener submain
Transaction Control Area (TCA)
uvpublistener
xadmin
Subscriber Packet File (SPF)
uvsh
Replication Buffer
Replication Buffer
uvrw
Publisher Files
Log Extension Files (LEF)
Log Extension Files (LEF)
Subscriber Files
48
Application Server
“We need SMART Models, THIN Controllers, and DUMB Views”
50
Application Server U2 Database • Model
View
Controller 51
Application Server U2 Database • Model
View
Controller 52
Application Server
U2 Database 24/7 Phantoms Callable Subroutines for Business Logic
Session Memory Customizable DB Commands & Keywords Customizable Security Framework Select result ‘snapshots’ ©2012 Rocket Software, Inc. All Rights Reserved.
53
Commands & Keywords
VOC SELECT PersonModel CreditCardModel stdSELECT ©2012 Rocket Software, Inc. All Rights Reserved.
54
Customizable DB Commands
VOC SELECT • V SELECT
PersonModel • C /byteCode/PersonModel
CreditCardModel • C /byteCode/CreditCardModel
stdSELECT • V SELECT
©2012 Rocket Software, Inc. All Rights Reserved.
55
Customizable DB Keywords
VOC SELECT • C /byteCode/CustSelect
PersonModel • C /byteCode/PersonModel
CreditCardModel • C /byteCode/CreditCardModel
stdSELECT • V SELECT
©2012 Rocket Software, Inc. All Rights Reserved.
56
Customizable Security Framework
VOC SELECT • V SELECT • Security SecControl
PersonModel • C /byteCode/PersonModel • Security SecControl
SecControl • C /byteCode/SecControl
©2012 Rocket Software, Inc. All Rights Reserved.
57
Customizable Security Framework
VOC SELECT • V SELECT • Security SecControl
EVAL • K EVAL • Security SecControl
SELECT people WITH name EVAL “TRANS(‘users’, @ID, 1, ‘X’)”
©2012 Rocket Software, Inc. All Rights Reserved.
58
Select result ‘snapshots’
Sharing within a session
©2012 Rocket Software, Inc. All Rights Reserved.
Sharing between sessions
Save to disk • Resume same processing after outage
59
Additional tooling
XAdmin
©2012 Rocket Software, Inc. All Rights Reserved.
61
Web Services
Expose U2
SOAP Web Services • Automatic WSDL generation
as a
Web Service
RESTful Web Services • JSON-based
62
Business Intelligence & Analytics
63
Business Intelligence & Analytics
64
Network/Systems Monitoring
©2012 Rocket Software, Inc. All Rights Reserved.
65
©2012 Rocket Software, Inc. All Rights Reserved.
66
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
67
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
68
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
69
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
70
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
71
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
72
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
73
AeroText: Unstructured Data
©2012 Rocket Software, Inc. All Rights Reserved.
74
Summary Rocket Software; enterprise solutions Rocket U2 DBs; Airlines to 911 Flexible data structure & metadata Plays nice with Relational DBs
©2012 Rocket Software, Inc. All Rights Reserved.
75
What’s next? www.rocketsoftware.com/u2 Free personal editions
Email:
[email protected]
Twitter: @itcmcgrath Tech Conferences: U2 University UK: 26-28 June, Australia: 14-16 August u2u.rocketsoftware.com ©2012 Rocket Software, Inc. All Rights Reserved.
76
Disclaimer THE INFORMATION CONTAINED IN THIS PRESENTATION IS PROVIDED FOR INFORMATIONAL PURPOSES ONLY. WHILE EFFORTS WERE MADE TO VERIFY THE COMPLETENESS AND ACCURACY OF THE INFORMATION CONTAINED IN THIS PRESENTATION, IT IS PROVIDED “AS IS”, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. IN ADDITION, THIS INFORMATION IS BASED ON ROCKET SOFTWARE’S CURRENT PRODUCT PLANS AND STRATEGY, WHICH ARE SUBJECT TO CHANGE BY ROCKET SOFTWARE WITHOUT NOTICE. ROCKET SOFTWARE SHALL NOT BE RESPONSIBLE FOR ANY DAMAGES ARISING OUT OF THE USE OF, OR OTHERWISE RELATED TO, THIS PRESENTATION OR ANY OTHER DOCUMENTATION. NOTHING CONTAINED IN THIS PRESENTATION IS INTENDED TO, OR SHALL HAVE THE EFFECT OF: • CREATING ANY WARRANTY OR REPRESENTATION FROM ROCKET SOFTWARE(OR ITS AFFILIATES OR ITS OR THEIR SUPPLIERS AND/OR LICENSORS); OR
• ALTERING THE TERMS AND CONDITIONS OF THE APPLICABLE LICENSE AGREEMENT GOVERNING THE USE OF ROCKET SOFTWARE.
©2012 Rocket Software, Inc. All Rights Reserved.
77
Trademarks and Acknowledgements The following are trademarks or registered trademarks of Rocket Software, Inc.: Dynamic Connect, SystemBuilder, U2, U2 Web Development Environment, UniData, UniVerse, and wIntegrate. IBM, the IBM logo, AIX, and DB2 are trademarks of IBM in the United States and other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. Microsoft, SQL Server, Windows, and Excel are trademarks of the Microsoft group of companies. UNIX is a registered trademark of The Open Group. Other company, product, and service names mentioned herein may be trademarks or service marks of others.
©2012 Rocket Software, Inc. All Rights Reserved.
78
