... is the default). Located under local.oplog.rs. Holds every CRUD operation (Insert/Update/Delete/Commands). MongoDB R
Triggers in MongoDB
Antonios Giannopoulos and Jason Terpko DBA’s @ Rackspace/ObjectRocket linkedin.com/in/antonis/ | linkedin.com/in/jterpko/ 1
Introduction
Antonios Giannopoulos
Jason Terpko
www.objectrocket.com
2
Overview • • • • • •
What is a Trigger Why is Useful Application Triggers Oplog Tailing 3.6+ Streams Use Cases
www.objectrocket.com
3
What is a trigger? A trigger is a database object that is associated with a table. It will be activated when a defined action is executed for the table. Actions usually are: - INSERT - UPDATE - DELETE It can be invoked before or after the event.
www.objectrocket.com
4
MySQL trigger definition
www.objectrocket.com
5
Trigger example (BEFORE)
*For Deposits an AFTER event trigger is preferred J
www.objectrocket.com
6
Trigger example (BEFORE)
www.objectrocket.com
7
Trigger example (AFTER)
www.objectrocket.com
8
Trigger example (AFTER)
www.objectrocket.com
9
Foreign Keys Foreign Keys may also considered as triggers, but less flexible
www.objectrocket.com
10
MongoDB Application Triggers
• • •
Implementation Options Considerations Examples
www.objectrocket.com
11
Application Based Triggers Self-Implemented • • • •
Maintained internally Compatibility Flexibility Added Development Cycles
Open-Source Package • Publically maintained • Contributions and tested by a larger user base • Potentially restricted to specific versions, client and server
www.objectrocket.com
12
Self-Implemented (Before)
www.objectrocket.com
13
Self-Implemented (After)
www.objectrocket.com
14
Python (mongotriggers)
www.objectrocket.com
15
Node.js (mongo-oplog)
www.objectrocket.com
16
Oplog Tailing • • • • •
The oplog Tailable cursors Replica Set Sharded cluster Advantages/Disadvantages/Co nsiderations
www.objectrocket.com
17
The oplog Capped collection (FIFO) Fixed Size - oplogSizeMB (5% of free space is the default) Located under local.oplog.rs Holds every CRUD operation (Insert/Update/Delete/Commands) MongoDB Replication Idempotent by design
www.objectrocket.com
18
Anatomy of the local.oplog.rs
www.objectrocket.com
19
Anatomy of the local.oplog.rs
www.objectrocket.com
20
Anatomy of the local.oplog.rs ts: timestamp of the oplog entry t: election "term" h: unique hash v: version of the oplog op: Type of operation (insert/update/delete/commands) ns: Database & collection affected o: The new state of the document after performing the change o2: Contain the _id field of the affected document ui: Collection’s UUID wall: timestamp of the oplog entry fromMigrate : Sparse field, is true when the operation comes from balancing www.objectrocket.com
21
Tailable cursor Equivalent to the tail Unix command with the -f option Remains open after the client exhausts the results in the initial cursor Ideal for capped collections (where indexes are not practical) MongoDB replication uses tailable cursors to read the oplog Initial scan is expensive – It scans the entire collection Available on the vast majority of drivers www.objectrocket.com
22
Trigger implementation
www.objectrocket.com
23
Trigger implementation
www.objectrocket.com
24
Using Replica Set Primary’s and Secondary's oplog are identical
A tailable cursor is enough
Trigger action must loop through the primary
www.objectrocket.com
25
Using Sharded Cluster Oplog is not visible thought the mongos A tailable cursor per shard Adjust to topology changes Must filter balancer events Security issues s1
s2
Trigger action must loop through the mongos www.objectrocket.com
26
Considerations Only AFTER INSERT/UPDATE/DELETE supported Rollback an operation Handle nodes rollbacks (Replica set state) Avoid replay the same operations Keep up with the replication pace (Dedicated members using replica set tags) www.objectrocket.com
27
Considerations Apply before commit/commit errors (w:majority or w:n, n>1) Preserve the order of operations – sharded clusters only Filter out migration events (fromMigrate) – sharded clusters only Filter out update events on Orphans – sharded clusters only
www.objectrocket.com
28
Change Streams
• • • • •
Topology Change Events Examples Sharded Clusters Considerations
www.objectrocket.com
29
What is a Change Stream? (2) (6)
(4) (5) (3)
XX
1) Real-time 2) Driver Supported
(1)
3) Resumable
4) Secure 5) Synchronized s1
s2
6) Flexible www.objectrocket.com
30
Event Document (Insert)
www.objectrocket.com
31
Event Document (Update)
www.objectrocket.com
32
Event Document (Update - Full)
www.objectrocket.com
33
Event Document (Replace)
www.objectrocket.com
34
Event Document (Remove)
www.objectrocket.com
35
Event Document (Invalidate)
www.objectrocket.com
36
Python Stream – watch()
www.objectrocket.com
37
Flexible (Pipeline Stages) $match, $addFields
$match, $project
www.objectrocket.com
38
Sharded Clusters (2) (1)
1) Coordination 2) Rate of Change
(3)
(5)
(4)
3) Cold Shard(s) 4) Geographical Dist. 5) Orphans
s1
s2 www.objectrocket.com
39
Additional Considerations • Majority Read Concern / Engine • Replica Member Availability / Arbiters • Oplog Size • Collection Dropped or Renamed (Invalidation) • Large Documents and Maximum Document Size
www.objectrocket.com
40
Use Case •
Shard Key Analysis
www.objectrocket.com
41
Use Cases Auditing Event Scheduler Caching Selective Replication Disaster Recovery … www.objectrocket.com
42
Mongo to Mongo connector Reads the oplog on source and replays on destination source
dest
Tailable cursor
Use Cases • Selective replication • Multisource replication • Disaster recovery • Heterogeneous Replication www.objectrocket.com
43
Use Case: Shard Key Analysis Define a shard key is challenging: • Can break your application • Not an easy task to revert it (requires downtime) Shard Key limitations (two out of many): • Shard Key Value in a Document is Immutable. • NULL values are not allowed
www.objectrocket.com
44
Use case: Shard Key Analysis Query system.profile • Requires extra room as default is 1MiB • Adds overhead (extra writes) • Snapshot of operations Oplog or Streams • Already present • No extra overhead • Covers bigger duration www.objectrocket.com
45
Shard Key Analysis - Oplog
www.objectrocket.com
46
Shard Key Analysis - Streams
www.objectrocket.com
47
Questions?
www.objectrocket.com
48
Rate My Session
www.objectrocket.com
49
We’re Hiring! Looking to join a dynamic & innovative team? https://www.objectrocket.com/careers/ or email
[email protected] www.objectrocket.com
50
Thank you! Address: 401 Congress Ave Suite 1950 Austin, TX 78701 Support: 1-800-961-4454 Sales: 1-888-440-3242 www.objectrocket.com 51
![[PDF] Download MongoDB in Action: Covers MongoDB ... - Google Sites](https://m.moam.info/img/260x300/pdf-download-mongodb-in-action-covers-mongodb-goog_6477c719097c47a9708c177d.jpg)
![[PDF] Download MongoDB in Action: Covers MongoDB ... - Google Sites](https://m.moam.info/img/260x300/pdf-download-mongodb-in-action-covers-mongodb-goog_6477c547097c4786708c1454.jpg)
