Visual Programming: Cooking the Spaghetti - Gotocon.com

Visual Programming: Cooking the Spaghetti GOTO Chicago '13

Rick Lane [email protected] @R1ck_L4n3

@trading_tech

@GOTOChgo

First things first ● Visual Programming < Regular Programming (usually) ● But, there's a time and a place ● How to make it work, when it makes sense to work

Who I am ● Chief technology officer at Trading Technologies ○ TT makes software for professional traders around the world ● Background in wargaming: terrorist network modeling and analysis, agentbased modeling, etc. ● Many similarities between these industries: a knowledge gap between the idea generators and the technologists

e, Lakin, Fred, Visua cienc S r l Grammars for V e put isual Languages, Seattle, Washing Com f P to ro n, ce o Ju ed ly in . gs 19 P t 87 of a . the American A,ss nd Key weor y,ds Dep Raan jedevPahrnases: visual gr sity ociation for Artificial Intellige r Eco mnv e am d pirer v m i M ar nce, icsa no tation, declarativ al tio na Stu real t Blis te Un dyl ,gr1ap hicsa,rgviasu e programSm a u t KAe.7 r n 9 yw e 9 p, in tt g, vi 3 su , vi o n sual ruele IEEE : Lordgiscala IsalItem o E acs, vmacs, VP eg.C ndngPua OrIV ag s s, formal visual languages, info uBsuerrn-e hrge , L: Parsinga, nVgPuLasses: F Symposium o asier to W 3 / intte r t, rM s it rmal e VI.A: Formal defin l n fa o m M rms/3, V r acregadreesig V a 1992 ed ition of VPLs, VP PL-II.A.4 isouuarl oLfaFnoguag trix Meatn-bipausla t n Pr,oBcer M Meyer is s L-II. . u a e tion Pro es, dBsehreg eend ,g BeyondnIc Torm-b d on Kaenyg din Aslls: Y ed: F grams V e To a e d w s i n ar a n e ds s , ,oS Prowce r u o L N red e N ew d f p d . . o M G s etaphorsndfosran ainn rway, A isually o dvaat ags d A ler, ndMof e Ate IIo Vdissua prleQaue Iby ncio .g tAt,m L Eic P. hBth ugust 2 r Textua :SE euna dsry da ruarsneIn drnVleiW orbks La estt: , rn apr erfa ng hee hoG eon ua lvR p s einnege ge n 4 s F ts a In e fo lly? An B 2tia r, te r r la b t u S o Lo r 7,l 1In9fo b pa . a nd ce ti m h a F s s on o l a l to e i , n o C M 19 z D I d n m at rm s 93 KeKyey 9 n r o i ab t r h ). n d m la 3 n as t ip . ation Sys eFroface g Eve e Systems, Glasg lingrsg,a1re99 nguage s/3, Vcrete wowror mua s, ges, : dsds F 4 o 4 n . ng ow anan s s r P Late d a , , m t IE 11 P ing A hr dnP m 3-V13 Detec as m PL5, ra -II.A.4 s toInGteernna IL.Lo es: VPL-EII. EP og I:m hM Ju Pr al ly r S d V su a 19 Vi L y I. A 92 s in .7 C . e ng t (p dli a i s ub p e t .1 o : an gi i : osciula . hed : sCiuom aoliLFoonua Forsm/3s,/3V en nza to nEv an ge ch mng e rm s,stiv gnoitnivelis oa ldB VreP d t-H W Ap A -isbra II.pr ith G e 1995. orm b o R , o .1 V s cla p a d : V D r De e tr ia A anVP P k m gr u a r, s u am e s L y ble . c a p J d L92 m Am ti h d S l at 19 II L. o n o ic , F 8, l L o .A len -1 n la a n a Al a u 15 ng s p : . d n, s .4 n n s pt ua n tio an J . , y Se s ac ge g th M e duas-4p : FoSe 0, s, Romreough P intV re tPio,LAbstr V :ab aes .A A, 34 6,t, Marhga g III set W n V r Ha Bu L, -et .B e m r tle VP rn : a r at s D e 0 e at r , b ag i s, s n a S gu ge a 6 , as I e lan ua u t. e s d I languages ed an d spresad aElyE,EMrspectivael Prog 5 d PWorkshop on Visual Lang ist-btas eotuee hOtI,ed spreads heLsh anEE TR 919 ratemm y 27-2 -Orien 92 6-14,aO dsIE r-)b, a1, s1M 4: Form-baseddan e A. r e II. Ld h p o VP , e d a i l s/3 n e a w rm P 9 A tg Fo c n r based la guag tober, 41992 eveal ly ges sLa oste Key Keywords and Phrases: ngng is oufaua -na .99LV4isuaLl a 7,igl1h nguageepu s,t,VJo of ges, ur ming (P H P m . y L s a ut r r O g al lin g e I ic nd I o ph ha V I t .A: Ab III.E: Even aint Pr VL: 95l.Specification of Gra m Proc. on stracutle-based strha oGnra Gr 5-9,V1is9ua C , m iu r Laki s o f e o s t b p la 1 s n ho 1 m e e ic . s: R ion, VPL.N n irFor n Cm I.Age nVla X Sym terface generatio L-Iua es RKa.elan , Septe sed languag NIU Png y ndv, T. e E n E , S a d Cordy, JamV s in U rm e , se r u , a y is 3, w agbuted Implementation of Multi.t, GCeomp nstraint-b l langua utri uleVsPL-V.E: osrt92 19 Rs, ge Hltardin a ri,, A lanDgis dd e ting, pu it ie s r ic r F m om na v o m at u C tio r e w om ig nc C u a a ut an y t e Fu A d : : i r a D n n.A 2II.A.5 ng w raph R nto, October N cas dL-.P odelultfo easnd,dPJhr tioVes :LV M. POcto ngu i-para -IIP ag.an hras wites nrP Views as a M rative Work, Toro 1 Workshop G ith l pe . na a h M oo io M C ds at . : : d or a el l o w R s 4 b rte 8 , L T ey . po e e 9 K l l r up rn a s rP.19and gUes: c Query 9G .AomputeSQL r-S vs. Visual Languages vs. Boursier, M. Mainguenaud, Hypermaps, 5th , Nrnd uKa inSpatial IIC To h.r.a 9 mgre WG'9 L: -Extended P3 IB Vis -on :u lasis A 1p0th0e,xC1tLanguages -ur o3m a , VePon Gee adn, dT. C rfe g aLgm eora StaNnicpho agpre yrw oan s rm ehsnc ron ,ra8Fo , Proc. asl Report A nthV dha s e n P o s a G e u io m u t id , w l e International of Symposium Spatial Data Handling, Charleston, USA, August 3-7, 1992. s i g t t y P c ia gs i d n s a in e a ge ic c n u ed y ration laTng K Koe rplaosge w Rhra,inPgro PLt-iv F ce dsio hn ecua Asso at enaO geSne Ve.A iting X lm sE A,npp thutael D orms, P .E: User-hinRteerfa d.rlic Z,PCns e,sNVPL-II: wrce Uese .P :CIV also: anLId.-A Keywords and Phrases: Language D :sGe Viis ,r aA Impera Classifications S . r H : r U 1 V , sed a x 6 L, 9 p r R . P I t ) V u a t o e 9 a , I . s, L r I f h g 1 . v ge lo t c g M L nng an Ba: le ational la -I GA. .R Keyw Sc e19192 2, Language, Proc. WG'89 raphrw regG 3.5 ule-ba eraua L en gau 6 : VH o L e R P f . ( 9 D n a l : 8 nc V 9 d n b s s a la 1 Fu 4 2 : e t e l o i o 1 , s u cation n A . g A ic Specifi , i II. g u a p e t y flowtelaGraph urm a s, Introdu CMP/S a Gramm urtpruoctar cBased :V es, o a Attribu an, A rIG dLfo mr Vegrgla -II.A RES, n PROG to hdction Tfetchhnic K A, a em Poshr aneg sPDaes ngounatgro .ASum Schuerr, LAndrea ranP yu ArdPas w seS(Nagl ndan orw s, VPL 5, 1989. s, ed.), ia ndygm euldFeuScienc ey ncttiic rddsor K rningreM., alVerlag elsS ,e151-16 e g S sads p er : l s Spring v C a R a , 1 a . P u t L 411, V i o e r n , LNCS r e, s s g o B p P d p 0 ter I w . a h , n Compu I in o u g S o L pts y 7 n I is rassiuem onteliarm 5 -V.A: G heoreti al padigm laP Co rt, UnivRule-b uaege in SCIaLl-,LVV LP-h c sConce on Grapht WorkshopKe S es:Charles NPiC ersity oased eP Ap, p2Novis: 0 es, VP P.raand VPL-II. nesuExploring VPImpera aam s: uA.8: nfeA.11: langua ara tges, tern Design 9d-n2C n: dVlangua eA, Visual Glinert, Laboratory for of aradigm pu Multi-p r,altV agEphraim osNorton, c1VPL-II. Lo li iges, u -IrIf.,AD. N le tive fArrays, n, lt p renProcessor ie inguPhrase l TItaesra P s: VPL-II.A.6: c L p r-pIVosthe . M m P 4 S 3 u Keywordslaand L o : d : r mtio r , m I r p 8 c n D a I e . o ania, o t e p N e . a e A s n A u u u . g d t la e I a t p , o I n Z in i . n . r v p a f . D Journal of Visual Languages m and Computing, Vol. 3, Number 2, 135-159, S June 1992. 5 g g v a lo is L t u o A u , u P . w n : m : t a C ges langua lanFguunacg age2s8, 1V nme anford rans gAelssastig BuVPL-II. ion languages, reas, mmatic in,g1I al programTm -1D8e8-1VPL-II.B.2: 1m ndDiagra rnetet,rr,B.1: 98 epts inVPL-II.A.1: A lang Computational sn,aVPL laetnor VPL-IV.A: olanguages, ngtuie le oncNovis, Univw Keywords and Phrases: 1991. Concurrent n9te2r.fIconicelanguages, g in 9 c la C , M g u in y . e h S ic n a t c iv 1 uage ro l I a t n r s e I S v g r I a , lang .A.2: 1P99 Langu h-theaoret M aectewe irGrapho(innm er l floersi erm representation p G ars t Gramm m t , I n B , e on op e u r n Worksh o a t : n t Int. o p 4th 4 v c Proc. approaches, VPL-VI.C.6: Visual o issues e . t 6 r g c ars, a C , a s . li r s,ePROG . dural on e Graph Gramm gees Confe Based ndVHL-La Geo , JoTuH rroceeisdual l ty, Augu g nguage I.A achaA I.pBr:etP I a RES: I A I u L A Schuerr,KAndrea g L e a ein P ll n h r P b t e ywG V n g s V la st s an ingsiso d al ofraVses: n L. Amatic 532, Springer Verlag W racutio o sh,i Report AIB 90-16, p R ep 9, 7s,tion arga 989. also:latTechni ncal dVcis m ges a,n641-65 11991. -1rd h ter LNCS e, andto isuaScienc , m g a . 1 P c t V b ual f, VPL-II 1980. Compu a n u 9 r c abApplica P le g a d g and Their om O n L n l r P l d , a s ia f , a L V e 3 L t h I o I D r s a I s l 4 n n l . r a : a C a p d 1 n a a t r c f 1 b e u . n g . s o i 4 8 on, V ny,D.L t and Simulating Execution Norton, Charles P.agGlinert, for Designing ofrcoProcessor lers) .A.5: io :au active Environment r,m S IeI.sB: FEphraim , 13 Jo -and tnt, VAisrVisual ersne 1990. Key,tw pe Germa aly orms/3 u ntRule-b P P adshee Itis RWTH Aachen e t V a e V , i u L r , o m e B p s n I d . n s ased m e I u ig I A.11: o g M d . VPL-II. C s a ges, EWorkshop t , VVisual Syst a s: IEEE Skokie, Octoberin1990. aradigm langua oRmpuVPL-II. : EA.6: PL langua areon es,ges, anMulti-p tive ngu1990 edl DaIllinois, veanrg gle As ta Abs227-232, sA.8: VPL-II. aLanguages, CaArrays, and losw tin-b Keywords t Impera l fm rls ga hanudaVPL-II.A.1: .g4:Concurrent rm nd M ,languages, rve Slanguages, VPL-IV.A: Lgan-gIIu.A olan,Phrase o a ntroe t 2 l e r . o , Pro F a s 9 F L : C e c Keywords and Phrases: Novis, VPL-II.B.2: Iconic Computational li o 4 r : P n 6 . t n is r io P e B 0 a A m . V ges . ll t I langua I n , u I I mmatic a A I n Diagra M l B.1: ceed b h , posteVPL-II. o in t L r L a a langua(ges, ndoM le sed an Iteraatrio a Dec ba VP chn 199 , VP , srhgoap Visual 2 s / rp k Am e r ings s a g approaches, VPL-VI.C.6: representation issues m a la r f d W u p o o r 4 g a F s s . r n r)E : Keywo 89 IEeE p t e iv m s of la r , t r e.adsh e Visu ed s l Fo a hrEas Burnett, I a b 19rds an s 1a9n9d5PIE u 0 t 9 e is a e 9 e V l e 1 , t P h t t e n E s b r e n d d t d o a e n u r gramm sed la s, J prea M. Bgurrating Keywo Phrases: F Symrgpaorseiu tm nguag ing iclgPorreits -based and s mA e o a d o e r a M n m s c h d , m V A s r n V , / is o a ) 3 P . 2 F A , u ( L : L V a 1 n 4 -III.A.1 imat -II.lAL. an , Allen uting, PL-I : Data Ambler es and Comp Forms/2I.,AV.4P:LForm- guages, Da ion into a D e : b r g s c m a a e la u s s s g ed and ra ta Lan Phra spread dt, German tive Visual P rds and y, sheetr Keywo based September ogramming Langu langua 1995. age ges

Who I am not

● I am not a visual programming academic ● There is a sea of research on the various forms, families, perils and merits of visual programming ○ You won’t find any of that here

Low-latency trading & VPL Communication Barriers

Safety

Why I decided to build a VPL ● I didn't! - I stumbled upon it ● I was still learning the business ● I had insanely aggressive deadlines (see: minutes) for writing and deploying production code It was built out of necessity

I got tired of these guys yelling at me

Modules made my life easier

int qty = currOrderQty - _officialQty; if (qty > 0) { OrderProfileBase orderProfile = null; AutospreaderSyntheticOrderProfile aop = null; if (currInstrument.IsASEInstrument) ed { m d i e T e //aop = new AutospreaderSyntheticOrderProfile(currInstrument.OrderFeed, g g eg d L aop = TopMostAlgorithm.CreateAutospreaderOrderProfile(currInstrument, r Hed 2 a e r r orderProfile d = eaop; Syntheti Sp r c} O ter o Stop else Qu { orderProfile = TopMostAlgorithm.CreateAlgoExchangeChildOrderProfile(cur

}

//orderProfile = new AlgoExchangeChildOrderProfile(currInstrument.Order Arb AlgoExchangeChildOrderProfile algoExchangeChildOrderProfile = (AlgoExch algoExchangeChildOrderProfile.LeaveWhenPaused Routi=neleaveOnPause; algoExchangeChildOrderProfile.LeaveWhenCanceled = leaveOnCancel; if (IsFlagColorSet) algoExchangeChildOrderProfile.Color = flagColor; M

arke Mak t er OrderAction.Add;

orderProfile.Action = orderProfile.QuantityToWork = Quantity.FromInt(m_CurrInstrument.Instrument, orderProfile.BuySell = m_Buy ? BuySell.Buy : BuySell.Sell; orderProfile.OrderTag = m_Flag; orderProfile.OrderType = m_OrderType;

But modules weren't enough ● Bugs could still spell disaster ● I wanted the traders to be in control ○ Just piecing them together took work and programming know-how ○ This "work" wasn't going to add any business value

The VPL was born ● I built a GUI that let them visually link these modules together ● I didn't know I was building a VPL ● Algo Design Lab (ADL)

The "Deutsch Limit" “The problem with visual programming languages is that you can't have more than 50 visual primitives on the screen at the same time.” -Peter Deutsch

The "Deutsch Limit" ● The limit of 50 primitives is arbitrary ● We rely on our IDE to help us solve this problem ○ "Find References/Usages" ○ "Go to Definition/Declaration" ○ "Find in Files" ○ Refactoring ● Mental Map

The solution is threefold 1. Domain specificity 2. Hiding the "dirty stuff" 3. Mitigating the "Deutsch Limit"

1. Domain specificity "The ability for the general public to understand a VPL is inversely proportional to the likelihood that it will actually do something useful." -me ● This is really no different from a well-made API ○ Abstracts away unnecessary granularity ● Unspecific: Scratch, AppInventor, etc. ● Specific: Lego Mindstorms

2. Hiding the "dirty stuff" ● All VPLs hide some of the generic dirty stuff ● Goes hand-in-hand with domain specificity ○ If not domain-specific, it's hard to identify the dirty stuff ● Examples of dirty stuff in automated trading ○ In-flight Orders ○ Fairness ○ "It just works"

3. Mitigating the "Deutsch Limit" ● ● ● ● ● ●

Browser-based navigation Show Up/Downstream Blocks Block searching Componentization Breakpoints "Jump Blocks"

We want our users to eat fruit Kathy Sierra - http://goo.gl/hKd95

● Cake vs. Fruit ● We want them to focus on idea generation, not implementation ● It's also about safety

Let's do an experiment ● Build a simple trading strategy: ○ Place an order to buy 100 shares of stock (e.g. GOOG) at the last-traded-price (LTP) ○ Whenever the LTP of GOOG changes, we should change our order's price to match ○ If the LTP of GOOG *ever* falls below $770, we should cancel our order ● Scratch-->Domain Specific Language-->ADL

stock=GOOG;volume=5433728;LTP=782.03;

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Domain Specific

Algo Design Lab (ADL)

(that's it.)

Domain Specific

Domain Specific

Wrap-up* ● VPLs can work for a very narrow set of industries and applications ● Domain specificity ● Hiding the dirty stuff ● Mitigating the "Deutsch Limit" ● Are our users fruit or cake eaters? * Please exit through the back for processing

Questions?

Appendix

Flowcharts are evil “The flowchart is a very poor abstraction of software structure.” -Fred Brooks

Flowcharts are evil ● Poor representation of the actual effort needed to realize the intended goal ● Flowcharts tend to be written after you write code, not before ● Too many assumptions; too abstract