Machine Learning and Data Science Course. Purpose/Objective. The National .... interpretation models covering all topics
Request for Proposal Machine Learning and Data Science Course Purpose/Objective The National Association for Business Economics (NABE) requests proposals from qualified, interested parties (course developers) to design, develop, and deliver a comprehensive training program for business economists in the field of Machine Learning and Data Science. The selected proposal (and course provider) will receive a contract with measured deliverables. A contract with measured deliverables means that detailed statements of work will be provided and prospective course providers will be required to meet specific content requirements on a detailed development schedule. Content providers:
Must be open to accepting comments from the NABE Machine Learning Content Management Working Group and incorporating that feedback into the course material. Will be paid at intervals based on deliverables (progress payments). Will teach the first course (pilot) November 13 – 14, 2017, and accept comments and feedback that may need to be incorporated into the final course deliverables. Following the pilot will provide NABE the completed/final version of the course in electronic and print formats.
Proposals are currently requested in the following subject area: Machine Learning and Data Science Course developers may submit a proposal for a Machine Learning and Data Science course. The course should be designed to be delivered in 14 hours (2 days) of classroom training. A combined technical (content) and price proposal should be submitted for the course. The price proposal should include 1) a firm‐fixed price for design and development and 2) a firm‐fixed price for delivering (presenting) the program (one‐time presentation, including expenses). The best way to teach skill‐based applications is through the use of practical exercises, case studies, group work, and exercises involving the visualization, interpretation, and presentation of data. The slide presentation, generally used in lecture teaching formats, will provide a path to follow that corresponds to the overall course outline in the Statement of Work. The course must be interactive, featuring participation and encouraging involvement. The following descriptions are used to describe the level of detail in the Scope of Work: National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected]
Brief Review – approximately 1‐2 minutes and possibly a PowerPoint slide, part of a larger topic where concepts will need to be understood by attendees. Moderate‐ additional time should be spent to ensure that these concepts are understood as they relate to business economics. In‐depth – considerable time should be spent on these topics to make sure students have a complete grasp of these concepts and how they relate to business economics. Application – a case study, breakout session, exercise or role‐play should be used to emphasize the concept and how it relates to business economics using real world examples. Developers must include course content that reflects both basic and advanced principles and practices in applied economics as it relates to machine learning. A detailed content outline (Scope of Work) developed by NABEs Machine learning Content Management Working Group is attached. Audience The audience for this course will include a wide spectrum of knowledgeable, practicing applied economists with little or no formal training or extensive experience in machine learning, data science or in working with very large datasets. Attendees could be looking to build an applied understanding of the space for leadership and/or direct implementation. For course development purposes, providers should presume that these experienced practitioners are new to the subject area. Developer should assume audience will have knowledge/background in: time‐series econometrics and regression analysis, statistics/probability, calculus, and command line programming (STATA/EVIEWS). Objectives This course is for applied business economists. Its purpose is to expose students to the data, tools, technologies, and methods used by tech economists and data scientists to inform product and business decisions. The course must be developed within a framework and context of adult learning emphasizing participation, interaction, and practice in both the preparation of the materials and the teaching approach. The emphasis is on machine learning vis‐à‐vis theoretical or academic instruction – although it is understood that a theoretical framework may be necessary to establish a foundation for an on‐the‐ job application. Course is to include hands‐on examples and applications in R or other relevant software (20‐25% of the course). Attendees will leave the course with the following: National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 2
A key understanding of the differences and complementarities of applied econometric and machine learning methods, including common applications in industry. The ability to execute on a range of econometric and machine learning techniques in R, or other relevant software. The context for collaborating effectively with data scientists and machine learning engineers.
Scope of Work for a two‐day course Day 1 1. Introduction a. Machine learning, A.I., deep learning defined b. Overview of goals and similarities/differences of machine learning vs. econometrics c. Mapping terminology of machine learning to econometrics (e.g. target vs. independent variable) d. How to decide when to use ML instead of econometrics 2. Basic concepts (Critical to ML, less familiar to economists) a. Supervised /unsupervised distinction b. Prediction in regression c. Optimizing Mean Squared Error i. bias‐variance tradeoff d. Overfitting problem e. How to deal with overfitting i. train‐test‐validate methodology ii. cross‐validation f. Regression examples i. penalized objective function ii. Ridge, lasso, elastic‐net g. Classification examples i. What is objective? ii. Confusion matrix iii. Example: logistic v tree 3. Regression and Classification: More methods a. Tree methods b. Boosting, bagging c. Ensembles d. Applications to prediction policy problems 4. Unsupervised machine learning, clustering, and segmentation a. Text mining National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 3
b. c.
Clustering techniques Applications
Day 2 1. ML & Causal inference a. Natural experiments overview b. Average treatment effects under unconfoundedness i. Low‐dimensional ideas—regression adjustments, propensity score weighting, and double‐robust ii. Average treatment effects with ML 1. Double selection LASSO (Belloni, Chernozukov, and Hansen) 2. Double machine learning (Chernozhukov et al) 3. Residual balancing 4. BART‐based methods 5. Targeted Maximum Likelihood(Athey, Imbens and Wager) iii. Applications—e.g. Advertising effectiveness studies c. Personalization/ heterogeneous treatment effects i. Evaluation of A/B test ii. Evaluation of observational studies d. Instrumental variables & price elasticities i. Review of standard approaches from econometrics, e.g. IV ii. LASSO (Chernozukov et al) iii. IV with covariates & parameter heterogeneity (e.g. “generalized random forests,” “Deep IV”) e. Difference‐in‐differences i. Applications/examples with big data, from tech firms i –e.g. Nosko & Tadelis eBay experiment ii. High‐dimensional methods –e.g. Athey, Imbens et al – 2. Partnering with data scientists – a. Type of work data scientists are doing b. In A/B testing – an economist can think about using this to estimate the parameters of interest rather than just does A or B perform better ‐ price elasticity, etc. c. Using past experiential bucketing to learn about causal effects of treating randomized encouragement trials? Components, Materials, Deliverables Each course developer will provide:
A full curriculum Table of Contents (syllabus)
National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 4
A comprehensive Student Guide. A standalone course guide with extensive applied business economics guidance, concept explanations, practical exercises, case studies and data interpretation models covering all topics in the syllabus A comprehensive Instructor Guide to include the Student Guide and PowerPoint presentation A PowerPoint slide Presentation
The work product deliverables will reflect the principles, but will teach the skills (how to do it). This skill‐ based product must be original work. In addition, the course developer must show the course reference material (texts and reference sets from which the work product is derived). The text references will show the underlying principles, background materials, or concepts. The source materials will serve the following purposes:
Provide resources for students who have not taken courses in the subject area, need a refresher, or may have to research the subject area in the future. Inform the NABE Working Group who must evaluate the work product in light of the text references to determine how the source documents directly support the applied activities.
The Table of Contents, Student Guide, Instructor Guide, Electronic Slides and the Examination must be original work and will be owned by NABE. The Student Guide must include the following sections:
Introduction (what the course is about) Table of Contents (the subjects to be covered) Learning Objectives (student skills at course end) Assignments (how the course will progress) Extensive machine learning guidance and concept explanations Course Activities (exercises) case studies and data interpretation models Reference Materials (text references) Print copies of PowerPoint slides Summary of Course (what was covered) Closing Remarks (tying course together)
The Instructor Guide must include the following (in addition to all of the materials in the Student Guide):
Talking Point Outlines (for each Course Activity, e.g., Exercises) Suggested Solutions (for each exercise under Course Activities) Teaching Notes (for each slide)
The developed and completed course will be owned by NABE under a work for hire arrangement that will be included in the contract. All print, digital, and derivative rights to the course will be owned and copyrighted by NABE. The course developer must include deliverables for live, in‐person sessions. NABE National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 5
will deliver the course in multiple formats to provide NABE with scalability of delivery in anticipation of a demand for this course. Questions All parties submitting a question about these RFPs or indicating an interest in responding to the RFP should submit to
[email protected] and will receive a response on or before June 30, 2017. Evaluation of Proposals The course program and materials will be evaluated for comprehensiveness in design, fulfillment of objectives, and the alignment of objectives with NABE’s goals as described above. Evaluation factors will include:
Overall costs/price elements Creativity in design Comprehensiveness of program Adherence to the Statement of Work Experience/qualifications of the developer(s) and the development team, if appropriate Trial Run of the course (finalists only)
The Trial Run – as a part of the overall evaluation – is a two hour live or web‐based slice of a brief module of the course. The Trial Run is an opportunity for developers to demonstrate the materials, methodology, and instructional performance in course delivery. Time Schedule and Milestones The contract will provide more details. The targeted completion date (providing final materials to NABE) for the course is Monday, October 30, 2017. The first course will be offered in Seattle, WA, November 13‐14, 2017. Costs and Pricing As mentioned above, developers must include a combined technical (content) proposal and a price proposal. A firm‐fixed price proposal for the design and development of each course is one element of the price proposal. A firm‐fixed price for the delivery (presentation and expenses) of one course (a representative each‐time teaching fee) is the other element. Key Dates Timeline National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 6
RFP Issue Date: Tuesday, June 27, 2017 RFP Questions/Clarifications: Monday, July 3, 2017 RFP Answers/Clarifications: Wednesday, July 5, 2017 Proposal Submission(s) Date: Monday, July 10, 2017 Trial Run Presentation/by Invitation: Monday, July 24, 2017 Contract Award(s): Monday, July 31, 2017 Measured Deliverables followed by progress payments one week later: Monday, August 14, 2017
Monday, August 28, 2017
Monday, September 11, 2017
Monday, September 25, 2017
Monday, October 16, 2017 – final materials due for pilot
Pilot to be scheduled on – November 13‐14, 2017
Final Course Materials Due to NABE ‐ Monday, December 4, 2017 Program/Contract(s) Completion Date: Monday, December 18, 2017
Submittals Course developers should submit their completed proposals in electronic format by Monday, July 10, 2017 to:
Tara Munroe Associate Director of Education and Professional Development National Association for Business Economics
[email protected]
National Association for Business Economics; 1920 L Street NW, Suite 300, Washington, DC 20036 www.NABE.com;
[email protected] 7
