learning problems (e.g. Snake, Pong, or other games in OpenAI Gym or PyGame Learning Environment). New relationships will be attempted to be established ...
Project Proposal
Name: Killian Rutherford (KRR2125) Term: Summer 2017 Advisors: Albert Boulanger, Mehmet Turkcan Course: COMS 6901 E sec 057
Overview This proposal entails tentative goals for the quantum computing research project with the Center for Computational Learning Systems (CCLS), Columbia University under the supervision of Albert Boulanger and Mehmet Turkcan. This follows from the similar project I have been conducting in the previous semester.
Summary of Last Semester Last semester began by attempting to solve the CartPole problem1 by using the 1QBIT Software Development Kit (SDK). Specific focus was given to their Simulated Quantum Annealing (SQA) functions and techniques, which were slightly different to dealing with DWAVE embeddings as past projects had done.2 In a phone call to 1QBIT, light was shed on the methods that needed to be used in Quantum Reinforcement Learning, as had been done in their own paper on Deep Boltzmann Machine Reinforcement Learning.3 They stated that Quantum Monte Carlo (QMC) methods were the way forward instead of having to deal with DWAVE embeddings. Unfortunately, collaboration could not be set up with 1QBIT on the front of their QMC code, and we are waiting for them to set up a potential academic collaboration server. 1
Killian Rutherford KRR2125
Project Proposal
Hence, my focus towards the end of last semester shifted towards using classical Deep Learning Models to simulate a DWAVE annealer, given the recent developments in the use of Deep Learning methods to simulate quantum system energies. [3] [4] This simulation primarily generated random binary graph combinations (with randomized weight connections), and were fed into the DWAVE annealer to obtain outputs (averaged over 1000 trials per input, 50,000 total training examples). These were used to feed into a Neural Network (NN) to learn and compare to the DWAVE Annealer output. This worked well for simple graph inputs (2 or 3 binary nodes, 1 or 2 randomly weighted connections). Using Mean Squared Error (MSE) as the loss function, such results were able to be obtained: Num Nodes Graph 2 3 10
Dense/ Highway Dense Dense Highway
Num Nodes in layer 64 128/512 46
Num Layers 6 6 6
Dropout Coeff 0.75 0.5 0.0
Batch Size 64 64 64
Num Epochs 50 50 50
Dropout was added after each layer.
As can be seen, relatively good accuracy was obtained for the smaller graphs, although not for larger, more realistic problem application sizes.
Project Description and Goals I retain the following paths as my main goals for the summer: 1. Develop the QUBO Simulator, trying out different techniques with Highway/Residual Networks [5] as well as other more tailored graph Deep Learning Models (aside project) [6]. However, focusing on the 10 node random weights reproduction is the main task. The end goal is to use this simulation to replace the DWAVE quantum computer and the restrictions put on solving QUBO problems. 2. Continue to approach the CartPole problem, focusing on QMC methods.
2
MSE Test Error 6 ∗ 10−4 6 ∗ 10−3 0.290
Killian Rutherford KRR2125
Project Proposal
3. If the latter is successful, move on to more challenging reinforcement learning problems (e.g. Snake, Pong, or other games in OpenAI Gym or PyGame Learning Environment) New relationships will be attempted to be established with the SANDIA National Lab, especially focusing on their QMC efforts as outlined in their DWAVE report. [7] The goal is to experiment with QMC code, exploring alternatives to the DWAVE path.
Material/Development As was mentioned in the previous semester reports, I had been given access to a machine, Docker, with 8 processing cores and 64gb RAM. This allowed me to parallelise example-generating code to give a considerable speed up (10x) as well as build large NN models due to the bigger RAM, experimenting with larger numbers of layers and nodes. I have also recently been given access to another machine, Quantum, with the same specifications as Docker, which will give me even more flexibility when looking at the 10 node QUBO simulator (in terms of data creation and NN models). Hence, I will continue connecting remotely to these machines in CCLS, which will also allow me to use the DWAVE functions. 4 I have been utilizing Keras and Lasagne to build my Deep Learning models. It is still yet to be determined what the situation is regarding the remote server that will be set up by 1QBIT to enable us to utilize their code. Equally, as mentioned above a call has been set up with SANDIA to discuss potentially using their QMC software. I may also need GPU access to be able to run code efficiently. This may involve working on Columbia University cluster computing servers (Yeti or Habanero), or setting up an Amazon AWS Instance.
Schedule for work: 1. June: Focus on goal 1, developing the QUBO simulator 2. July: Look at replacing DWAVE functions with QMC in Anaan’s code to solve CartPole.
3
Killian Rutherford KRR2125
Project Proposal
3. August: Continue with QMC work, potentially dealing with other architectures apart from Boltzmann Machines, and potentially look at harder problems than CartPole.
Notes 1
(Can be found in the OpenAI Gym): Given a pole balanced on a moving cart, an agent has the option to either apply a force of a constant magnitude to the left or to the right. The goal is to keep the pole balanced on the cart for as many iterations as possible. The problem is considered fully solved if it remains balanced for 200 iterations 2 The project of the Fall 2016 semester involved building a Hopfield Network, and using a DWAVE annealer to update weights in this Neural Network (NN). A mapping or embedding function is needed to convert fully connected graphs of nodes and weights to the DWAVE quantum computer chimera graph. However, when applied to problems such as CartPole or hand-written digit classification, the results would converge to spurious minima. For more information, see [1] 3 In this paper, a Deep Boltzmann Machine (DBM) was trained to learn a maze, using SQA to simulate quantum sampling. Further, the use of a quantum Boltzmann Machine (QBM) was demonstrated to improve performance in this reinforcement learning task over the DBM [2] 4 Quantum has the IP address which was given to DWAVE to use their libraries
References [1] J. Dorband. A Boltzmann Machine Implementation for the D-Wave. arXiv preprint arXiv: 1606.06123v1, 20 Jun 2016. [2] D. Crawford, A. Levit, N. Ghadermarzy, J. Oberoi, P. Ronagh. Reinforcement Learning Using Quantum Boltzmann Machines. arXiv preprint arXiv:1612.05695v2, 25 Dec 2016. [3] K. Mills, M. Spanner, I. Tamblyn. Deep Learning and the Schrodinger Equation. arXiv preprint arXiv:1702.01361v2, 8 Jun 2017. [4] L. Oliveira et al. Jet-Images – Deep Learning Edition. arXiv preprint arXiv:1511.05190v3, 22 Jan 2017. [5] R. Srivastava, K. Greff, J. Schmidhuber. Highway Networks. arXiv preprint arXiv:1505.00387v2, 3 Nov 2015 [6] M. Henaff, J. Bruna, Y. LeCun. Deep Convolutional Networks on GraphStructured Data. arXiv preprint arXiv:1506.05163v1, 16 Jun 2015.
4
Killian Rutherford KRR2125
Project Proposal
[7] O. Parekh et al. Benchmarking Adiabatic Quantum Optimization for Complex Network Analysis. SANDIA REPORT; SAND2015-3025, April 2015.
5
