An FPGA Based Resources Efficient Solution for the ...

2014 IEEE International Conference on High Performance Computing and Communications (HPCC), 2014 IEEE 6th International Symposium on Cyberspace Safety and Security (CSS) and 2014 IEEE 11th International Conference on Embedded Software and Systems (ICESS)

An FPGA Based Resources Efficient Solution for the OmniVision Digital VGA Cameras Family Elmar Yusifli∗ , Reda Yahiaoui† , Saeed Mian Qaisar‡ and Tijani Gharbi§

∗‡§ NanomedicineLab,

† Institut

Franche-Comte University, 16 route de Gray, 25000 Besancon France Femto-ST, Franche-Comte University, 32 avenue de l’Observatoire, 25000 Besancon France Email: [email protected], elmar.yusifl[email protected]

Abstract—Camera is an elementary component of the modern systems employed in robotics, defense, surveillance, automation, etc. In order to keep the system compact, embedded camera solutions are developing. A proper system integration requires an effective camera driver. In this context, an efficient FPGA based digital camera driver is devised. The proposed solution is compatible with most of the Omni Vision digital cameras. The developed system functionality is verified with a developed testing methodology. The system on chip implementation resources are computed. It shows that the devised solution outperforms compared to the coexisting ones in terms of resources utilization.

I.

C++ can be compiled by the microcontroller compiler, which translates it into microcontroller specific assembly. Finally the microcontroller CPU step-by-step executes these instructions in a serial fashion. The above discussion shows the microcontroller based solution short comings. It can not achieves a parallel execution, a vital requirement of the modern vision solutions. On the other hand the FPGA technology employs the parallel execution and concurrent computing. Therefore, FPGA based digital camera drivers are more advantageous than microcontroller based solutions. It opens the possibility to use them on-fly image processing, face recognition, object detection, color detection and other applications.

I NTRODUCTION

This work is part of a project, aimed to enhance the embedded camera systems. The motivation is to achieve resources efficient and configurable solutions while keeping the system cost effective.

II.

The proposed solution is compatible with most of OmniVision digital Cameras. It can be easily done by configuring the camera resolution, clock frequency etc. The required configuration is done by changing the camera configuration register contents [5].

The vision solution is a developing technology. The recent advancements in robotics, military, micro technologies and automation technologies have tremendously increased the vision processing demand. Interesting solutions have been developed for a variety of applications like automated video surveillance [1], optical flow [2], Micro Air Vehicles [3], etc. In order to facilitate the system embedding, almost all modern cameras have digital interfaces. It renders into simpler implementation and cost effective solutions.

Any VGA OmniVision Camera module costs around 20$. The solution is also tested on Cyclone EP2C5T144C8N board that costs also 20$. With an external 1MB (Mega Byte) SRAM module, AS7C38098A [15] a complete camera interface costs 55$. It shows the proposed system cost effectiveness. The configuration can be done with the developed optimized driver. The devised solution can be integrated into any robot or automation system. In place of microcontroller driven image sensor solutions, FPGA based OV family digital VGA cameras drivers are proposed. The proposed solution is specifically developed and optimized in VHDL. It is successfully tested on OV7725, OV7670 cameras. It can parameterize cameras by accessing their configuration register. Both grayscale and color modes are available. The acquired image is translated into horizontal and vertical signals and pixel color values according to the employed color mode. The color RGB mode is coded in 16 bits: 5 bits for green, 6 bits for red, 5 bits for blue. In total it gives 65536 colors. The grayscale mode is coded in 8 bits. It renders into the 256 gray scales. Each acquired image is memorized without any interruption. FPGAs are the natural choice, because of their sufficient I/O pins availability and high frequency interface they are compatible with all modern memory chips. On the other hand, microcontrollers internal memories are limited to max 50 kilo bytes. Moreover, due to

OmniVision Technologies proposes one of the cheapest and widest employed CMOS image sensors [4]. They are specialized in digital and analog image sensor production. This work deals with the OmniVision Technologies OV digital VGA cameras family [5]–[7]. The proposed solution can be easily configured for any component of this family. However, at first time a study is made on the OV7725 digital camera [5]. This is a VGA (Video Graphics Array) color image sensor. It can provide RGB565, YUV422 and RAW color systems with maximum 120 frames per second in QVGA (Quarter Video Graphics Array) mode. The maximum image resolution is 640 × 480 pixels. The camera modes can be modified by amending its configuration register contents [5]. The purpose is to realize a fast, simple and cheap FPGA based camera interface with a personal computer. C++ based drivers for CMOS Cameras are developed [5] and these solutions are compatible with microcontrollers [8], [9]. It can be downloaded and used directly for the microcontroller configuration. The driver programmed in 978-1-4799-6123-8/14 $31.00 © 2014 IEEE DOI 10.1109/HPCC.2014.107

P ROPOSED SOLUTION

566

DE1 Board

RS232 Port

PC

FPGA (Cyclone EP2C20F484C7)

Camera Driver

OV VGA Camera

Data Acquisition Interface Camera Configuration Interface

BUTTONS

Fig. 1.

Memory write encoder

Data Transfer Controller

SRAM Driver

SRAM (512 Kbytes)

Memory read decoder

UART Driver

VGA Driver

VGA 4 bit resistor network

VGA Monitor

Fig. 2.

The design of Camera SCCB driver interface

Fig. 3.

Flow chart of camera driver

The proposed system architecture

a lack of I/O pins and interface frequency performance they are very limited in driving the external memory chips. The ALTERA Cyclone family FPGAs are based on SRAM technology. The image frames are streamed to the VGA monitor for real-time surveillance. A data transfer interface, on board RS232 connector with High speed USB to UART converter [17], has also been integrated in the proposed solution. It can be employed for PC based software image processing and visualization applications. A. Design and Implementation The proposed solution architecture is shown on figure 1. Digital camera interface is realized via the DE1 GPIO port. The main system blocks implemented on the FPGA are the digital camera driver, the memory write encoder/decoder, SRAM, UART and VGA drivers. Images received from camera are stored on a graphic memory for further utilization. The Camera is driven by the camera driver block. It simultaneously configures the camera and receives images from it. A real time data memorization is achieved via the memory write controller block. It encodes images for the memory driver. The memory driver can simultaneously perform the data write and the data read to and from the memory. Images from the memory are streamed to the VGA monitor. UART driver arranges with memory controller and can transmit the data to a required destination.

Start false

true

odd byte

G7 G6 G5 G4 G3 G2 G1 G0 7 0

true

Color mode

true

odd byte

R4 R3 R2 R1 R0 G5 G4 G3 7 0

8

+

G2 G1 G0 B4 B3 B2 B1 B0 7 0

8

pix_col pix_lig

Fig. 4.

1) The Camera Configuration: The camera configuration is achieved by programming certain camera registers. In OV family VGA Cameras up to 150 registers can be programmed. The configuration data exchange between the FPGA and the camera is achieved via the SCCB (Serial Camera Control Bus) [18] interface, based on the i2c protocol. Where, FPGA is the master and camera is the slave [19].

false

address calculator

8

16 memory_data

memory_adr

Flow chart of memory controller write block

P CLK) are also employed for proper image acquisition. Each image data begin on falling edge of V SYNC. Number of H REF rising edges is equal to the image lines. For each line 2 bytes data correspond to one pixel. The camera driver block reaches data by using V SYNC, H REF, P CLK and DATA wires and gets out data on 2 counters: pix col (column counter), pix lig (line counter) and a pix data that correspond to a color code per pixel. Then memory controller wr block calculates the memory address of each pixel from the counters for storing them into the graphical memory. The flow charts of camera driver and memory controller write block are shown on figure 3 and figure 4.

Figure 2 shows schematics of the SCCB driver block and the parameter block which realizes the camera configuration. The parameter block contains camera configuration parameters. The SCCB driver configures camera by using the data stored on the parameter block. The SCCB driver block also confirms the proper configuration parameters reception by the camera. 2) Image Acquisition: Camera data is captured and transmitted by 8 bits data bus. The 3 wires (V SYNC, H REF,

The data acquisition and the data storage solution schematic is shown on figure 5. Driver solution is capable to configure 567

to memory driver

is easy and very effective to program any dual port memory block in VHDL. Dual port RAM block allows data read and write at different frequencies. It allows to get images in streaming mode.

Fig. 5.

The employed Cyclone 2 chip composes 239616 memory bits. Therefore, the maximum picture resolution that we can store should not be more than 160 × 120 pixels in 8-bits grayscale mode (153600 bits). In this case, the driver solution can get 120 images per second with OV7725 sensor module.

The design of Camera driver and memory controller write interface

Later, DE1 on board SRAM chip is used as graphic memory [20]. It improves the proposed system performance in terms of available memory. SRAM data storage capacity is 512Kbyte (262144 addresses of 16 bits word). It makes possible to store any image data with 320 × 240 pixels in 16 bits color mode (120fps) and 640 × 480 pixels in 8 bits grayscale mode (60fps). Onboard SRAM chip have one address port to read and write. In streaming mode some conflicts occurs while simultaneously reading or writing the data. In the aim to resolve the conflict a write first priority is adopted. Images can be read if writing access to the RAM is stopped.

Fig. 6. Image captures in gray (640×480 pixels) and color modes (320×240 pixels) Real time image display

In test system, it is also possible to send captured image to any external device using on board USB to UART bridge. Instead of showing the captured images on the monitor the UART driver (data send cntrl block) reads each pixel data from the memory and (uart transmission block) serialize and send it to the external device. IV.

Fig. 7.

In literature several SoC (system on a chip) based camera drivers exist [1], [10]–[14]. These solutions have been implemented on Altera or Xilinx FPGAs. In [10], authors have used analog cameras, which are driven by the Altera SOPC IP Core and processed by the NIOS processor based software. [1], uses CMOS VGA camera in grayscale mode which is driven by an embedded microcontroller. These solutions are more resource expensive compared to the proposed one. In [3], a high resolution Terasic Camera is used in streaming mode. The controller is realized on Cyclone IV FPGA. It uses total 1739 logic elements and 1062 registers with 328.29mW thermal power dissipation. In the case of camera driver (320 × 240) with Altera Cyclone ii EP2C35 total 6203 logic elements and 3182 registers are used [10]. [1], uses more resources even in QVGA mode (320 × 240 pixels). Using PTZ camera VGA vision system with Xilinx Embedded Development Kit (EDK) is more expensive and consumes more because of SoC implementation [11]. The realization of 640×480 pixel video driver on Xilinx FPGA uses 9094 Slices (38%), 11451 slices Flips-Flops (24%) and 12883 four input LUTs (27%) [11]. In [12], an OV7670 camera is driven by hardware. It also drives a SDRAM through NIOS processor. However, no resolution or resource utilization statistics are reported. A common approach in these solutions is the use of C based source codes. These C codes are later on translated into the HDL (Hardware Description Language) with specific translators [1], [10]–[12]. It implements a microprocessor based solution on the FPGA. These implementations are not optimized and employ more FPGA resources compared to the

Complete system design

the camera and get an image in two modes: grayscale and color. In grayscale mode the camera sensor is configured to YUV422 mode and just Y byte luminance is used. In color mode camera sensor is programmed to RGB565 mode and 65536 color palette is employed. The image resolution is a function of the used graphic memory. An image example is shown on figure 6. The captured image color display quality is limited because of the ALTERA DE1 VGA DAC 4-bits resolution [16]. This short coming can be recovered by using a higher VGA DAC resolution card. III.

R ESOURCE U TILISATION

T EST AND VALIDATION

To observe the system functionality a VGA monitor is attached. It shows a realtime captured image visualization. Figure 7 shows the complete system design with OV VGA digital camera, FPGA development card, VGA monitor, USB RS232 bridge and image acquisition unit. The memory driver block drives physical Random Access Memory (RAM) for storing and reading data. Firstly, we have tested our system solution with the FPGA internal memory. It 568

proposed solution (cf. Table I).

R EFERENCES [1]

In comparison to these realizations we have made a Camera driver, optimized for any application that uses total 612 logic elements (3%), 214 logic registers (1%) that uses 0 bits internal memory (cf. Table 1) .The implemented solution power dissipation is 74.62mW (cf. Table II).

[2]

[3]

These results demonstrate that the proposed solution outperforms compared to the cited ones in terms of power efficiency and heat dissipation.

[4] TABLE I.

R ESOURCES UTILIZATION SUMMARY OF C YCLONE II EP2C20F484C7 FPGA

Resource name

Quantity used

Quantity available

Total logic elements Total combinational functions Logic registers Total pins Total memory bits Embedded multiplier 9-bit elements Total PLLs

612 607 214 82 0 0 0

18752 (3%) 18752 (3%) 18752 (1%) 315 (26%) 239616 (0%) 52 (0%) 4 (25%)

[5]

[6] [7]

[8]

[9] TABLE II.

P OWER A NALYSIS R ESULTS OF C YCLONE II EP2C20F484C7 FPGA

Resource name

Quantity used

Total Thermal Power Dissipation Core Dynamic Thermal Power Dissipation Core Static Thermal Power Dissipation I/O Thermal Power Dissipation

74.62mW 0mW 47.36mW 27.25mW

[10]

[11]

[12]

V.

C ONCLUSION

An efficient digital cameras driver has been developed. A complete system design has been described. The proposed solution is compatible with most of OV digital VGA cameras. It can be achieved by simply updating the camera configuration registers contents. The proposed solution not only drives camera for image acquisition but also manages a real time data memorization. A test methodology has been developed for the proposed solution verification. Results have shown a proper system functionality. The system on chip resources utilization has been computed. It is shown that the devised solution outperforms compared to the concurrent ones in terms of configurability and resources utilization. A real time image-processing tool has been developed in C++ with Qt-Creator. In order to increase the system resolution an important task is to develop a higher data rate interface between PC and the front-end module. In this context an Ethernet link integration is in progress. In future, the developed solution will be employed with the micro-conveyor for planar micro-manipulation [23]. A complete control of twodimensional conveyance will be developed in order to achieve more precise objects conveyance and positioning.

[13]

[14]

[15]

[16]

[17] [18]

[19] [20]

[21] [22]

ACKNOWLEDGMENT This research is supported by the French National Agency for Research (ANR) - SmartBlock project (http://smartblocks.univ-fcomte.fr/) and SyVad project (http://syvad.univ-fcomte.fr).

[23]

569

R. Brzoza-Woch, A. Ruta, and K. Zielinski, “Remotely reconfigurable hardware software platform with web service interface for automated video surveillance,” Journal of Systems Architecture, vol. 59, no. 7, pp. 376–388, 2013. G. K. Gultekin and A. Saranli, “An fpga based high performance optical flow hardware design for computer vision applications,” Microprocessors and Microsystems, vol. 37, no. 3, pp. 270–3286, 2013. O. Ranganathan and A. I. Rasheed, “Design and development of camera interface controller with video pre-processing modules of fpga for mavs,” SASTech, vol. 12, no. 1, 2013. [Online]. Available: http://www.msrsas.org/docs/sastech journals/archives/May2013/2.pdf OmniVision, OmniVision Technologies. [Online]. Available: http: //www.ovt.com ——, OV7725 Color CMOS VGA (640480) CAMERACHIP Sensor with OmniPixel2 Technology datasheet. [Online]. Available: http: //www.ovt.com/products/sensor.php?id=80 ——, OV7670/OV7171 CMOS VGA CameraChip with OmniPixel Technology. [Online]. Available: http://www.voti.nl/docs/OV7670.pdf ——, OV7740 Ultra High Sensitivity, low power CMOS VGA Sensor. [Online]. Available: http://www.ovt.com/download document. php?type=sensor&sensorid=83 D. Crispell, Implementation of a Streaming Camera using an FPGA and CMOS Image Sensor, Brown University Std., 2009. [Online]. Available: http://www.lems.brown.edu/∼dec/docs/fpga camera.pdf A. Rowe, C. Rosenberg, and I. Nourbakhsh, “A low cost embedded color vision system,” Intelligent Robots and Systems, vol. 1, pp. 208–213, 2002. [Online]. Available: http://www.cs.cmu.edu/∼cmucam/ Publications/iros-2002.pdf Y. S. Jinbao Yuan, Mingsong Chen, NIOS II Based Intellectual Property Camera Design, Brown University Std., 2007, altera Third Prize. [Online]. Available: http://www.altera.com/literature/dc/2007/c3b.pdf Y. Said, T. Saidani, F. Smach, M. Atri, and H. Snoussi, “An fpga-based smart camera system, third international ieee conference (ipwis13),” 2013. R. Wang, Z. Mi, H. Yu, and W. Yuan, “The design of image processing system based on sopc and ov7670,” Procedia Engineering, vol. 24, pp. 237–241, 2011. T. Khan and K. Wahid, “Universal bridge interface for dvp-compatible image sensors,” Microprocessors and Microsystems, vol. 35, no. 6, pp. 547–556, 2011. H. Sun, R. Cai, and Y. Wang, “Design and implementation of high-speed digital cmos camera driving control timing and data interface,” SPIE Proceedings, vol. 6358, 2006. [Online]. Available: http: //proceedings.spiedigitallibrary.org/proceeding.aspx?articleid=1331848 A. Memory, AS7C38098A - 512K 16 bit High Speed CMOS SRAM. [Online]. Available: http://www.alliancememory.com/pdf/sram/ fa/as7c38098a.pdf ALTERA, ALTERA DE1 Development and Education Board. [Online]. Available: http://www.altera.com/education/univ/materials/boards/de1/ unv-de1-board.html FTDI, FTDI Chip - USB RS232 converter. [Online]. Available: http://www.ftdichip.com OmniVision, Serial Camera Control Bus (SCCB) Functional Specification. [Online]. Available: http://www.ovt.com/download document.php?type=document&DID=63 NXP, I2C bus specification and user manual. [Online]. Available: http://www.nxp.com/documents/user manual/UM10204.pdf ISSI, IS61LV25616AL 256K 16 bit High Speed Asynchronous CMOS SRAM. [Online]. Available: http://www.issi.com/WW/pdf/ 61LV25616AL.pdf Digia, Qt Project. [Online]. Available: http://qt-project.org [email protected], Qextserialbase - Serial port library for Qt2, Qt3, Qt4 and Qt5. [Online]. Available: https://code.google.com/p/ qextserialport/ R. Yahiaoui, R. Zeggari, J. Malapert, and J.-F. Manceau, “A mems-based pneumatic micro-conveyor for planar micromanipulation,” Mechatronics, vol. 22, no. 5, pp. 515–521, 2012.