disk

File Organization (Dosya Düzenleme) Ders Notları #1 (Disk Organization & Performance) «Alim-i mursid, koyun olmalı; kuş olmamalı. Koyun, kuzusuna süt; kuş yavrusuna kay verir.»

1

Textbook & References  File Structures, Michael J. Folk, Bill Zoellick, Greg Riccardi,

An OO Approach with C++, Addison Wesley,1998  Database Design &Implementation, Edward Sciore, John Wiley, 2009

2

Definitions-1  File Structure is a combination of representations for data in files and of operations

for accessing the data on disk.  Data structures: deal with data in the main memory  File structures: deal with the data in the secondary storage  Main operations on file structure : search, add, remove, update, sort (external sorting) , merge  Main Metrics: simplicity, complexity, scalability, programmability, and maintainability  Structures for static files vs. dynamic files differs a lot.  Structures differ according to the media as well.  A rough History Access Methods:

3

 Sequential Search  simple index search (before 1960 )  tree structures ( BST-1960s, AVL-1963, B-tree (B+-tree) – 1970s  Simple Hashing (before 1980)  Dynamic Hashing (after 1980)

Based on •usage characteristics of data •data type (basic vs. multi-dim data) •physical characteristics of machine

Definitions-2  Physical File is the particular collections of bytes stored in disk.  Logical file is the view of physical file from the standpoint of application

   



program. There are thousands of physical files on disk, but the program can have only 20 logical files. OS make the connections between physical file and logical files. Example: int fd = open (filename, flags[,mode]); Read/Write: First OS make the connection then read/write with using logical descriptor. Physical Devices as Files. The program access the “file” without knowing whether the file comes from disk, tape, another computer, keyboard(stdin), screen (stdout)...  % list.exe > myoutput  % prog1 | prog2  % list | sort

4

MAIN GOAL  Increase reliability while increasing the speed at the lowest cost

 Data is inevitably scattered over disk pages in real world.  To decrese the speed, we have to minimize the number of access by

 clustering (temporal & locality) as far as possible  Getting e/t you need at one acess.  The physical characteristics of the hardware together with data

structures and the algorithms are used to predict the efficiency of file operations.  Now we will study physical characteristics of the hardware...

5

Storage Hierarchy Machine instructions

Cost/unit increase

Cache

1M, 15 cycle

(~40 times expensive)

Main Memory 1G-16GB, 200 cycle (10 nanosec.) volatile media

Capacity decrease

100.000 times slower

USB Flash Storage

Reliability increase

Non-volatile media

Secondary Storage Non-volatile media

(like 10sec vs. 10days) 80G, 10 milisec.

CD-RW DVD-RW Floppy Disk Serial access

Magnetic Tape storage

6

technologies used in storage  Cache memory: «static RAM» (SRAM)  Primary memory: «dynamic RAM» (DRAM)  Secondary memory:  HDD (Hard Disk Drive): magnetically store and R/W by magnetic arm

 SSD (Solid state Disk): electronic disk

 CD_ROM (Compact Disk-Read Only Memory): store optically, read by laser  WORM (Write Once Read Many)

 DVD (Digital Video Disk): store optically, read by laser. (smaller wavelength and laser type)

 BD (Blue-Ray Disk): uses blue-ray technology

7

technologies used in storage  USB Flash memory: EEPROM (NAND-type)  100 times faster than HDD, 100 times expensive than HDD  Wears out! «Wear-level» technique to lessen the rewrite limit problem.

 Magnetic Tapes:  cheap storage for archieval purposes  Only Sequential acces  Charactericstics of «9-track tape»  Tape density: bpi = Bpi (6250 ~ 30.000)  Tape speed: ips (30 ~ 200)  Gap size: 0,3 ~ 0,75 inch (i) frame

track

«9-track tape» Data block

8 Gap (R/W durma/kalkma için bırakılan alan)

Tape length  How much tape is needed to store 1 million 100-B records if we have tape

with 6250 bpi and 0,3 inch-size gaps?  g: length of gap size  n: # of data blocks  b: length of data block

s= n * (b+g) Blocking factor (bf) = # of records / block bf=1b=100/6250 Bpi = 0,016 i  s= 1 million * (0,3+0,016) =26.333 feet bf=50b=100*50/6250 Bpi = 0,8 i  s= (1 million/50) * (0,3+0,8) = 1833 feet Effective recording density (erd) =«#of Bytes /data_block» /length required to store data block

bf=1 100 /0,316 = 316,4 bpi bf=50 100*50 / 1,1 = 4545,45 bpi 9

Tape data trasmission rate  Nominal trans. rate = Tape density (bpi) * Tape speed (ips)  Effective trans. rate = effective recording density (erd) * Tape

speed (ips)  If we have 200 ips tape, determine the nominal and effective trans rates? Nominal trans. rate= 6250 Bpi * 200 ips= 1250 KBps bfr = 1  effective trans. rate = 316,4 Bpi * 200 = 63,3 KBps bfr = 50  effective trans. rate = 4545,45 bpi * 200 = 909 KBps

10

Basic Organization of DISK (hard or floopy):

Since there is only 1 datapath to the computer, only 1 read/write head can be active at a time.

11

Disk Organizations: Sector,Cluster  It has to do with abstraction. Improves disk access by decreasing ts.  Sector – the fix-length, smallest addressable portion of a disk.

Typically 512 bytes- 4K.

 Cylinder-Head-Sector addressing (physical CHS addressing) to access the

sector, then bring it into memory (buffer)  OS does not use CHS addressing, rather it uses LBS (logical block addresing) which orders all sectors from 0 to ‘the last sector’. If needed, a firmware (bios) on disk converts LBS adress to CHS address  Cluster is the smallest unit of space that can be allocated to a file by

OS. OS views the file as a series of clusters. A cluster has a fix number of contiguous sectors.  FAT in File Manager (a module in OS) ties physical sectors to the logical clusters by using FAT. (A) track (B) geometrical sector (C) track sector (D) cluster 12

Disk performance metrics-1  Note: B = byte

1KB=1024B (210B), 1MB=1024KB (220B), 1GB=1024MB (230B)  Capacity= C seek time (ts), rotation speed (tr), transfer rate (tt)  C = (# of platters) * (# of tracks/platter) * (# of B/track) example: 80GB,160GB ts = the time it takes for the actuator to move the disk head from its current location to requested track, ex: tsmin=0, tsmax=15-20msec, tsave.= (1/3)*tsmax= 5ms. The slowest part of total cost. tr = the time spent to move the head over the requested sector. Ex: 10000rpm  6msec is the full rotation time. in average ~1/2 of

full rotation, tt = (# of bytes transfered / # of bytes on a track) * rotation time the speed at which bytes pass by the disk head, to be transfered to/from memory. TOTAL ACCESS TIME= ts+tr+tt  transfer_rate: B/msec (Sample value= 100MB/sec) 13

Example1 10.000 rotateperminute disk Bytes / sector= 512 Sectors / track = 170 Tracks / cylinder = 16 Ave. Seek time= 8 msec Ave. Rotational delay= 3 msec transfer rate= 1/6 * (512*170)= 14500 bytes / msec Transfer time for a single sector? = (6/170)msec

Example 2: in real disk we transfer at least a sector, why? 10.000rpm tsave= 5 msec Transfer rate=83MB/sec Transfer time for 1B= 1/83MB = 0,000012 msec Transfer time for 1MB = 0,012 msec Average access to 1 B= 5 + 3 + 0,000012 = 8,000012 msec Average access to 1 MB = 5 + 3 +0,012 msec= 8,012 msec That is why we transfer a sector for each access !... 14

Internal / external fragmentation  fragmentation means that something is broken

into parts that are detached, isolated or incomplete.  May occur at al level of organization. Sector, block, cluster, file...  Since the sector size is fix, usually there is no convenient fit between records and sectors. This leads to “internal fragmentation” within the sectors”. Think similiar unused holes at different levels..  Sector spaning is a simple solution for this problem. Disadvantage?

aaaaa

aa---

bbbbb

aaaaa

aabbb

bbccc

ccc--

 Disk may have lots of small-sized chunks of unallocated blocks, but no large

chunks. Thus it may not be possible to allocate space for a large file, even though disk has plenty of free space. This is called “external fragmentation”. 15

Block-level interface(block,page )  Block is a sequence of bytes.  Adv.:  OS hides hardware details (like different sector sizes, different addressing) by block-level interface. OS maintains mapping b/w blocks and sectors.  Blocking increase througput (successful data transfer rate).  A Block size is at least 1 sector-size and determined by OS.. OS views the disk as a series of blocks.   



Block numbers start from 0. BF = # of records stored in each block. While Page is a block-sized area allocated in main memory. OS provides several methods to acces disk blocks.  readblock(n,p): read data from block-n to page-p  writeblock(n,p): write data in page-p to block-n.  allocate(k,n): find k contigious available blocks as close to block-n as possible.  deallocate(k,n): deallocate k contigious blocks as close to block-n as possible. OS keeps track of which blocks are available for allocation. 2 basic strategies exists:  Disk map  Free list

16

Block size:  Block contention increase with larger blocks. Thus, OLAP applications/web

search applications, which has higher random access prefers small-sized block.  Desicion support/data warehouse applciations, which has higher sequential access prefers large-sized block.

17

In terms of

Prefered size

Application

Block contention

Small

OLAP

Random row access speed

Small

OLAP

Sequential row access speed

Large

Desicion support Data warehouse

Disk performance metrics-2 (Additional metrics)  Block transfer time (btt)= (B/ transfer_rate)  bulk_transfer_rate (btr) = rate of transfering useful

bytes in the blocks (kullanışlı veri transfer hızı) btr = B/(B+G) * transfer_rate (G is gap size) Bulk time to transfer of k consecutive blocks on the same cylinder? ts + tr + (k* (B/bulk_transfer_rate))

18

File-level interface  Client views file a sequence of bytes. (No notion of block,

sector..) . Client can directly access to a byte in the file..  OS hides the details from client. For example, in the below code, blocks are accessed through pages. I/O buffers are allocated.... How many disk access requires for f.read() and f.write()? Look at “journey of a byte” at the following slides

 f.seek() method performs 2 conversions:  specified byte position  logical block reference (simple)  Logical block reference  physical block reference

(depends on file system implementation) 19

File implementation strategies Contiguous Allocation: each file as a sequence of contiguous blocks. Simplest strategy. Both internal and external fragmentation Extent-based allocation: similiar to contiguous allocation. Reduces internal/external frag. by storing a file a sequence of fix-length extents. File is extended 1 extent at a time Indexed allocation: extend file 1 block at a time. Least possible amount of fragmentation. Keeps track of allocated blocks of the file with a special index block. We need multiple level of indexing for large files Ex: UNIX file system

20

File implementation strategies

Extent-size: 8 blocks

 Junk’ s 21th logical block  53.block (clustering)  701.block (extent)

 Junk’ s 2 th logical block  16.block (indexing) 21

Example #3 (effect of sector spanning ) 512 bytes per sector 63 sectors per track 16 tracks per cylinder 4092 cylinders  Disk capacity? 512*63*16*4092  We have a file with 50.000 fixed-length records. How many cylinders does the file requires if each data record requires 240 byte? In case of sector spaning: Cylinder capacity = 512*63*16 = 516,096 bytes File size = 50,000*240 = 12,000,000 bytes Number of cylinders required = 12,000,000/516,096 = 23.25 In case of Internal fragmentation: (no sector spanning) File requires 25.000 sectors 63*16 = 1008 sectors per cylinder 25,000/1008 = 24.8 cylinders required. Analysis:  Sector spanning has an adv., because it requires less space for the file. On the other, some records can be retrieved by accesing two sectors.This is the disadvantage.  Observe fragmentation problem at different levels…(like cluster..)

22

Example #4 ( file access performance) 10.000 rotateperminute disk, 512 B/sector, 170 Sectors / track, 16 Tracks / cylinder, tsave = 8 msec. Allocate a file having 34.000 records, each of which is 256-B-length. a.) with cluster size of 4 KB. b.) with a cluster size of 1track.

34.000*256B

8704 KB file

Track-based access 100 tracks (not contigious)

cluster-based access 256-byte

34.000 records

. .

Tracksize:170*512 byte

Cluster size=4096 B This is 4096 / (170*512) of track There are 2125 clusters

Track-based sequential access=

(8msec+3msec+6 msec)*100=1.7sec

cluster-based random access= (8msec+3msec+(1/21.25) *6msec)*2125

=23,97sec 23

Example #5 (random sector access time) # of platters: 4 8192 track/platter-surface 256 sectors / track 512 bytes / sector

disk çapı: 3.5 inches (1 inche: 2,54cm) Gaps take %10 of the track space. Rpm: 3840 The head takes 1 msec for every 500 cylinder plus 1 msec for start/stop.

 What is the best, worst and average random sector I/O=?

min. = 0.05 msec max. = 33.05 msec ave. = 14.65 msec  If a block is 4096 bytes, what is the best, worst and average random I/O=? min. I/O = transfer time = 0.5 msec

24

Example#6 (file access performance) # of platters: 4 8192 track/platter-surface 256 sectors / track 512 bytes / sector





disk çapı: 3.5 inches (1 inche: 2,54cm) Gaps take %10 of the track space. Rpm: 3840 The head takes 1 msec for every 500 cylinder plus 1 msec for start/stop.

How much does it take to read 1 MB of data which is all stored in consecutive tracks? Answer: 138.8 msec If all sectors are scattered on the disk  30.003,2 msec ( ½ minutes) How much does it take to read 1000 MB of data which is all stored in consecutive cylinders? Answer: 126.005,8 ( 2 minutes and 6 sec) If all sectors are scattered on the disk  30.003.200 msec ( 8 hours +20 minutes)

25

Example #6-devam (effect of disk access algorithms) Requested cylinder

Arraival time

Complete time (by fifo algoritm)

Complete time

1000

0

7.85

7.85

(1.)

3000 7000 2000 8000 5000

0 0 20 30 40

20.7 37.55 56.6 77.45 92.3

20.7 37.55 77.5 48.4 63.25

(2.) (3.) (6.) (4.) (5.)

(by elevator algoritm)

26

A journey of Byte Write(append) the 1-B value in «ch» in program to textfile

write (textfile, ch, 1) A system call to OS

File Manager handles the request logical

FM access the information of the textfile i.e about the physical location(cylinder,track..) of the file.. FM uses FAT to locate the location of sector that is to contain the byte FM finds an available I/O buffer space then read the “sector” from disk into the system buffer in MM.

physical

Then write ‘ch’ into the appropriate place in the sector in MM. FM give instruction to I/O processor where the byte is stored in the MM and where it need to be sent in the disk. I/O processor check if the drive is available and also buffers the chunks of proper size of disk.

I/O processor sends data to disk cotroller controller instructs the drive to move the arm to the proper track and wait until the proper sector come under the arm and then sends the sector bit-by-bit.

Do not send the sector immediately to disk.Why? In which case do we need to send immediately? 27

I/O Processor / direct memory acces controller

 I/O processor: handles the task of communicating disk, process

independently from the main cpu.  I/O processor (a special purpose device) take the commands from OS and communicates with disk controller.  Once the buffer is full, I/O processor send the sector’s bytes, one at a time, as soon as the controller is available. User prog. char «c» in data area

1) 28

File Mgr. in OS: char «c» in system buffer

I/O processor/ DMA controller

Disk controller

DISK 2) 1-) Move mode / locate mode: to eliminate data transfer OH. 2-) Scatter input / gather output (vectored I/O): to eliminate 2step process to scatter OH and useful data of block.

Disk Controller  DISK CONTROLLER CONTROLS THE DISK while hiding the details of disk access.  Disk controller is an interface b/w computer and disk-drive. Transfers R-W request/from disk,

controlling disk arm, provides reliability by applying checksums to sectors, remap the bad sectors.  Disk controller moves the head to the correct position, correct track, correct sector for reading and writing.  ATA(advanced tech. attachment)=IDE(integrated drive electronics) EX: 133 MB/s with ATA/133, 150 MB/S with SATA-serialATA, ATAPI  2 IDE port on PC, each port can atmost access 2 disk, one master and the other is slave.  SCSI (small computer system interface) a system bus standart coordinating many type of devices on a single bus. Provides a basement for RAID disks. Ex:max 16 devices in ULtra 320 SCSI.

IDE

SCSI

Cost

Cheap

Expensive

#of devices

2

16

Maintainance

Easy

Hard

Usage

At home

Business

Speed

133MB/s

320 MB/s 29

Disk Bottleneck, Improving Disk Access Time  “CPU rate (high performance network) is dramatically higher

than Disk I/O transfer rate” This causes disk bottleneck.  Solution: (read 3.1.8)  Multiprogramming (cpu works on other jobs while waiting for the data to arrive)

 Cylinders (2 tracks nearby ? at the same cylinders)  Disk cache  Parallelism (example: Disk Striping, RAID): This helps to achieve better reliability as well.

 Efficient use of RAM(buffering)

30

Disk Cache  Disk cache – A kind of buffering! Block of memory set aside to

contain blocks of data from disk. Disk cache is bundled with disk drive.  Improves performance.  When data is requested from secondary storage, the file manager looks into the disk cache to see if it contains the requested data.  Compare the following access times: transfer a sector= ts +1/2 tr + sector rotation time transfer a track = ts + tr

Which one do you prefer? Transfer sector or transfer track?  The real value of disk cache is prefetching.

31

Disk Striping  Two 20GB drives are always faster than a single 40GB drive. Because

simultaneous access to 2 sectors. 2 problems arises:  Cost increases  Load balancing is required for disks to be working as uniformly as possible.

 Using Two 20GB drives is efficient if both can be always kept busy.  To increase efficiency, we have to balance the workload among the multiple

disks..For balancing workload, Disk Striping can be used.  Disk striping uses the disk controller to hide the smaller disks from the OS, giving it the illusion of large single disk.



32

Striping distributes database among small drives equally.

Disk reliability, improving disk reliability  2 reasons to decrase reliability: Magnetic material can degenerate Head crash  2 approaches to increase reliability. These tasks are governed by disk controller again. Mirroring Storing Parity (use a single disk to back up any # of other disks)

33

High speed and robust Problem: •Cost is high because High number of disks..

2 problems: •Note that thare are 4 disk accesess for a single sector write. •More vulnerable to non-recoverable multi-disk failure

RAID

 In high speed networks, Storage area Network (SAN) provides RAID

(redundant array of independent disks) organization. RAID supports large data, provides reliability, resource sharing, performance improvement, disk striping.  RAID-0 : only striping, no guard against disk failure RAID-1: mirrored striping RAID-2: byte striping, error-correcting codes instead of parity (difficult to implement, no longer used) RAID-3: byte striping and 1 parity disk RAID-4: sector striping and 1 parity disk. RAID-5: similiar to RAID-4, but parity information is distributed among disks.(Nth sector of each disk stores parity info.) RAID-6: similiar to RAID-5, store 2 kinds of parity info., thus needs another disk for additional parity info.

34

Buffer Management  Working with large chunks of data in MM so that  Read data in memory multiple times (caching)

 the # of access to disk is reduced..

 Sysem buffer vs. user buffer

 Buffer manegement by OS: organizing >2 buffers including system buffers..  coordination uses some techniques such as Least Recently Used, FIFO, clock-replacement algorithm..  How many buffers do we need?

1: Even if we ( the program) transmit data in only one direction, 1 buffer causes problems like I/O bound processing..(CPU wants to be filling the buffer at the same time that I/O is being performed= Enabling I/O-CPU overlapping ONLY by using at least 2 buffers!! Fig.3.22 )

2: At least 2 (one for input, the other for output): still similiar problems occur.

 Solution : Apply Multiple buffering strategy :Tradeoff: (as cost of memory decrease using

many buffers is possible) the more buffers there are  but the more complex management transfer the «buffer» to the disk with 1 is required. access when either

Buffer pool with 4 buffers (pages) 1 buffer for each page 35

•The page is being replaced •File is closed • For data integrity purposes (Recovery management)