Adit Cookbook Pages

What is a virtual array and why would you want one?

Those who remember early Fortran and Basic programming on Digital PDP computers and the like will know very well what they are. They were numeric or string arrays that your code treated as normal “in memory” arrays but which the system runtime stored on a disk. You could even treat them as persistent storage holding data between sessions. Most programmers of that time implemented simple indexed file structures using virtual arrays as the index structure as you could use a “binary chop” to search the array for a given value - but I am getting ahead of myself.

Why might you need one? Well sometimes you need a realy big fat array to process some stuff but your program starts to choke on it - your system starts paging like crazy and process times get silly. If you knew this was going to happen at the start you would have designed things differently but now you have loads of code written that works with one (or more) array. The solution is simple, make your array virtual by placing the actual data on a local hard drive. The program structure remains almost the same and response times will pick up dramatically. You might think that OS paging would be the optimal solution but it is not - you can even beat Windows 2000 without any trouble at all.

A practical implementation may well need to include a sort (see our page on sorting and in particular sorting disk based files) as your incoming data may not be suitably ordered. You can also optimise a search through an array by using a “binary chop”.

OK this is a long preamble to a simple process - we substitute the array name with a function call and add two subs to create and then delete the temporary file we are using to store the values on your local hard drive. Using a local hard drive is usually preferable to a server disk as it is likely to be much faster than the network response. This process should adapt to any programming language but there is a slicker Visual Basic approach that takes advantage of the fact that VB forms are classes. If you are working in Visual Basic 6 then click just here!

The code below assumes that your array is an array of types and that the type is called myType. It is also assumed that the array is zero bound. I have used a LEN() function to get the size of the type and this should make the point that if the type contains strings they should be of a fixed length.

Public VirtFile as String
Public VFileNumber as Long

Public Function myVirtArray(VElement as Long, VAction as Integer, Optional VData as myType) as MyType
   Select case VAction
       Case 1 ‘Set the Array Element to the value passed in VData
            Put #VFileNumber, (VElement + 1), VData
           myVirtarray = vData
       case 2 ‘Retrieve the specified Array Element
           Get #VFileNumber, (VElement +1), myVirtArray
   End Select
End Function

Public Sub MakeVirtArray()
   Dim Reclen as Long

   recLen = Len(myType)
   VirtFile = GetTempFile()
   VFileNumber = FreeFile
   Open VirtFile For Random As VFileNumber Len = RecLen
End Sub

Public Sub KillVirtArray()
   Kill VirtFile
End Sub

Public Function GetTempFile() As String
   Dim PathString As String, FileName As String
   Dim RetLength As Long, RetValue As Long
  
   On Error Resume Next
   PathString = Space$(260)
   FileName = Space$(340)

   RetLength = GetTempPath(260, PathString)
   If RetLength > 0 Then
       'the returned value is the length of the Path to the temporary directory
       'not including the null terminating character
       PathString = Left$(PathString, RetLength)
       RetValue = GetTempFileName(PathString, "tmp", 0, FileName)
       If RetValue <> 0 Then
           RetLength = InStr(FileName, vbNullChar) - 1
           FileName = Left$(FileName, RetLength)
       Else
           FileName = ""
       End If
   Else
       FileName = ""
   End If
   GetTempFile = FileName
End Function

Implementing the code

You will need to search through your code locating all of the current lines addressing your array and substitute the relevant call to the myVirtArray() function. If you have taken a glance at the preferred VB solution then you will see how simple the changes required could be if your language supports similar functionality to the VB6 Class Properties.

Dealing with errors

It would be a good idea to do some sort of bounds checking for your virtual array in the myVirtArray() function. You could thus avoid raising errors associated with disk file access in parts of the code that are unaware that there is any disk activity involved in the process.