May 18, 2014

Prepending to a file.

Prepending to a file is relatively expensive, but to what extent? I decided to check. I wrote 2^23 integers to a file, in ASCII, separated by a newline. On my ThinkPad with spinning rust, it takes a little longer than 12 seconds. The file size is around 31 MB.

In order to “prepend” an integer to the beginning, we essentially have to shift all of the other elements forward and then write what we need at the beginning. So we write everything again, and then some. I wrote a small Go program that does this, and here are the results:

  āˆ‚ [-]: go run middle_insert.go 
Time to insert 4194304 integers: 12.202411374s
Time to prepend an int to the file: 579.472752ms

Prepending is 21 times faster! Dat cache.

A few notes worth mentioning:

package main

import (

const N = 1 << 22

func exitOnError(err error) {
	if err != nil {

func main() {
	f, err := os.Create("scratch.txt")
	defer f.Close()

	start := time.Now()
	// Write a bunch of ints
	for i := 1; i <= N; i++ {
		fmt.Fprintln(f, i)
	fmt.Println("Time to insert", N, "integers:", time.Now().Sub(start))

	fStat, err := f.Stat()

	toBeInserted := fmt.Sprintln(0)

	start = time.Now()
	os.Truncate(f.Name(), fStat.Size()+int64(len(toBeInserted)))

	// Mmap! Let the OS do the work.
	sl, err := syscall.Mmap(int(f.Fd()), 0, int(fStat.Size())+len(toBeInserted),
		syscall.PROT_READ|syscall.PROT_WRITE, syscall.MAP_SHARED)

	// This feels like cheating :-P
	copy(sl, append([]byte(toBeInserted), sl...))
	fmt.Println("Time to prepend an int to the file:", time.Now().Sub(start))
Next read these:
Nov 23, 2023
Jan 11, 2023
Mar 5, 2015