Skip to content
Permalink

Comparing changes

Choose two branches to see what’s changed or to start a new pull request. If you need to, you can also or learn more about diff comparisons.

Open a pull request

Create a new pull request by comparing changes across two branches. If you need to, you can also . Learn more about diff comparisons here.
base repository: diskfs/go-diskfs
Failed to load repositories. Confirm that selected base ref is valid, then try again.
Loading
base: v1.5.1
Choose a base ref
...
head repository: diskfs/go-diskfs
Failed to load repositories. Confirm that selected head ref is valid, then try again.
Loading
compare: v1.5.2
Choose a head ref
  • 5 commits
  • 19 files changed
  • 2 contributors

Commits on Mar 3, 2025

  1. extract byte diff comparison test utility into common testhelper (#283)

    Signed-off-by: Avi Deitcher <avi@deitcher.net>
    deitch authored Mar 3, 2025
    Copy the full SHA
    9238e8f View commit details

Commits on Mar 6, 2025

  1. squashfs: fix directory inode size (#286)

    Due to SquashFS spec
    https://dr-emann.github.io/squashfs/squashfs.html#_directory_inodes
    Section 5.2, it says about directory inode file size
    
    '''
    	This value is 3 bytes larger than the real listing. The Linux
    kernel creates "." and ".." entries for offsets 0 and 1, and only after
    3 looks into the listing, subtracting 3 from the size.
    '''
    
    In go-disks the size is calculated by serialize the 'directory'. A
    directory serialization includes multiple 'directoryHeader's and
    'directoryEntryRaw's. This aligns with the dir_size calculation logic
    with well-known squasjfs-tools project
    
    https://github.com/plougher/squashfs-tools/blob/master/squashfs-tools/mksquashfs.c#L1549
    
    Note that the file_size is set to 3 larger than the size.
    
    Thus here, we set directoryLocation.size to 3 bytes larger and the new
    value will be used to set basicDirectory.fileSize.
    
    Fixes #285
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    Xynnn007 authored Mar 6, 2025
    Copy the full SHA
    6693615 View commit details

Commits on Mar 10, 2025

  1. SquashFS | Fix some bugs (#289)

    * squashfs: add support for Disk.CreateFilesystem API
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * filesystem: add Close() method for cleaning jobs
    
    When building filesystems, we usually have some temp files or
    directories. The `Close()` method will help to do clean jobs when we
    destroy the FileSystem struct.
    
    Fixes #288
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * squashfs: fix example disk block size
    
    squashfs requires block size must be >= 4KB.
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    ---------
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    Xynnn007 authored Mar 10, 2025
    Copy the full SHA
    173d3fa View commit details
  2. Squashfs: fix error format when file larger than block size (#287)

    * squashfs: fix blocks start field in FileInode
    
    Due to SquashFS, the first field of both File inode and extended file
    inode is called 'blocks start', that means
    
    ```
    The offset from the start of the archive to the first data block.
    ```
    
    Link https://dr-emann.github.io/squashfs/squashfs.html#_file_inodes
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * squashfs: fix fragment block location
    
    The granularity of fragment data writing is block, so we don't need to
    return an unaligned fragment amount but 'sized chunks'
    
    everytime we write a fragmentdata block, the variable location should be
    added with a size of 'blocksize', which means that we have written a new
    fragment block.
    
    The variable `location` will be reused to tell new accumulated
    fragmentblocks position.
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * squashfs: clear cumulative variable for fragment data
    
    The variable fragmentData is used to record if the current fragment data
    is enough to occupy a block. Once a block is written, it should be
    cleared for a new round of loop.
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * squashfs: set file inode's frag index when no fragment data
    
    Due to spec, frag index of file inode
    
    https://dr-emann.github.io/squashfs/squashfs.html#_file_inodes
    
    ```
    An index into the Fragment Table which describes the fragment block
    that the tail end of this file is stored in. If not used, this is set
    to 0xFFFFFFFF.
    ```
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    * lint: fix go fmt lint error
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    
    ---------
    
    Signed-off-by: Xynnn007 <xynnn@linux.alibaba.com>
    Xynnn007 authored Mar 10, 2025
    Copy the full SHA
    6e6c2a9 View commit details
  3. test for squashfs writing and reading errors (#284)

    * test for squashfs writing and reading errors
    
    Signed-off-by: Avi Deitcher <avi@deitcher.net>
    
    * lint fixes
    
    Signed-off-by: Avi Deitcher <avi@deitcher.net>
    
    ---------
    
    Signed-off-by: Avi Deitcher <avi@deitcher.net>
    deitch authored Mar 10, 2025
    Copy the full SHA
    414258f View commit details
2 changes: 1 addition & 1 deletion disk/disk.go
Original file line number Diff line number Diff line change
@@ -177,7 +177,7 @@ func (d *Disk) CreateFilesystem(spec FilesystemSpec) (filesystem.FileSystem, err
case filesystem.TypeExt4:
return ext4.Create(d.Backend, size, start, d.LogicalBlocksize, nil)
case filesystem.TypeSquashfs:
return nil, filesystem.ErrReadonlyFilesystem
return squashfs.Create(d.Backend, size, start, d.LogicalBlocksize)
default:
return nil, errors.New("unknown filesystem type requested")
}
7 changes: 7 additions & 0 deletions examples/iso_create.go
Original file line number Diff line number Diff line change
@@ -25,7 +25,14 @@ func CreateIso(diskImg string) {
fspec := disk.FilesystemSpec{Partition: 0, FSType: filesystem.TypeISO9660, VolumeLabel: "label"}
fs, err := mydisk.CreateFilesystem(fspec)
check(err)
defer func() {
if err := fs.Close(); err != nil {
check(err)
}
}()

rw, err := fs.OpenFile("demo.txt", os.O_CREATE|os.O_RDWR)
check(err)
content := []byte("demo")
_, err = rw.Write(content)
check(err)
8 changes: 7 additions & 1 deletion examples/squashfs_create.go
Original file line number Diff line number Diff line change
@@ -16,13 +16,19 @@ func CreateSquashfs(diskImg string) {
log.Fatal("must have a valid path for diskImg")
}
var diskSize int64 = 10 * 1024 * 1024 // 10 MB
mydisk, err := diskfs.Create(diskImg, diskSize, diskfs.SectorSizeDefault)
mydisk, err := diskfs.Create(diskImg, diskSize, diskfs.SectorSize4k)
check(err)

fspec := disk.FilesystemSpec{Partition: 0, FSType: filesystem.TypeSquashfs, VolumeLabel: "label"}
fs, err := mydisk.CreateFilesystem(fspec)
check(err)
defer func() {
if err := fs.Close(); err != nil {
check(err)
}
}()
rw, err := fs.OpenFile("demo.txt", os.O_CREATE|os.O_RDWR)
check(err)
content := []byte("demo")
_, err = rw.Write(content)
check(err)
4 changes: 3 additions & 1 deletion filesystem/ext4/directory_test.go
Original file line number Diff line number Diff line change
@@ -2,6 +2,8 @@ package ext4

import (
"testing"

"github.com/diskfs/go-diskfs/testhelper"
)

func TestDirectoryToBytes(t *testing.T) {
@@ -17,7 +19,7 @@ func TestDirectoryToBytes(t *testing.T) {
b := dir.toBytes(bytesPerBlock, directoryChecksumAppender(sb.checksumSeed, 2, 0))

// read the bytes from the disk
diff, diffString := dumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
diff, diffString := testhelper.DumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
if diff {
t.Errorf("directory.toBytes() mismatched, actual then expected\n%s", diffString)
}
5 changes: 5 additions & 0 deletions filesystem/ext4/ext4.go
Original file line number Diff line number Diff line change
@@ -693,6 +693,11 @@ func Read(b backend.Storage, size, start, sectorsize int64) (*FileSystem, error)
// interface guard
var _ filesystem.FileSystem = (*FileSystem)(nil)

// Do cleaning job for ext4. Note that ext4 does not have side-effects so we do not do anything.
func (fs *FileSystem) Close() error {
return nil
}

// Type returns the type code for the filesystem. Always returns filesystem.TypeExt4
func (fs *FileSystem) Type() filesystem.Type {
return filesystem.TypeExt4
5 changes: 3 additions & 2 deletions filesystem/ext4/groupdescriptors_test.go
Original file line number Diff line number Diff line change
@@ -5,6 +5,7 @@ import (
"os"
"testing"

"github.com/diskfs/go-diskfs/testhelper"
"github.com/go-test/deep"
)

@@ -60,7 +61,7 @@ func TestGroupDescriptorToBytes(t *testing.T) {
}
b := gd.toBytes(sb.gdtChecksumType(), sb.checksumSeed)
expected = expected[:64]
diff, diffString := dumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
diff, diffString := testhelper.DumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
if diff {
t.Errorf("groupdescriptor.toBytes() mismatched, actual then expected\n%s", diffString)
}
@@ -94,7 +95,7 @@ func TestGroupDescriptorsToBytes(t *testing.T) {
descriptors: groupdescriptors,
}
b := gds.toBytes(sb.gdtChecksumType(), sb.checksumSeed)
diff, diffString := dumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
diff, diffString := testhelper.DumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
if diff {
t.Errorf("groupDescriptors.toBytes() mismatched, actual then expected\n%s", diffString)
}
3 changes: 2 additions & 1 deletion filesystem/ext4/superblock_test.go
Original file line number Diff line number Diff line change
@@ -4,6 +4,7 @@ import (
"reflect"
"testing"

"github.com/diskfs/go-diskfs/testhelper"
"github.com/go-test/deep"
)

@@ -32,7 +33,7 @@ func TestSuperblockToBytes(t *testing.T) {
if err != nil {
t.Fatalf("Failed to serialize superblock: %v", err)
}
diff, diffString := dumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
diff, diffString := testhelper.DumpByteSlicesWithDiffs(b, expected, 32, false, true, true)
if diff {
t.Errorf("superblock.toBytes() mismatched, actual then expected\n%s", diffString)
}
135 changes: 0 additions & 135 deletions filesystem/ext4/util_test.go
Original file line number Diff line number Diff line change
@@ -2,7 +2,6 @@ package ext4

import (
"bytes"
"fmt"
"testing"
)

@@ -64,137 +63,3 @@ func TestMinString(t *testing.T) {
})
}
}

// dumpByteSlice dump a byte slice in hex and optionally ASCII format.
// Optionally but position at the beginning of each row, like xxd.
// Optionally convert to ASCII at end of each row, like xxd.
// Can show positions at beginning of each row in hex, decimal or both.
// Can filter out all rows except those containing given positions in showOnlyBytes. If showOnlyBytes is nil, all rows are shown.
// If showOnlyBytes is not nil, even an empty slice, will only show those rows that contain the given positions.
func dumpByteSlice(b []byte, bytesPerRow int, showASCII, showPosHex, showPosDec bool, showOnlyBytes []int) (out string) {
var ascii []byte
// go through each byte.
// At each position:
// - if we are at the end of a row, print the ASCII representation of the row.
// - if we are at the middle of a row, add an extra space
// - if we are still in the byte slice, print the byte in hex with a space before it.
// - if we are past the end of the row, print spaces.
showOnlyMap := make(map[int]bool)
for _, v := range showOnlyBytes {
showOnlyMap[v] = true
}
// run by rows
numRows := len(b) / bytesPerRow
if len(b)%bytesPerRow != 0 {
numRows++
}
for i := 0; i < numRows; i++ {
firstByte := i * bytesPerRow
lastByte := firstByte + bytesPerRow
var row string
// row header includes optional position numbers
if showPosHex {
row += fmt.Sprintf("%08x ", firstByte)
}
if showPosDec {
row += fmt.Sprintf("%4d ", firstByte)
}
row += ": "
for j := firstByte; j < lastByte; j++ {
// every 8 bytes add extra spacing to make it easier to read
if j%8 == 0 {
row += " "
}
// regular byte, print in hex
if j < len(b) {
hex := fmt.Sprintf(" %02x", b[j])
if showOnlyBytes != nil && showOnlyMap[j] {
hex = "\033[1m\033[31m" + hex + "\033[0m"
}
row += hex
} else {
row += " "
}
switch {
case j >= len(b):
// past end of byte slice, print spaces
ascii = append(ascii, ' ')
case b[j] < 32 || b[j] > 126:
// unprintable characters, print a dot
ascii = append(ascii, '.')
default:
// printable characters, print the character
ascii = append(ascii, b[j])
}
}
// end of row, print the ASCII representation and a newline
if showASCII {
row += fmt.Sprintf(" %s", string(ascii))
ascii = ascii[:0]
}
row += "\n"

// calculate if we should include this row
var includeRow = true
if showOnlyBytes != nil {
includeRow = false
for j := firstByte; j < lastByte; j++ {
if showOnlyMap[j] {
includeRow = true
break
}
}
}
if includeRow {
out += row
}
}
return out
}

// diff
type diff struct {
Offset int
ByteA byte
ByteB byte
}

// compareByteSlices compares two byte slices position by position. If the byte slices are identical, diffs is length 0,
// otherwise it contains the positions of the differences.
func compareByteSlices(a, b []byte) (diffs []diff) {
maxSize := len(a)
if len(b) > maxSize {
maxSize = len(b)
}
for i := 0; i < maxSize; i++ {
switch {
case i >= len(a):
diffs = append(diffs, diff{Offset: i, ByteA: 0, ByteB: b[i]})
case i >= len(b):
diffs = append(diffs, diff{Offset: i, ByteA: a[i], ByteB: 0})
case a[i] != b[i]:
diffs = append(diffs, diff{Offset: i, ByteA: a[i], ByteB: b[i]})
}
}
return diffs
}

// dumpByteSlicesWithDiffs show two byte slices in hex and ASCII format, with differences highlighted.
//
//nolint:unparam // sure, bytesPerRow always is 32, but it could be something else
func dumpByteSlicesWithDiffs(a, b []byte, bytesPerRow int, showASCII, showPosHex, showPosDec bool) (different bool, out string) {
diffs := compareByteSlices(a, b)
// if there are no differences, just return an empty string
if len(diffs) == 0 {
return false, ""
}

showOnlyBytes := make([]int, len(diffs))
for i, d := range diffs {
showOnlyBytes[i] = d.Offset
}
out = dumpByteSlice(a, bytesPerRow, showASCII, showPosHex, showPosDec, showOnlyBytes)
out += "\n"
out += dumpByteSlice(b, bytesPerRow, showASCII, showPosHex, showPosDec, showOnlyBytes)
return true, out
}
5 changes: 5 additions & 0 deletions filesystem/fat32/fat32.go
Original file line number Diff line number Diff line change
@@ -489,6 +489,11 @@ func (fs *FileSystem) writeFat() error {
// interface guard
var _ filesystem.FileSystem = (*FileSystem)(nil)

// Do cleaning job for fat32. Note that fat32 does not have side-effects so we do not do anything.
func (fs *FileSystem) Close() error {
return nil
}

// Type returns the type code for the filesystem. Always returns filesystem.TypeFat32
func (fs *FileSystem) Type() filesystem.Type {
return filesystem.TypeFat32
2 changes: 2 additions & 0 deletions filesystem/filesystem.go
Original file line number Diff line number Diff line change
@@ -46,6 +46,8 @@ type FileSystem interface {
// SetLabel changes the label on the writable filesystem. Different file system may hav different
// length constraints.
SetLabel(label string) error
// Close will cleanup the temporary files created by the filesystem generation steps
Close() error
}

// Type represents the type of disk this is
8 changes: 8 additions & 0 deletions filesystem/iso9660/iso9660.go
Original file line number Diff line number Diff line change
@@ -289,6 +289,14 @@ func Read(b backend.Storage, size, start, blocksize int64) (*FileSystem, error)
// interface guard
var _ filesystem.FileSystem = (*FileSystem)(nil)

// Delete the temporary directory created during the iso9660 image creation
func (fsm *FileSystem) Close() error {
if fsm.workspace != "" {
return os.RemoveAll(fsm.workspace)
}
return nil
}

// Type returns the type code for the filesystem. Always returns filesystem.TypeFat32
func (fsm *FileSystem) Type() filesystem.Type {
return filesystem.TypeISO9660
8 changes: 4 additions & 4 deletions filesystem/squashfs/const_internal_test.go
Original file line number Diff line number Diff line change
@@ -36,7 +36,7 @@ func testGetFirstInodeHeader() *inodeHeader {
}
func testGetFirstInodeBody() inodeBody {
return extendedFile{
startBlock: 0,
blocksStart: 0,
fragmentBlockIndex: 0,
fileSize: 7,
fragmentOffset: 0,
@@ -95,7 +95,7 @@ var (

// this is for /foo/filename_0
testBasicFile = &basicFile{
startBlock: 0,
blocksStart: 0,
fragmentBlockIndex: 0,
fileSize: 0xb,
fragmentOffset: 0xc,
@@ -240,7 +240,7 @@ func GetTestFileSmall(f fs.File, c Compressor) (*File, error) {
}
testFs.compressor = c
ef := &extendedFile{
startBlock: superblockSize,
blocksStart: superblockSize,
fileSize: 7,
sparse: 0,
links: 0,
@@ -269,7 +269,7 @@ func GetTestFileBig(f fs.File, c Compressor) (*File, error) {
fragSize := uint64(5)
size := uint64(testFs.blocksize) + fragSize
ef := &extendedFile{
startBlock: superblockSize,
blocksStart: superblockSize,
fileSize: size,
sparse: 0,
links: 0,
2 changes: 1 addition & 1 deletion filesystem/squashfs/file.go
Original file line number Diff line number Diff line change
@@ -47,7 +47,7 @@ func (fl *File) Read(b []byte) (int, error) {
// 5- read in and uncompress the necessary blocks
fs := fl.filesystem
size := fl.size() - fl.offset
location := int64(fl.startBlock)
location := int64(fl.blocksStart)
maxRead := len(b)

// if there is nothing left to read, just return EOF
Loading