0.0.2

Changelog

• 507e30a - allow top-level 'offset restore' - allow empty if-blocks
• f6e132a Merge branch 'rework-nodes'
• 46b5fd6 add 'import' directive to read data from external file, which is required in some file formats
• 32f92fa add 'parse: continue' directive and remove fatih/color
• 31bd95a add FILE_NAME constant and no_ext() function to return filename without extension (needed for multi-file formats)
• 594cb18 add basename() function
• 9587164 add basic templates for archives/{lzma,zstd}, systems/macos/lzfse
• 6593c60 add bik template
• 6af770c add built-in function sevenbitstring()
• 2729ad9 add compression:pkware + more templates
• 594662f add f32 datatype, fix unmapped bytes count
• 7691a5a add more systems/nitendo-wii templates
• 90d67dd add more templates
• 4835882 add more templates
• 5dc5fb0 add more templates
• 025b5fc add more templates
• 0300fee add more templates
• d2720ca add more templates
• 4f291a9 add more templates
• f9d5d56 add more templates
• 9ad6a40 add more templates
• e08e97c add more templates
• 4e00772 add note
• 71d36d2 add skeleton
• 6cac285 add some Wwise templates
• 1530da0 add some files
• 0c9b59b add some more templates
• 21665b1 add some more templates
• 6c21140 add some more templates
• 2a3834e add some more templates
• 6f16e4e add some more templates
• 4d9cfc6 add some more templates
• 0fb59e2 add some more templates
• 1a02089 add some more templates
• 1617dc8 add some more templates + speed up text output
• 8f438e9 add some templates
• e8ea5f2 add some templates
• 7826f5c add some templates
• 302b762 add some templates
• fa1a942 add some templates
• aa5f84c add some templates
• b70eef1 add some templates
• ed7408c add some templates
• e0b0ba4 add some templates
• 00dd11a add some templates
• 6169d0c add some templates
• 48d17aa add some templates
• 29cb87e add some templates
• 62b1ef2 add some templates
• 3b04f16 add some templates
• aadb67c add some templates
• 6846fc5 add some templates
• 2f70b30 add some templates & rearrange some
• 6027cf7 add support for LZO1X compression, compressed:lzo1x
• 68f82ef add template systems/windows/vssver
• 21c96a2 add varint variant (vs64), and some more templates
• b41ffb3 added some templates
• 442fb21 added some templates
• 1c79681 archives/arj: decode unix timestamps
• 6420fb7 bump dependencies
• cf9c29d ci: update release workflow
• 25b2143 cleanup: drop field.Value and use int64 for size & offsets
• 28e1b01 cmd/feng: add --decimal flag to show offsets in decimal rather than default hex
• e66da24 cmd/feng: add --extract flag to allow extraction without specifying --out-dir. the basename of input file will be used
• d8f6a0d cmd/feng: add --offset to change start offset
• db398d1 cmd/feng: add --template arg to force parse input file with specific template
• 138a8c6 cmd/feng: add --time flag to measure template processing & parse time
• 8801c7f cmd/feng: also extract data from child nodes
• ea903e3 cmd/feng: fix --offset
• f48f1bd cmd/feng: improve --extract output text
• 966e519 cmd/feng: read user templates from ~/.config/feng
• c0ba194 cmd/feng: speed up --brief, and add some more templates
• 4095d38 cmd/renamer: remove app (out of scope for project)
• 5179827 comments
• 5ad4343 compression/lzf: rework to use io.Reader and generate a bytes.Buffer
• c6f909e count bytes read
• 0632962 diagnostics: warn on unpopped offsets
• 1099386 do some fewer copies
• ae588de dont evaluate plain filenames in 'filename' directive
• 2683b52 eval: refactor and reuse data, large performance gain on larger files
• 72bd3f8 expand custom structs as child nodes. add --tree argument to cmd/feng
• a03ed92 extend magic matching to match on a combination of magic bytes and file extension + update windows ico and pcx template to use it
• 8ef9b4b fix TestEvaluateStructsAndLayout() test
• b9a7858 fix TestFieldPresent
• 556cd0e fix presentation of i8,i16,i32,u64,i64 values
• fcb5d70 fix presentation of signed values
• 65b7b66 fix some tests
• 628c1b0 fix some tests
• 38a657c gamecube: improve some formats
• 6f5f70e games/data_design_interactive/wadh: add format
• 9afec4d images/bpg: use vu32
• 57f926e implement 'filename' directive, use it when extracting files
• c5c3910 improve TEMPLATE.md
• b0d19d1 improve some templates
• bea5163 improve some templates
• 0d49062 improve systems/windows/xnb template
• d66ebde improve template
• 0737e90 improvements with nested custom structs
• e125f50 internals: rework utf16[2] to mean 2 byte length, rather than 2 characters
• 6305e68 internals: switch to afero and port tests to use it
• 8ee312c magic: inherit top level extensions for magic matches
• f45afc0 make 'offset' a reserved struct name
• ee0a257 mapper: fix setting of raw text labels
• 88616f4 mapper: refactor eval functions
• 592e79c mapper: rename var
• 97d384e minor fixes
• 2aeecf8 minor fixes
• 163570d minor fixes
• 09e1d18 minor tweaks
• 349ab0f minor tweaks + another template
• 3aaba9b minor updates
• 6c76b4e no_magic: improve auto matching using filenames list of regexp patterns
• 705ddad present u64 array + improve yaml template error
• bc299dc pretty print file size
• 7364048 rearrange some templates
• ce5692e refactor
• b3d3942 refactor
• 87d884a refactor
• 2e1e2a6 refactor
• b9df65f refactor
• 51e154b refactor and speed-up
• 5bbbd6e refactor and update dependencies
• ecba58e refactor args
• 87b513e refactor data extraction and add template for game ObsCure
• 0a77946 refactor of compression handling
• e7f274a refactor to improve evaluation speed
• 2ec1e4f remove 'self.offset' syntax. OFFSET gives current offset value
• bef2e8c show ascii in no match header dump
• 88eb4a0 show hex dump of first 0x40 bytes if unmatched file
• 374a682 simplify
• 55a24a2 simplify
• 747cd07 smoketest: add missing file
• 653fd40 speed up evaluation with fl.evaluateExpressionWithExistingVariables()
• f6bd6f9 speed up reading
• efb1b27 sync changes
• f577abb sync notes
• bccb36a systems/android: improve dex endian swapping
• e10271d systems/gamecube: add hvqm4 template
• 8e72150 template matcher: auto-match no_magic templates in case of a matching extension
• 25db765 template: add 'sjis' format for shift-jis encoded text
• 022c087 template: add data type dostimedate
• c5176ba template: add ext(filename.ext) function
• e89b43a template: add list_val() function to read a field value from another struct
• 62afc30 template: add lzf decompression
• 47a7c12 template: add rgba32 color type
• 3eee45e template: add support for 'encryption' directive and encrypted:u8 type
• 287d4e6 template: add zlib_loose compression to allow strict errors
• 0ae4f19 template: implement xor_key directive
• 15bca1c template: let peek_ functions to accept ints
• 0ca58fd templates/games/mmfw: ImageBuilder game asset format
• 79db08b templates/meta/riff: RIFX (big endian) support
• 46c20b4 templates: add bitset() function and asciinl data type
• 42cb7b5 templates: add cleanstring() function and fix a presentation bug
• 97c14ae templates: add data type u24
• 4e0f73a templates: add extractor template for godot .pck files
• 82b23d3 templates: add lzma2 compression
• d6c0b6c templates: add some new ones
• 324b14c templates: add support for compressed:gzip stream
• c115d9b templates: add utf8z data type
• e6d0085 templates: allow magic offset block to specify endian
• 9fb7c45 templates: allow patttern matching on arbitrary length ascii
• 5ea19f3 templates: improve systems/xbox360/live.yml
• a36db6d templates: minor cleanup
• 42a2cad templates: recognize GDSC (godot compiled script file)
• bc52656 templates: recognize WAVEBK11 from Spider-Man 2
• 9c4c1af templates: remove outdated TODOs and some cleanup
• 94735eb update README
• b23e4a9 update README
• 08b657d update README
• 15d5422 update dependencies
• a89e87b update dependencies
• f6bb74b update dependencies
• 8c4e06e update some templates
• a01b7d9 ux: error on invalid layout lines in template
• def0924 various improvements
• 67a731c wii stuff

0.0.1

Changelog

• 3e81893 README: add link to Discord
• 082a118 TEMPLATE.md: expand 'Endianness' section
• ead6f54 WIP - if statement rewrite
• ef58606 WIP - smoketests
• aa46d40 add % mapping in coverage
• 047141c add 'SQLite format 3' skeleton
• b9cff03 add 'until' directive to grab all data until pattern matches
• b47024d add LICENSE
• 6aa9e7c add Sony Playstation SFO format
• 39da158 add TODO
• 72952ce add XPR0 format (xbox) and draft for .res files from MX Unleashed (xbox)
• b95933f add XSB (xbox sound bank?) format skeleton (magic 'SDBK')
• 93d1869 add Xbox XBE format
• 60839ce add ascii and asciiz data types
• 12c0c87 add asf (windows media) format skeleton
• 38fa7d3 add compression:deflate format
• f238f35 add data type time_t_32 (unix timestamp)
• aa638a4 add data type utf16z, zero (00 00)-terminated utf16 string
• 36771d2 add directive 'parse: end' to manually stop format parser (used by archives/rar)
• 74025ae add filetime data type
• c6c47a1 add font/ttf format skeleton
• d83ef37 add format skeletons for XBOX XSB and XWB formats
• f3899ce add gameboy advance header format
• 6ef8001 add gameboy header format
• f98b233 add java .class format skeleton
• 6836bcd add lz4 archive format + UnityFS asset bundle format
• 2069e3b add more gamecube formats
• 037d136 add new format systems/ps3/ps3_npd
• b1f0594 add new formats archives/ar, systems/os2/ftcomp, windows/compress-kwaj
• c3947c9 add new formats archives/{iso,teledisk,vdi,winimg}, images/bpg, network/torrent
• 1ce2fc8 add new formats documents/chm, fonts/{dfont,pfb,ttc,type1,x11snf}
• 97466d1 add new formats systems/flash/swf, systems/python/pyc
• 923e6df add nintendo 64 header format
• 4a047f8 add nintendo ds header format
• 4478fc9 add offset directive and field.index special iteration index for complex parsing
• b62c85b add offset() and len() functions
• 1271551 add references to some synalysis grammars
• 6f17c50 add references to some winhex templates
• 7d2b84e add skeleton templates for systems/flash/flv, media/{mkv,mp3}, systems/machos/macho-o
• 9752452 add smoketests for more formats
• ba84b69 add support for archives/{iso,mozlz4}, add datatype compressed:lz4
• 7722dc9 add types dostime (16 bit) and dosdate (16 bit)
• 11ee56f add utf16le data type
• 34283ad add windows/regf template
• c4ef1bb adjust presentation
• 2ebbbfb allow constants in if-expressions
• 766aca8 allow ranged units in if-expression
• df33ba9 android: add dex and 'android resource' format templates
• 2ed3bdd archive/7zip: improve format by using vu64
• 04de994 archive/tar: improve format
• 897a1ed archive/xz: improve format and add vu64 data type (variable encoded u64)
• b09ce38 archive: add arj format skeleton
• 19f03bc archives/ar: finish format + rearrange some templates
• cf04f74 archives/lha: improve template
• 15f89bc archives/rar: improve format
• 259e2ec archives/zip: enable deflate extraction + rearrange some templates
• cccc376 archives: add bzip2 skeleton
• a8f63c9 archives: improve arj format
• 1ae128a bit match: fix a typo
• e381aed bump dependencies
• ebc3684 ci: add .goreleaser.yml
• 802f5ff ci: fix TestCompareWithReferenceParse and reduce debug output
• ea94316 cmd/feng: add --brief for a 1-line description of matching template, disable unimplemented --verbose flag
• 0fc0373 cmd/feng: add --cpu-profile and --mem-profile argument for debugging
• 3371032 cmd/feng: add --extract-dir to force stream extraction (currently only comp:zlib
• 6f417f3 cmd/feng: auto match input file vs all templates
• 2fb2ba7 cmd/feng: default to hide raw values, show them with --raw
• 332c5a2 cmd/feng: improve bit and eq matched fields in prensentation
• 2e9a288 cmd/feng: limit amount of hex data displayed
• 8619ce4 cmd/feng: propagate template error better
• e9c42a5 cmd/feng: report amount of unampped trailing bytes, if any
• a619202 cmd/feng: simple data lister
• 02f011f cmd/feng: switch from kingpin to kong
• 82877a4 cmd/renamer: add --verbose flag
• 37c1d35 cmd/renamer: new cli to mass rename files to the correct extension
• 81d80b2 cmd/smoketester: print out the slowest mapped files at end of run
• 920b4b6 cmd\feng: present data type and decode simple formats
• 7e5a12d convert templates for layout format
• b4dfff5 databases/sqlite: improve format
• 4d6c940 decorate structs with label value
• 333d77f documents/ole_compound: rename & improve format
• bfb5b34 documents/pdf: slightly improve
• 0deb883 documents: add pdf format skeleton
• 26bab0a documents: new format skeletons for doc, hlp, rtf, wri and archives/ar
• 70cbca6 embed.FS templates
• f6160cc enable smoketest running with github actions, fixes #1
• 1f266d7 expand FILE_SIZE to size in bytes
• 2df5b67 expand self.offset
• 9ae13cb expand template structs as data types, improve images/gif
• fd334bd expand variables in ranges
• f359a75 fix some tests
• 3ce5f5e fix: map string representation of complex types in eval.go
• 2b2d6e9 fonts/otf: finish format + add alignment(val,align) helper
• beecf56 fonts/ttc: improve format greatly templates: evaluate vectors and make usable by expressions, such as offset: V1Header.Offsets[self.index] (fonts/ttc) present: show vector values
• cb757cf fonts/woff2: improve format
• 77f5879 fonts/woff: greatly improve template
• 15069cf fonts: add skeletons for eot,otf,woff,woff2
• 6c70080 fonts: improve eot format
• 748c17d fonts: improve otf format
• 75c45d6 fonts: improve ttf format
• 786631f gamecube/wii: add basic detection of .nkit.iso images
• 4c05d83 image/bmp: improve template
• cf65974 image/bmp: parse test16.bmp
• c4347d5 image/gif: add smoketest
• 4b960c9 image/gif: minor improvements
• 59a169d image/ico: add smoketest
• 2dc9c4b image/jpeg: add smoketest + fix DHT size calculation
• 240e3f2 image/pcx: add smoketest
• 40c1d2f image: add smoketests for remaining supported formats
• 5d2cb19 images/pcx: disable magic because too many false positives
• adddd10 implement 'if Field in (1,2,3) and 'if Field notin (1,2,3) statements
• c491d2f implement MatchedPattern
• 517b698 implement access to field.offset and field.len properties from template
• 22fa79d implement data types i8,i16,i32,i64
• 82b90b2 implement file magic matching for templates
• 5ff2811 implement sized structs
• 1bc599e implement start:length absolute address syntax
• a1ae9da implement string concatenation and use it for label directive expansion, eg: label: self.Key + " = " + self.Value
• af9a9d8 implement struct slices
• bc77b70 improve archives/{arj,lz4}
• 18ca813 improve image/jpeg template
• 3d7ab50 improve images/bmp template
• ff0ea78 improve images/tiff
• 8fcb9a1 improve logging and update notes
• 6672675 improve parsing for PS4/PS5 CNT files
• b357ba0 improve presentation of sized fields
• 3d1837a improve regf template
• bf1cfd0 improve regf template
• 2caf74b improve systems/unix/elf and systems/ps4/ps4_self
• 8ad9936 improve templates/systems/ps3/ps3_pkg
• 1cc5bfe initial commit
• b6bdbfe mapper: accept eof errors as valid parse for otherwise valid mapping
• 56ce99f mapper: add 'data: unseen' directive to mark data for 'SAMPLE PLEASE' prompt + update TODO
• 439a33f mapper: propagate eof
• 71f3478 mapper: store file size
• be520c6 mark up some more templates with 'data: unseen' to prompt for SAMPLES
• a643e21 minor tweaks
• d560e2c misc improvements
• ce116b5 only evaluate datafield ranges once + misc fixes
• 350157f presentation: present 'eq' field matched values in field data type
• 1097913 progress git,ico templates
• 5ecea4f progress png template
• ab418d2 rar: disable unfinished parsing
• 285f8b4 rar: improve template
• 78a128c rar: remove unneeded parse directive and update docs
• 1e0d40e rearrange test files
• 5d54243 refactor
• b972dda refactor
• 4463806 refactor GetValue
• 616947d regenerate smoketests with more width + add comp:zlib data type
• a0e7a86 remove unused u8[12:34] absolute offset format, is better achieved with offset directive
• 4ae28ae rename archives/7z to archives/7zip
• 8b37aaa restored smoketests for working formats
• dcf3be2 rework eval
• ff0d4fa rework eval more
• c47295a simplify a test
• 0d2e964 smoketest: add another sample
• 1c8f46a smoketest: add image/dds
• 7a6f6ef smoketest: enable working tests
• f0a72f6 smoketest: generate reference presentations
• afcf0b4 sync last changes
• 5ccd2d3 systems/lua: small improvements
• dcb29df systems/macos: add pef executable template
• e5fe46c systems/nintendo-wii-u: add some basic formats
• f380513 systems/unix: add ELF file format skeleton
• f9c89b8 systems/xbox/xbe: improve format
• 6f1d581 systems/xbox: minor improvements
• b9fc80d systems: add luac 5.2 bytecode format
• 4ea80a1 template jpeg: SOF0
• 5d852f2 template: add peek_i16() and peek_i32() functions, improve windows/exe-mz
• 7dbef50 template: add vu32 - variable-length encoded u32, as used by fonts/woff2
• e923219 template: allow top-level definition of endian
• 7ca8e33 templates: minor improvements
• 4edadd3 tiff: remove link
• f78238f update README
• 5eb9758 update dependencies
• 98ea594 update dependencies
• 9aac648 update dependencies
• 60485b6 utf16 string cleanup
• cb90b2b utf16String: strip trailing NULL bytes
• 76cd730 windows/exe-mz: detect MZ-LX header
• 3beabdd windows/exe-mz: detect MZ-PE header
• ff565b7 windows/exe-mz: improve MZ-NE format
• 45ee8fa windows/exe-mz: improve format with LE and NE headers + show bitfield pattern index and size in presentation
• 1f84dbe windows/exe-mz: improved MS-DOS exe format
• bd71c9b zip: greatly improve parser