Nada Labs

I was contacted by a visitor of this site asking for the following:

‘I read your nice article on file format reverse engineering and was wondering if you could give me a small tip / hint about compression / encryption. I am trying to understand a constant size file format and need to know if by any chance the file is compressed or encrypted in a simpler way, which leaves hope in cracking it.

…

In the case you would like to have a look at the files, I generated 2 pairs. The first file pair differs only in that one variable. The second file’s name and caption are set to “;1″;, the file 2b to “;1111111…”; (31 chars)’

As the reader seeked advice on how to proceed further and provided enough information to investigate the problem, I took a look. (continue reading…)

I had a need to convert GeoRSS files used with iMapPlot to a Google Earth kml which I accomplished with the following shell script. It’s ulgy but gets the job done. The script can also be downloaded: rss_process.zip

#!/bin/bash
infile=$1
outfile=`echo $infile | sed -e 's/\.xml/.kml/'`
grep -E '^<(title|description|georss\:(point|line)|link)>(.*)</\1>' "$infile" | sed -r -e 's/(-?[0-9]{2,3}\.[0-9]{1,}) (-?[0-9]{2,3}\.[0-9]{1,})/\2,\1,0/g' -e 's|<(title)>(.*)</\1>|\t<Placemark>\n\t\t<name>\2</name>|' -e 's|<(georss:point)>(.*),(.*),0</\1>|\t\t<LookAt>\n\t\t\t<longitude>\2</longitude>\n\t\t\t<latitude>\3</latitude>\n\t\t\t<altitude>0</altitude>\n\t\t\t<range>500</range>\n\t\t\t<tilt>0</tilt>\n\t\t\t<heading>0</heading>\n\t\t</LookAt>\n\t\t<Point>\n\t\t\t<coordinates>\2,\3,0</coordinates>\n\t\t</Point>\n\t</Placemark>|' -e 's|<(georss:line)>(.*)</\1>|\t\t<LineString>\n\t\t\t<tessellate>1</tessellate>\n\t\t\t<coordinates>\2</coordinates>\n\t\t</LineString>\n\t</Placemark>|' -e '1 s|.*<Placemark>|<?xml version="1.0" encoding="UTF-8" ?>\n<kml xmlns="http://www.opengis.net/kml/2.2">\n<Document>\n|' -e '$ s|(.*)|\1\n</Document>\n</kml>|' -e 's|<(link)>(http://.*)</\1>|<description><a href="\2">\2</a></description>|' | sed -r -e '/<\/description>/ {
N
/\n.*<description>/ {
s/<\/(description)>.*\n.*<\1>//
}
}' | sed -r -e 's|<(description)>(.*)</\1>|<\1><![CDATA[\2]]></\1>|' > "$outfile"

An attempted explaination

The grep command only grabs the tags from the RSS file that we are interested in.

The first sed statement 's/(-?[0-9]{2,3}\.[0-9]{1,}) (-?[0-9]{2,3}\.[0-9]{1,})/\2,\1,0/g' finds two sets of numbers seperated by a space and swaps their order while seperating them with a space and appending ,0. This converts the ‘latitude longitude’ coordinates into ‘longitude,latitude,altitude’ as used by Google Earth.

The second statement converts the title block into a Placemark and name block.

The third formats a point placemark, providing look at information while the fourth converts a line placemark.

The fifth inserts the kml header while the sixth inserts the kml footer.

The seventh sed statement converts a link block into a description block with a HTML link in it.

The second sed command merges two description blocks into one by removing the first close block and the second open block.

sed -r -e '/<\/description>/ {
N
/\n.*<description>/ {
s/<\/(description)>.*\n.*<\1>//
}
}'

This is accomplished by looking for the </description> block and loading in the next line if it is found. The next line is then checked for the <description> tag and if found a replacement is performed removing the two tags and any characters in between.

The third sed command wraps the description in a CDATA block to allow HTML to be included in the description.

As part of my GPS Logger project I needed to make sure that I could initialise and talk to a SD card over a SPI bus. The BusPirate is an excellent tool for testing  the physical and datalink layers. All parameters can be checked and adjusted on the fly without having to write any code. When the time comes to write code for the GPS logger, it will work first time as all the kinks and quirks were quickly worked out with the BusPirate.

This post covers SPI bus setup, card initialisation, reading and writing individual sectors.

(continue reading…)

So you have a file that you know contains something good, if only you could read it. Your searching efforts for documentation proved fruitless, required a significant outlay or to sign a NDA. Looks like you are going to need to reverse engineer the file format so you can use it. Most applications tend to use custom file formats for various reasons. These files are usually containers where other files are kept or application data.  A hex editor (xvi32) and C compiler (VS2008) were used to discover the layout of the culprit file. (continue reading…)