TerraFetcher


Version 11.06 2010.05.19 1345 UTC -0700

Download (130kB ZIP file) What's new Wish list

TerraFetcher is a program that will download large, irregular regions of images from Microsoft's TerraServer. The output of the program is a set of data files compatible with Doug Cox's wonderful USAPhotoMaps app. TerraFetcher will download 4 and 64 meter/pixel TOPO images, 1 and 8 meter/pixel B&W aerial photographs, and .25 and 4 meter/pixel "Urban" color images.

NEW: I grabbed a few 'Cartographic Boundary Files' from the U.S. Census Bureau, and converted them for use with TerraFetcher. The resulting 3271 files, included in this 6.6 MB ZIP file, consist of "regions" that delimit all of the states and counties in the United States (as well as state-equivalent and county-equivalent entities, like Puerto Rico, D.C., and the various boroughs and census areas in Alaska). These files might be useful if you want to grab tiles for an entire state or county, without having to create the region files yourself. IMPORTANT: If you use these files, you must download version 10.23 of TerraFetcher. Some of the counties have "holes", and processing the files requires the "excluded region" feature of 10.23.

MERGING DATA FILES: A new utility, TFMerge, will read PTR and DTA files from two directories and write them to a third. This might be useful if you had used TerraFetcher to download tiles into separate directories (perhaps two adjacent counties, each in its own directory), then wanted to combine the files for use within USAPhotoMaps.
Usage is pretty straightforward:
TFMerge <Input_Dir_A> <Input_Dir_B> <Output_Dir>
This will take all PTR and DTA file pairs in the directory "Input_Dir_A", find the corresponding file pairs in "Input_Dir_B" (if any), and write the combined file pairs to "Output_Dir". If there are input files with the same names (that is, they're the same row of tiles), the input files will be combined into a single pair of PTR and DTA files in the output directory. If two files contain the "same" tile (that is, files representing the same row of tiles happen to overlap), the tile from the file in Input Directory A will be the one written to the output file.
The Output Directory must exist - the utility will not create it for you.
Note that if the file already exists in "Output_Dir", it will be overwritten. Copy/move files to temporary locations, as desired, prior to running the utility. This isn't necessary if "Output_Dir" is the same as "Input_Dir_B" (see below).
The utility can be downloaded via this link.
Note that this is a command-line program. You must open a command prompt and run the program from there. Trying to run it from Windows Explorer will just show you the program's usage text.
Example: If you had a bunch of files in the default USAPhotoMaps location and a single "county's" files in another directory, you could run the utility like this:
TFMerge "D:\MyCounty" "C:\Program Files\USAPhotoMaps\USAPhotoMapsData" "C:\Program Files\USAPhotoMaps\USAPhotoMapsData"
Note that Input_Dir_B and Output_Dir are the same directory. This is OK. The result of the above is that all tiles in "D:\MyCounty" are added to the USAPhotoMapsData directory - the utility will get each PTR/DTA pair in "D:\MyCounty", combine it with any existing, matching pair in "...\USAPhotoMapsData", and write the combined pair back to "...\USAPhotoMapsData".
2005.09.10: changed TFMerge so that it if Input_Dir_B and Output_Dir are the same directory, it won't perform a read/write of every file in Input_Dir_B after combining Input_Dir_A and Input_Dir_B. The new sequence of steps is:


There is no "Help file" available yet. That's in the works (or, at least, a text file describing the program and its use). For now, you can look at the included text files and read the rest of this page to get a basic idea of the program: TerraFetcher.txt describes the format of the file used to retrieve regions of data, and JPEGs.txt shows how to create very large JPEG images from the downloaded data.

I started writing this app when I wanted to download all of the USGS TOPO images for California from TerraServer. In USAPhotoMaps (at the time), I could only get a single rectangle of information at once, and these rectangles were limited in size and to one UTM zone. While this was just fine for most purposes, getting all of CA posed two difficulties: First, there were going to be an awful lot of rectangles, in two UTM zones. Second, because they were rectangles, I'd be left with a lot of "triangles" along the CA-NV border - I'd have to either live with those, come back later and delete them one tile at a time, or make my rectangles extremely small. TerraFetcher let me define a set of "waypoints" that delimited the state (I created the waypoints in Garmin's MapSource, then copied/pasted them to the text file used as input by TerraFetcher). I then clicked the Start button in TerraFetcher, and left it alone for 12 hours. The result was a set of USAPhotoMaps-friendly files, 4,848 of them containing 8.37GB of data, that represented all of the TOPO maps for California on TerraServer, in both UTM zones (10 and 11).

The Preview button will scan the contents of the USAPhotoMaps data directory and the file of regions, counting up tiles that exist, need to be deleted/replaced, and/or need to be downloaded - depending on the options selected. No data will be downloaded, and no files will be created/altered.

The Start button actually performs the specified operation(s). It first does the same thing as the Preview button, so that it can figure out what remains to be done (the info is used to calculate Progress and Time remaining stats).

Both operations, Preview and "actual", can be safely stopped at any time by clicking the Stop button. The program will finish writing any remaining output data.

The Time remaining stat is most accurate when: a) there is a lot of data to download and b) the program has retrieved a significant number of tiles. Unless both of these conditions are met, the Time remaining value can (and most likely will) fluctuate wildly.

The program can be used to create extremely large JPEG images from your downloaded data. The size of the output image is limited only by available disk space (though once you get over a certain size, maybe 400MB or so, you may have difficulty reading the JPEG file into any viewing or printing software you might have). You can scale the image down, to save space. This may be useful if you want to print an image of an entire state on a single sheet of paper - something like 100,000 pixels across 8.5 inches is likely a waste for most consumer printers. You could scale such an image down by a factor of 8, yielding 12,500 pixels across. This will reduce the actual size of the image (pixel count) by a factor of 64. The reduction in size of the JPEG file will vary, depending on how the encoder is able to compress the scaled image. When creating large JPEG images, you're limited to a single UTM zone per output JPEG.

There is also an "Overview" function. This will show a rudimentary image representing the tiles you have / don't have. The width of the displayed data is always a multiple of UTM zones, and its height is the bounds of the data you have. This feature is intended to show what tiles are missing from your data - it's not meant to be a pretty picture of the data itself. When the Overview is drawn, black represents data you have, while red is data you don't have. Black is drawn first (i.e. draw all black, as if you had everything, then draw red pixels for missing tiles). The result is that you may not actually see black pixels where you really have data - since the intent is to highlight missing data. You can put the mouse pointer over a "red" area, see the UTM coordinates for that point, then use USAPhotoMaps to zoom in on and (attempt to) download any small regions of missing tiles.

For example, if I look at my California TOPO data set, the Overview function shows me zones 10 and 11, and indicates that I have a few "blocks" of data missing. It turns out that these blocks really don't exist on TerraServer. When I first used this feature after the 12-hour download above, there was also a single red line of pixels across zone 11. This line showed me that I'd dropped a few consecutive tiles - I went into USAPhotoMaps and manually retrieved the missing data. (I don't know what caused the dropped tiles - there were no HTTP or other network errors, all files were intact, and there was no evidence of any other failure; I can only presume that TerraServer didn't send the tiles, or sent its "no data" JPEG (see below) in their place.)

TerraFetcher has undergone quite a bit of testing. I've had it write its output files to empty directories, as well as to the default folder used by USAPhotoMaps. I've noticed no corruption of any existing data files when writing to the latter location. However, before using the program on your existing USAPhotoMaps files, you may want to back those files up somewhere, at least until you're satisfied that TerraFetcher won't mess them up.

There is no "installer" provided with TerraFetcher. Merely unzip the contents of the downloaded TerraFetcher.ZIP to some handy location on your hard drive, and run the resulting TerraFetcher.EXE. (I run it from a "TerraFetcher" subdirectory of the main USAPhotoMaps directory, a peer to the USAPhotoMapsData directory.) If you don't have an appropriate "unzipping" utility, I recommend WinZip.

One important note for downloading: When TerraServer doesn't have a requested tile, it actually sends a 8321-byte "texture" tile. This is the BlankJPEG.jpg image included in the TerraFetcher ZIP file. When USAPhotoMaps receives this tile, it ignores it, so that it's not written to disk. This saves an awful lot of disk space if you're attempting to download areas for which there is no TerraServer data. TerraFetcher lets you emulate this behavior, but you must explicitly enable it: first, click the Select button below "Tile A" and browse to and select the BlankJPEG.jpg file. Next, select the Delete retrieved tiles, to save space option in the Download Data section and make sure Tile A is checked. As TerraFetcher receives tiles from TerraServer, it will compare them to the selected image. If there's a match (and there will be for TerraServer's "blank" tiles), TerraFetcher will discard the received data.

You can also replace the retrieved texture tiles with another JPEG. This has the potential benefit of saving space, while still letting you see what tiles were "empty". Included in the ZIP file are three additional JPEGs: black with a white 'X' through it, plain black, and plain red. While they're not the zero size of a discarded tile, they are still significantly smaller than the texture tile: 1710 bytes for the tile with the 'X', and 753 bytes for the 'plain' tiles. You can create your own blank tiles instead of using the 3 supplied - they must be 200x200 JPEG files, and can be monochrome or color. To use the 'replacement' feature, select the Replace retreived tile A with tile B option, set Tile A to BlankJPEG.jpg, and set Tile B to the JPEG file of choice. After doing such a replacment, you can, if you wish, come back later and delete the blank tiles from the data files: under "Existing Files", select the Process ALL existing files, and don't download option as well as the Delete tiles option. Set Tile A and/or Tile B to the tiles you want to delete, and check the Tile A and/or Tile B boxes. No data will be downloaded from TerraServer - the program will merely scan through your existing data files, removing all occurrences of Tile A and/or Tile B (as if you'd used the Delete option when downloading).

Revision History

Wish List

Some things I may (or may not) add in the future... Things I almost certainly will not add...

Copyright © 2004, 2005 Don Starr - All Rights Reserved