Version 11.06 2010.05.19 1345 UTC -0700
TerraFetcher is a program that will download large, irregular regions of images from
The output of the program is a set of data files compatible with
Doug Cox's wonderful USAPhotoMaps app. TerraFetcher will
download 4 and 64 meter/pixel TOPO images, 1 and 8 meter/pixel B&W aerial photographs, and .25 and 4 meter/pixel
"Urban" color images.
NEW: I grabbed a few 'Cartographic Boundary Files' from the U.S. Census Bureau,
and converted them for use with TerraFetcher. The resulting 3271 files, included in this
6.6 MB ZIP file, consist of "regions" that delimit all of the states and counties in the
United States (as well as state-equivalent and county-equivalent entities, like Puerto Rico, D.C., and
the various boroughs and census areas in Alaska). These files might be useful if you want to grab
tiles for an entire state or county, without having to create the region files yourself.
IMPORTANT: If you use these files, you must download version 10.23 of TerraFetcher. Some of
the counties have "holes", and processing the files requires the "excluded region"
feature of 10.23.
MERGING DATA FILES: A new utility, TFMerge, will read PTR and DTA files from two
directories and write them to a third. This might be useful if you had used TerraFetcher to
download tiles into separate directories (perhaps two adjacent counties, each in its own directory),
then wanted to combine the files for use within USAPhotoMaps.
Usage is pretty straightforward:
TFMerge <Input_Dir_A> <Input_Dir_B> <Output_Dir>
This will take all PTR and DTA file pairs in the directory "Input_Dir_A", find the corresponding
file pairs in "Input_Dir_B" (if any), and write the combined file pairs to "Output_Dir". If there
are input files with the same names (that is, they're the same row of tiles), the input files
will be combined into a single pair of PTR and DTA files in the output directory. If two files
contain the "same" tile (that is, files representing the same row of tiles
happen to overlap), the tile from the file in Input Directory A will be the one written to the output
The Output Directory must exist - the utility will not create it for you.
Note that if the file already exists in "Output_Dir", it will be overwritten. Copy/move files to
temporary locations, as desired, prior to running the utility. This isn't necessary if
"Output_Dir" is the same as "Input_Dir_B" (see below).
The utility can be downloaded via this link.
Note that this is a command-line program. You must open a command prompt and run the program from
there. Trying to run it from Windows Explorer will just show you the program's usage text.
Example: If you had a bunch of files in the default USAPhotoMaps location and a single "county's"
files in another directory, you could run the utility like this:
TFMerge "D:\MyCounty" "C:\Program Files\USAPhotoMaps\USAPhotoMapsData" "C:\Program Files\USAPhotoMaps\USAPhotoMapsData"
Note that Input_Dir_B and Output_Dir are the same directory. This is OK. The result of the above
is that all tiles in "D:\MyCounty" are added to the USAPhotoMapsData directory - the utility
will get each PTR/DTA pair in "D:\MyCounty", combine it with any existing, matching pair in
"...\USAPhotoMapsData", and write the combined pair back to "...\USAPhotoMapsData".
2005.09.10: changed TFMerge so that it if Input_Dir_B and Output_Dir are the same directory,
it won't perform a read/write of every file in Input_Dir_B after combining Input_Dir_A and Input_Dir_B.
The new sequence of steps is:
- For each DTA/PTR file pair in Input_Dir_A:
- read the pair from Input_Dir_A
- if the pair exists in Input_Dir_B, read it and combine with the pair from Input_Dir_A
- write the [combined] pair to Output_Dir
- If Input_Dir_B is not the same as Output_Dir, read each DTA/PTR file in Input_Dir_B and
write it to Output_Dir. (The previous version didn't have the "if not the same" condition)
There is no "Help file" available yet. That's in the works (or, at least, a text
file describing the program and its use). For now, you can look at the included text files and
read the rest of this page to get a basic idea of the program: TerraFetcher.txt describes
the format of the file used to retrieve regions of data, and JPEGs.txt shows how to
create very large JPEG images from the downloaded data.
I started writing this app when I wanted to download all of the USGS TOPO images for California
from TerraServer. In USAPhotoMaps (at the time), I could only get a single rectangle of
information at once, and these rectangles were limited in size and to one UTM zone.
While this was just fine for most purposes, getting all of CA posed two difficulties:
First, there were going to be an awful lot of rectangles, in two UTM zones. Second, because
they were rectangles, I'd be left with a lot of "triangles" along the CA-NV
border - I'd have to either live with those, come back later and delete them one tile at a
time, or make my rectangles extremely small. TerraFetcher let me define a set of "waypoints"
that delimited the state (I created the waypoints in Garmin's MapSource, then copied/pasted them
to the text file used as input by TerraFetcher). I then clicked the Start button in TerraFetcher,
and left it alone for 12 hours. The result was a set of USAPhotoMaps-friendly files, 4,848 of them
containing 8.37GB of data, that represented all of the TOPO maps for California on TerraServer,
in both UTM zones (10 and 11).
The Preview button will scan the contents of the USAPhotoMaps data directory
and the file of regions, counting up tiles that exist, need to be deleted/replaced, and/or
need to be downloaded - depending on the options selected. No data will be downloaded, and no
files will be created/altered.
The Start button actually performs the specified operation(s). It first does
the same thing as the Preview button, so that it can figure out what remains to be done (the
info is used to calculate Progress and Time remaining stats).
Both operations, Preview and "actual", can be safely stopped at any time by clicking
the Stop button. The program will finish writing any remaining output data.
The Time remaining stat is most accurate when: a) there is a lot of data to download and b)
the program has retrieved a significant number of tiles. Unless both of these conditions are
met, the Time remaining value can (and most likely will) fluctuate wildly.
The program can be used to create extremely large JPEG images from your downloaded data. The
size of the output image is limited only by available disk space (though once you get over
a certain size, maybe 400MB or so, you may have difficulty reading the JPEG file into any
viewing or printing software you might have). You can scale the image down, to save space. This
may be useful if you want to print an image of an entire state on a single sheet of paper -
something like 100,000 pixels across 8.5 inches is likely a waste for most consumer printers. You
could scale such an image down by a factor of 8, yielding 12,500 pixels across. This will reduce
the actual size of the image (pixel count) by a factor of 64. The reduction in size of
the JPEG file will vary, depending on how the encoder is able to compress the scaled image.
When creating large JPEG images, you're limited to a single UTM zone per output JPEG.
There is also an "Overview" function. This will show a rudimentary image representing
the tiles you have / don't have. The width of the displayed data is always a multiple of UTM
zones, and its height is the bounds of the data you have. This feature is intended to show
what tiles are missing from your data - it's not meant to be a pretty picture of the data
itself. When the Overview is drawn, black represents data you have, while red is data you
don't have. Black is drawn first (i.e. draw all black, as if you had everything, then draw
red pixels for missing tiles). The result is that you may not actually see black pixels where
you really have data - since the intent is to highlight missing data. You can put the mouse
pointer over a "red" area, see the UTM coordinates for that point, then use
USAPhotoMaps to zoom in on and (attempt to) download any small regions of missing tiles.
For example, if I look at my California TOPO data set, the Overview function shows
me zones 10 and 11, and indicates that I have a few "blocks" of data missing. It
turns out that these blocks really don't exist on TerraServer. When I first used this feature
after the 12-hour download above, there was also a single red line of pixels across zone 11.
This line showed me that I'd dropped a few consecutive tiles - I went into USAPhotoMaps and
manually retrieved the missing data. (I don't know what caused the dropped tiles - there were
no HTTP or other network errors, all files were intact, and there was no evidence of any other
failure; I can only presume that TerraServer didn't send the tiles, or sent its "no data"
JPEG (see below) in their place.)
TerraFetcher has undergone quite a bit of testing. I've had it write its output files to
empty directories, as well as to the default folder used by USAPhotoMaps. I've noticed no
corruption of any existing data files when writing to the latter location. However, before
using the program on your existing USAPhotoMaps files, you may want to back those files up
somewhere, at least until you're satisfied that TerraFetcher won't mess them up.
There is no "installer" provided with TerraFetcher. Merely unzip the contents of
the downloaded TerraFetcher.ZIP to some handy location on your hard drive, and run
the resulting TerraFetcher.EXE. (I run it from a "TerraFetcher" subdirectory
of the main USAPhotoMaps directory, a peer to the USAPhotoMapsData directory.) If you don't
have an appropriate "unzipping" utility, I recommend
One important note for downloading: When TerraServer doesn't have a requested tile, it
actually sends a 8321-byte "texture" tile. This is the BlankJPEG.jpg
image included in the TerraFetcher ZIP file. When USAPhotoMaps receives this tile, it
ignores it, so that it's not written to disk. This saves an awful lot of disk space if
you're attempting to download areas for which there is no TerraServer data. TerraFetcher
lets you emulate this behavior, but you must explicitly enable it: first, click the
Select button below "Tile A" and browse to and select the BlankJPEG.jpg
file. Next, select the Delete retrieved tiles, to save space option in the Download
Data section and make sure Tile A is checked. As TerraFetcher receives tiles from
TerraServer, it will compare them to the selected image. If there's a match (and there
will be for TerraServer's "blank" tiles), TerraFetcher will discard the received
You can also replace the retrieved texture tiles with another JPEG. This has the potential
benefit of saving space, while still letting you see what tiles were "empty".
Included in the ZIP file are three additional JPEGs: black with a white 'X' through it,
plain black, and plain red. While they're not the zero size of a discarded tile, they
are still significantly smaller than the texture tile: 1710 bytes for the tile with the
'X', and 753 bytes for the 'plain' tiles. You can create your own blank tiles instead of
using the 3 supplied - they must be 200x200 JPEG files, and can be monochrome or color.
To use the 'replacement' feature, select the Replace retreived tile A with tile B
option, set Tile A to BlankJPEG.jpg, and set Tile B to the JPEG file of choice.
After doing such a replacment, you can, if you wish, come back later and delete the
blank tiles from the data files: under "Existing Files", select the Process
ALL existing files, and don't download option as well as the Delete tiles
option. Set Tile A and/or Tile B to the tiles you want to delete, and check the Tile A
and/or Tile B boxes. No data will be downloaded from TerraServer - the program will
merely scan through your existing data files, removing all occurrences of Tile A and/or
Tile B (as if you'd used the Delete option when downloading).
Version 11.06 (non-.NET version now deprecated)2010.05.19 1345 UTC -0700
- Changed URL to point to msrmaps.com instead of (deprecated) terraserver-usa.com
Versions 10.30 and 11.05 (.NET) 2008.12.13 1530 UTC -0800
- Fixed bug in region edge detection (could/would result in filling an entire row to
the left of a horizontal region or path boundary)
- Added "squares" at each vertex in a path, to avoid the empty triangles at those
Versions 10.29 and 11.04 (.NET) 2007.12.18 1340 UTC -0800
- Changed TerraServer URL to terraserver-usa.com. Apparently, Microsoft eliminated
the old "terraserver.microsoft.com"
Version 11.02 (.NET version) 2005.09.11 1050 UTC -0700
- Increased the socket timeout value to infinite from the default of 10 seconds,
to avoid errors when the local machine's CPU and/or network connection is loaded.
Version 10.28 2005.09.01 1120 UTC -0700
- When creating a JPEG file, TerraFetcher now creates an INF file compatible
with UI-View. If you copy the .JPG and .INF files to UI-View's MAPS directory,
you should be able to open the "map" in that program. (Note: this
isn't tested, because I don't have UI-View - let me know if it doesn't work)
Version 10.27 2005.08.18 1205 UTC -0700
- Made the Overview window resizable, to allow a larger image area. You must resize
before pressing the Scan button, because much of the image
generation code uses the current size of the window to create and fill the
- Added an option the Overview window to filter tiles based on a "Region"
file (the same type of input file used for downloading tiles). If you select this
option, tiles you have will be shown in black, tiles you don't have will be shown
in red, and areas outsize the specified region(s) will be shown in grey.
Version 10.26 2005.08.15 1920 UTC -0700
- Fixed some exception handling used by 10.25's error logging.
- Fixed the mouse position display in the Overview window.
Version 10.25 2005.08.14 1200 UTC -0700
- Added some logging to the download procedure. If you're having problems with
downloading tiles, delete the file TF_TRLog.txt (in the same directory as
TerraFetcher.exe), retry the download, and email me the new TF_TRLog.txt
- Added support for the lower-resolution tiles that are stored in USAPhotoMaps'
'BigPhotoMapsData' directory. Check the 'BIG' checkbox on the various
TerraFetcher windows to use these tiles instead of the 'normal' data.
Version 10.24 2005.08.13 1040 UTC -0700
- Added UTM zone filtering to the "Overview" function.
- Fixed a bug that, if you were re-encoding while downloading AND deleting
the 'blank' JPEG tiles, would result in garbage tiles in place of the
'blank' tiles (this wouldn't affect tiles that weren't the 'blank' JPEG).
Version 10.23 2005.07.20 2000 UTC -0700
- Added "excluded" regions. If, when specifying a "Region=n" line in the
text file, you make n negative (that is, put a dash/hyphen in front of
the number of points in the region), that region will be excluded from
the download. This was added to handle the excluded polygons in the
Census data at the top of this page.
Version 10.22 2005.07.19 1530 UTC -0700
- Fixed a bug in testing whether tiles fall within a region (would only be
a problem if the defined region or path had a vertex on a tile boundary).
When the bug hit, an entire row of tiles would be downloaded, whether or not
each tile was inside the region.
- When creating a JPEG file, TerraFetcher now creates a "World" file.
This file will have a .JGW extension and contains:
- X-scale (meters per pixel in the X direction)
- Rotation in X direction (always 0)
- Rotation in Y direction (always 0)
- Negative of Y-scale (meters per pixel in the Y direction)
- Easting Coordinate of the center of the upper left pixel of the image
- Northing Coordinate of the center of the upper left pixel of the image
Version 10.20 2005.02.20 1020 UTC -0800
- The second release of this version number:
Drastically reduced the memory consumption (again) when converting GIF to
JPEG. I was previously allocating 120kB for each JPEG tile - the full size
of an uncompressed image. Now, I still allocate that buffer for the compression
process, but reallocate a much smaller buffer (10kB, or so - depends on the
compressed JPEG size) when the compression is complete. This will save many
megabytes of memory space across a single row of GIF -> JPEG tiles.
- Reduced the memory consumption for GIF -> JPEG conversion by about 160 kB.
- Handled some bad GIF files sent by TerraServer that would cause a crash. So
far, I've only seen these near Z4 430400e 7840000n, but without knowing why
TerraServer is sending them, they could very well appear anywhere.
- Added a couple of new options to the text file of regions. You can now
specify StartRow, StartNorthing, EndRow, and EndNorthing. These might be
useful if you stopped the download process and wanted to resume where you
left off, especially if what you'd already downloaded had a lot of empty
tiles (the "texture" JPEG) that you don't want to waste time
downloading again. See the sample text file TerraFetcher.txt for more
- Added another "blank texture JPEG" to the ZIP file: BlankJPEG2.jpg.
This tile is sent by TerraServer when the requested tile is outside the US
(or the request is otherwise invalid).
Version 10.19 2005.02.17 1550 UTC -0800
- Added support for TOPO maps in Alaska. They come down from TerraServer as
GIFs - a conversion to JPEG was added so that the resulting files will be
compatible with USAPhotoMaps.
Version 10.18 2005.02.12 1235 UTC -0800
- Fixed the problem with file naming for zones less than 10 (a difference
between TerraFetcher's and USAPhotoMaps' names).
- Removed the Windows NT (2000, XP) limitation. TerraFetcher will now run
on Windows 95, 98, and ME.
Version 10.17 2005.01.24 0905 UTC -0800
- Fixed a bug that prevented the generation of large JPEGs from B&W
1m Photo tiles.
- Fixed the download code so that existing files are not written if there
were no changes (i.e. no tiles deleted, no new tiles downloaded).
Version 10.16 2004.12.20 1421 UTC -0800
- Added the ability to specify "paths", along which TerraFetcher will
grab rectangles of a specified width. See the TerraFetcher.txt file for more
information and an example.
Version 10.15 2004.12.19 0830 UTC -0800
Some things I may (or may not) add in the future...
Things I almost certainly will not add...
- User suggestions (email
me if you have some)
- Printing of JPEG files. This is something best left to programs that already do it well.
Copyright © 2004, 2005 Don Starr - All Rights Reserved