Section Seven: Geoprocessing Specifically in ArcMap

As mentioned in the introduction for this chapter, one of the main goals of this class is to develop a foundation of geoprocessing tool comprehension.  That comes only from reading about what tools can do, associating them with a category of related tools, and taking minute before running a specific tool to predict an outcome based on the input layers.  Running the tool and examining the output either confirms or changes your future predictions, which in turn builds a broader geoprocessing tool comprehension.  These skills are GIS software independent, as geoprocessing tools all accomplish the same tasks.  The output of a clip tool run in ArcGIS will be the same as if the same data were to be run through a clip tool in QGIS, with the only real difference being the tool interface.  

When comparing the tool interface for ArcGIS to another software, like QGIS, ArcGIS is actually more "user friendly" and less assuming that the user really understands concepts of GIS - defining a vector or raster, knowing what geometry types are available, and having the ability to predict outputs.  As this class uses ArcGIS, the most common GIS software in use today, there are some specifics about the software that need to be covered, such as enabling extensions, filling in tool dialog boxes, the Results window, and knowing where to look to know if a tool is running, has completed successfully, completed with an error, or failed to run.  Like previous chapters, the goal of this section is not to memorize how tasks are completed in the software, but instead to introduce the ideas and reasoning behind the tasks so when the tasks are presented in lab, the seem a bit familiar.

Figure 7.18: Comparing the Clip Tool Interface for QGIS to ArcMap
QGIS_cliparcgis_clip
QGIS Clip ToolArcGIS Clip Tool
qgis_clip_choose_shparcgis_clip_choose_file
QGIS doesn't show shapefiles as a single item, but instead shows all the file and assumes the user knows to select the .shp file.ArcGIS offers shapefiles as a single file which is more user-friendly
qgis_save_outputarcgis_clip_save_file
Like the input selection box, the output box offer a large number of file types.ArcGIS only offers shapefiles to be saved inside folders and feature classes to be saved inside geodatabases (not shown).

7.7.2: Enabling Extensions and Launching Tools

ArcGIS, a proprietary software, costs money, unlike a opens source software like QGIS.  And it costs a fair amount of money.  With the exception of K-12 school and humanitarian non-profits, companies need to pay for not only the base software, but also for any upgrades and additional advanced toolboxes.  In order to save costs, companies can elect to purchase the number of base licenses they need to match the number of employees and then just a few copies of the more advanced tools (called extensions) to share among everyone.  This sharing process, in ArcGIS, is referred to as enabling extensions.  When an extension is enabled on one machine by a single technician on a shared company network, that extension cannot be used by any other technician on the same network until it is disabled by the first technician.  Think about it like a public library - Instead of purchasing tons of copies of one particular best-seller, they purchase a limited number of copies, then lend them out free of charge to their registered users. While the book is checked out, no one else can read it, as it is not physically available to any other reader while in possession of the first reader.  ArcGIS extensions work the same way - limited count extensions are only available to a few technicians at a time to check out (enable) and cannot be used by another technician until it is checked back in (disabled).  

To enable an extension in ArcGIS, specifically:

7.19: Enabling Extensions in ArcGIS
If an extension is available to a user but is simply not enabled, a warning message informing the user is presented.  When compared to the error message which appears when a tool is not paid for (below), this one looks quite a bit more scary when it actually means "need to check the box".extension_disabled
  1. open the Customize menu
  2. click on “Extensions” to launch the dialog box
extension_menu
  1. place a check mark in the box to the left of the extension name
  2. close the Extensions dialog box
extension_dialogbox

If the extension is not available (not paid for or all the copies are checked out), instead of a check mark appearing in the box, a pop-up will appear stating “ The extension could not be activated. There is no license name currently available”

extension_not_available

7.7.3: Launching Geoprocessing Tools

Within ArcGIS, geoprocessing tools are launched a couple of different ways: from the geoprocessing menu at the top of the software window, from the ArcToolbox window, and from the Search window.  These three locations are available in both ArcMap and ArcCatalog, since all of the geoprocessing tools can be run in either software. In general, tools in ArcCatalog tend to run faster and with less errors, especially more complex or table-based tools.  This doesn't mean tools will always fail or run slowly in ArcMap, as that isn't true, but when tools are run in ArcCatalog, there is no need for the software to draw any layers or organize a more complex layout of data and tools.  In ArcCatalog, tools just run and save the data where it's been told to save instead of processing a more in-depth interface.  

The Geoprocessing Menu

At the top of the window in both ArcMap and ArcCatalog is a series of menus, several of which should be fairly familiar right now.  One of the less familiar menus is the Geoprocessing menu which contains the six most used ArcGIS tools and the option to Search for tools within the software.  This menu is totally customizable by the technician, which is especially handy when those six tools are not your six most common tools.  There is also an option to create new menu that a technician can name and populate with the tools they might need, say for a long-term specific project.
geoprocessing_Menu

The ArcToolbox

The ArcToolbox is a collection of toolboxes and sub-toolboxes, organized by grouping similar tools together.  For example, the Analysis Tools toolbox contains four sub-toolboxes - Extract, Overlay, Proximity, and Statistics.  The Analysis toolbox "contains a powerful set of tools that perform the most fundamental GIS operations. With the tools in this toolbox, you can perform overlays, create buffers, calculate statistics, perform proximity analysis, and much more" (ArcGIS Help Menu).  From there, each sub-toolbox contains a group of similar tools which perform a series of related operations. If you look inside the Proximity toolbox, you'll find a group of tools which examine how data is spatially related to other data.  This toolbox explores ideas such as "What is the nearest fire hydrant to a specific office building?" (Near); "Which is the nearest fire hydrant to a whole series of homes represented by points?" (Generate Near Table); and "Where is the area which measures exactly 5 miles in infinite cardinal directions from a coffee shop represented by a point?" (Buffer).

coffee_shop_buff

For each individual tool found in the ArcToolbox, tools can be run in single mode, meaning the tool parameters are just filled out once and the result is (most often) a single dataset (vector, raster, or data table).  Tools can also be run in batch mode meaning the tool is run multiple times in a row with individual inputs and outputs, however, it results in many output files.

batch_geoprocessing

While the ArcToolbox may seem overwhelming at first, after spending some time looking for tools and examining the structure, it will not only begin to make sense, but you will also start to find other “new to you” tools, resulting in “Ooo. There’s a tool for that? Neat!”

arctoolbox

The Search Window

Searching for tools is a quick and easy way to find tools, especially if the toolbox is unknown. When the Tools option is selected in the Search window (as seen in this screenshot), the tool will only search for tools and not data or MXDs or images.  As the tool name is typed into the search box, suggested are presented with the main toolbox listed in parenthesis. 

Once you hit enter or click on one of the suggestions, the box populates with the best matches. The toolbox where the tool can be found is first in the list, followed by tools.  In the example, the Buffer tool is found in the Analysis toolbox, thus the Analysis toolbox is listed first.  Exact matches will be in bold, including in a tool name that contains the search word. Clicking on the tool name will launch the tool, clicking on the description below the tool name, listed in dark blue, will open the Help menu article for that tool, and clicking on the green toolbox path will open the ArcToolbox window, revealing where that tool lives.

tool_search_results

tool_search

7.7.4: Filling in Tool Dialog Boxes

Each geoprocessing tool is unique in the required inputs - vector or raster, specific geometry, numeric inputs, etc, but they are all the same in the fact they each take an 1. input layer (or layers), 2. require some parameters, and 3. have a line to define a name and place to save the new output layer. Some tools, such as those which are considered overlay, proximity, and extraction tools, require an “interaction” layer - the layer that defines the spatial comparisons as defined by the tool.

geoprocessing_flow

Saving Properly When Running the [insert name here] Tool

(Almost) every time you run a tool in ArcGIS, there will be an “Output Location” box (there are a select few tools that modify the input layer, thus do not require an output layer name and place). This box is tells ArcGIS where to save and what to call the output file of the tool after it runs.  Within the data model for Introduction to GIS, there is the Results folder, a place to save the output layer.  If you stick to the suggested data model and save all tool outputs to a single Results folder, you always know where the resulting layer from any tool is stored.

Figure 7.20: Looking at a Geoprocessing Tool Dialog Box
output_dialog_box

The Output Layer box.  Depending on the tool, the box will be called slightly different things.  Another note: ArcGIS will almost never have the heading "Output Shapefile", but instead will have "Output Feature Class" to reference any tool which produces a vector layer.  The software is already a giant software, space wise, and to have little things like the word "shapefile" would make it even larger and slower to run.

To designate a place to save the output, click on the folder icon  GenericOpen_B_32  to the left of the Output Feature Class box (in the case of vectors) and drive to your results folder (keep opening folders until you get to your destination). Once you've found the place you'd like to save the output file, you need to give your output file a memorable and meaningful name. ArcGIS will default to the name of the tool (buffer, clip, erase, etc) appended to the original file name, which is neither memorable or meaningful. It is your job, nay, your duty, to rename the file according to your goal and task at hand, for example, “River_Buffer_50_meters”. 

Figure 7.21: Providing a Logical Name and Location for Output Data
output_dialog_box

The Buffer tool has the first two requirements of (most) Geoprocessing tools - An input layer and an output location, using a memorable and meaningful layer name.

If a place and name is not assigned to a output dataset, the default location to save any tool is the Default Geodatabase and the default name is the name of the input layer with the name of the tool appended to the end.

Figure 7.22: Automatically Populated for Every Tool - The Default Geodatabase
output_dialog_box

Every tool initially populates the output location as the Default Geodatabase. It's necessary to change this location each and every time a geoprocessing tool is run in order to keep track of data as it's created.

Default Geodatabase

For those times you don’t designate a name and place for your output layer  facepalm , your data will automatically save to the default geodatabase. The default geodatabase is found in the same place on all machines with ArcGIS installed: C:\Users\username\My Documents\default.gdb. If you forget where your default geodatabase is, relaunching the tool and adding any file to the input line will automatically populate the output box with the path to your default geodatabase.  It's possible to change the default geodatabase per project, creating one in your Results (or simillar) folder, then telling the MXD where to default all tool outputs.  However, this change is for a single MXD, which is good if you're working on a large project, but kind of a hassle for simple, little, or quick projects.  It's also possible to permnanlty chagne the location of the default geodatabase, however, the end result is no different then the one set up by ArcGIS upon install.  Unless you do not have access to the Documents folder on that specific machine, it's generally accpetable to leave the default geodatabase as it is and learn the not very difficult path of "Documents > ArcGIS".

ArcGIS Help Article on Default Geodatabases

Green Circles, Yellow Exclamation Marks, and Red X’s, Oh My!

Geoprocessing tools all have some internal checks that occur when the tool is launched and again each time you populate a line in the dialog box. When the tool first launches, the required lines are marked with a green circle. This is the minimum amount of data the tool needs in order to do it’s job. As you move through each line of the dialog box, the tool continues to check your entry against the internal rules. If the entry you provided is acceptable, nothing happens (like the doctor says, “No news is good news”). If, however, the entry you’ve made doesn’t pass the test, the dialog box will put a yellow exclamation point (warning) or red x (error) next to the line’s description. Clicking on the symbol will result in a pop-up window explaining the warning or error. Warnings will (generally) allow you to continue with running the tool while errors will prevent the tool from being run and must be corrected before proceeding.

  • Geoprocessing - Required Field
    geoprocessing_required2
    The minimum fields required for a geoprocessing tool to complete it's job are marked with a green circle.
  • Geoprocessing - Warnings
    geoprocessing_warning2
    When the internal rule checks of a tool violates an internal rule, but the tool will still run, a yellow exclamation point marks the line with the warning. Click the symbol to read the associated warning.
  • Geoprocessing - Errors
    geoprocessing_error2
    When the internal rules of the geoprocessing tool are violated in such a way the tool cannot continue, the line will be marked with a red x. Click the symbol to read the associated error.
  • Geoprocessing - Warning and Error Message
    geoprocessing_warning_message2
    Click on the warning or error symbol to read the associated message for the line violating the internal rules.

“Show Help” and “Tool Help”

Every tool has an interactive tool built in that helps so very much when you are running tools - not just when you are learning, but for your whole GIS career to assure the proper input is going in the right box. To use the interactive help, first click the “Show Help” button in the lower right corner of the tool dialog box, then single click in any line to read the help for that input. To read the complete ArcGIS help article about the tool, use the “Tool Help” button found at the bottom of the interactive help slide-out window.

Figure 7.23: The Show Help and Tool Help Dialog Boxes
geoprocessing_tool_help

7.7.5: The Results Window

The Results Window shows two things: what tools are currently running and what tools have run in the past. Every time a tool is run in ArcGIS, it is added to a list the Results Window and that list is saved when only when the MXD is saved.  If an MXD is opened at a later time, the results window will show a "Current Session" and "Previous Sessions".  This is super handy to know what you did along the way and what your inputs and outputs were.  The Results Window can be opened from the Geoprocessing menu and docked by dragging and dropping it on a blue arrow. Once the window is open, use the plus signs to open each day/tool. From there, you can examine the specifics of each tool you ran in a session - the inputs and outputs, what the results were, and any additional messages produced.

  • The Results Window is not a replacement for your Methods document. Only tools with inputs and output are saved. Other actions are not. Sorry about that.
  • One of the most helpful things you can do with the Results window is the ability to relaunch any tool populated with the exact same parameters as the first time. You can then change the inputs and run the tool again without having to populate all of the boxes again. For example, if you ran a 200 meter buffer on a roads layer (output: Roads_Buffer_200m.shp), you can relaunch the tool, change the 200 to a 350, and simply change the _200m to a _350m without having to drive to the Results folder again. While the Buffer tool is a fairly easy tool to populate, there are other tools which take a lot more time and complex entries to run. In all cases, though, the ability to relaunch to tool with the same parameters is crazy handy.
Results_window

7.7.6: How Do I Know if My Tool is Running

"How do I know if my tool is running" is a very common question.  With ArcGIS (both ArcMap and ArcCatalog) there are three ways to know if your tool is running.  Prior to looking at the three methods, its important to first understand what it means to run a tool in the foreground, meaning the tool is running and the user can still interact with other portions of the software short of running another tool and running tools in the  background, meaning the tool takes over the software.  Tools running in the foreground is convenient for exploring other data and non-processor heavy tasks such as adjusting symbology while the background leads to less tool failure and a progress bar for the tool's progress, even though the trade off is no ability to do anything else until the tool is canceled or completed.

In the Background

  1. In the lower right hand corner of the ArcMap window
    • When you tool is running in background mode (the default), you will see the progress bar appear next to the scrolling coordinates in the lower right hand portion of the window.
      • It is known that, for some reason, the first tool you run in each ArcMap session (that is, every time you open a new or saved MXD) will not trigger the progress bar. Each subsequent tool will show the progress bar.
  1. In the "Results" window
  • When a tool is running, an hour glass appears in the 'Results' window. If your Results window is not docked, you can find it by clicking on Results  results_button  in the Geoprocessing Menu.
After a tool running in the background has finished, you will get a green check pop-up for a successful run and a red x pop-up for an unsuccessful run.
Geoprocessingsuccessbuffer_fail

In the Foreground

  1. When tools are set to run in the foreground, a progress window loads on top of ArcMap, functionally locking all other operations. You must either wait for the tool to finish or click "Cancel" to regain use of ArcMap.
    • When the tool is completed, the progress box will show a Successful time and date or a Failure time and date printed in red test.
    • In either case, the "Cancel" button will switch to a "Close" button
ForegroundrunningForegroundcomplete