Searching For Jim Gray A Technical Overview-Books Download

Searching for Jim Gray A Technical Overview

2020 | 8 views | 17 Pages | 1.83 MB

Searching for Jim Gray: A Technical Overview Joseph M. Hellerstein David L. Tennenhouse Electrical Engineering and Computer Sciences University of California at Berkeley

Copyright 2010 by the author s
All rights reserved
Permission to make digital or hard copies of all or part of this work for
personal or classroom use is granted without fee provided that copies are
not made or distributed for profit or commercial advantage and that copies
bear this notice and the full citation on the first page To copy otherwise to
republish to post on servers or to redistribute to lists requires prior specific
Searching for Jim Gray A Technical Overview
Joseph M Hellerstein and David L Tennenhouse
writing on behalf of a large team of volunteers
ABSTRACT nology to assist in search and rescue SAR 1 so that people whose
When Jim Gray disappeared at sea early in 2007 his friends and social network is not as well connected as Jim Gray s could under
colleagues quickly began discussing ways to mobilize their skills take analogous efforts In addition we hoped to review the tech
and resources to help rescue him That discussion evolved into an niques we used and ask how to to improve them further to make
unprecedented civilian search and rescue exercise which along the next search effort more effective To that end in May of 2008
with the US Coast Guard mission it augmented was eventually we convened a meeting of search participants and members of the
unsuccessful in locating Gray or his vessel US Coast Guard the day after a public tribute to Jim Gray at UC
In May of 2008 search participants and members of the US Berkeley This was the first opportunity for the virtual organiza
Coast Guard met face to face for the first time to discuss their ex tion that searched for Tenacious to meet face to face and compare
perience and try to extract constructive lessons This paper is an stories and perspectives
effort to distill some of that discussion for the computer science One sober conclusion the group quickly reached is that its spe
community We describe the emergent structure of the team and its cific lessons on maritime search and rescue could only have modest
communication the polytechture of the systems built during the impact as we detail in Section 2 With that established we still felt
search and some social and technical challenges that arose from that it would be constructive to cull lessons learned and identify
the experience technical challenges First maritime search is not a solved prob
lem and even though the number of lives to be saved is modest
each life saved is precious Second history shows that technolo
gies developed in one application setting often have bigger impact
1 INTRODUCTION in other settings We were hopeful that lessons we learned search
On Sunday January 28 2007 noted computer scientist Jim Gray ing for Gray could inform efforts during larger life threatening sce
disappeared at sea in his sloop Tenacious He was sailing single narios including civilian driven efforts toward Disaster Response
handed with plans to scatter his mother s remains near the Faral and SAR during natural disasters and military conflict And as part
lon Islands some 27 miles outside San Francisco s Golden Gate of the meeting we brainstormed a bit about the challenges of safety
As news of his disappearance spread through his social network and prevention as well
Gray s friends and colleagues began discussing ways to mobilize This paper is an effort to distill some of that discussion for the
their skills and resources to help authorities locate Tenacious and computer science community which has become increasingly in
rescue Gray That discussion evolved over days and weeks into an terested in disaster response e g efforts after the 2007 Kenyan
unprecedented civilian search and rescue exercise involving satel election crisis 2 and the 2010 Haiti earthquake 4 We docu
lites private planes automated image analysis ocean current sim ment the emergent structure of the team and its communication
ulations and crowdsourced human computing working in collab the polytechture of the systems built during the search and some
oration with the US Coast Guard The team that emerged included of the challenges that arose
computer scientists engineers graduate students oceanographers
astronomers business leaders venture capitalists and entrepreneurs
many of whom had never met each other before There was ample
access to funds technology organizational skills and know how The amateur effort to find Tenacious and its skipper started with
and a willingness to work around the clock optimism but little context on the task at hand We did not have
Even with these advantages the odds of finding Tenacious were any awareness of the practice and technology of SAR and only a
never good On February 16 2007 in consultation with the Coast vague sense of the special resources that Jim Gray s friends could
Guard and Jim Gray s family the team agreed to call off the search bring to bear on the problem
Tenacious remains lost to this day despite a subsequent extensive In this section with the benefit of hindsight we provide a back
underwater search of the San Francisco coastline 9 drop for our discussion of computer science challenges in SAR We
Jim Gray was famous for many things including a determina reflect first on the unique character of the search for Tenacious and
tion to transform practical experience and know how into scientific then describe the basics of maritime SAR as it is practiced today
challenges As the search for Tenacious wound down a number
of us felt that even though the effort was not successful on its own
2 1 The Tenacious SAR Effort
terms it offered a Jim Gray like opportunity for converting the par 1
The acronym SAR also refers to Synthetic Aperture Radar a re
ticulars of the experience into higher level technical observations mote imaging technology that was employed in searching for Tena
of more general interest One of our goals was to encourage efforts cious In this paper when we use the acronym SAR we refer ex
to democratize the ability for families and friends to use tech clusively to search and rescue
The search for Tenacious was in some ways a very unique effort 2 2 1 The Phases of SAR
and in others a typical case study in volunteer SAR The unique Search and Rescue technology is only needed when people get
ness had its roots in Jim Gray s persona In addition to being a into trouble From a public policy perspective it is cheaper and
singular scientist and engineer Gray was distinctly social cultivat more effective to invest in preventing people from getting into trou
ing friendships and collaborations across industries and sciences ble than investing in techniques to save them when they get into
The social network he built over decades brought enormous advan trouble Good discussions of boating safety can be found at http
tages to many aspects of the search in ways that would be very www uscgboating org and we cannot overemphasize the im
hard to replicate First the team that assembled to find Tenacious portance of safety and prevention in saving lives this was the con
included leaders in areas like computing astronomy oceanography sensus high impact topic at the end of the 2008 meeting in Berke
and business management Second due to Gray s many contacts in ley Appendix A
the business and scientific world the funds and resources available Even with excellent public safety SAR efforts are needed to han
to find were essentially unlimited including planes pilots satel dle the steady stream of low probability events raised by people
lite imagery and control of well provisioned computing resources who get into trouble The process of SAR involves four phases
Finally the story of a famous scientist gone missing attracted sig Notification Planning Search and Rescue Recovery
nificant media interest providing public awareness that attracted The first phase Notification plays a key role in shaping the
help with manual image analysis and tips on debris and wreckage rest of the process When notification of trouble occurs sufficiently
On the other hand a number of the general features that this quickly the Planning and Search phases become trivial and the
team wrestled with seem relatively universal to volunteer SAR ef SAR task can jump straight to the final Rescue Recovery phase
forts First the search got off to a slow start as volunteers emerged SAR gets harder when notification is delayed as it was in Jim
and got organized to take concrete action By the time all the ex Gray s case This leads to an iterative process of Planning and
pertise and efforts were in place the odds of finding a survivor Search Initial Planning is intended to be very quick and often
or even a boat were significantly diminished Second almost no consists simply of the decision to deploy planes to perform a vi
body involved in the volunteer search for Jim Gray had any expe sual sweep of the area where a boat is expected to be When
rience with SAR Finally at every stage of the volunteer search an initial alpha search is not successful the Planning phase be
the supposition was that it would last only a day or two more As comes more deliberate The second bravo search is planned via
a result there were disincentives to invest time in improving ex software that uses statistical methods to model probabilities of the
isting practices and tools and positive incentives for decentralized boat s location The US Coast Guard has developed a software
and lightweight development of tools and practices that had to be package for this process called SAROPS 3 SAROPS treats the
custom crafted boat location task as a probabilistic planning problem that it ad
If there are lessons to be learned from this effort they revolve dresses with Bayesian machine learning techniques Specifically it
around questions of both the uniqueness of the case and its univer accounts for prior information about weather and ocean conditions
sal properties The first category motivates efforts to democratize and the properties of the missing vessel as well as the negative in
techniques used to search for Tenacious some of which need not formation from the alpha search It uses a Monte Carlo particle fil
be as complex or expensive as they were in this instance The sec tering approach to infer a distribution of boat locations and makes
ond category motivates efforts to address common technological suggestions for optimal search patterns SAROPS is an ongoing
problems that arise in any volunteer emergency response setting effort that is updated with models of various vessels in different
states broken mast rudder missing keel missing etc The statisti
2 2 Maritime SAR cal training experiments to parameterize these models are expen
Given our experience the focus of the discussion in this paper sive exercises that place vessels underway to track their movement
is on maritime SAR As it happens maritime SAR in the United USCG continues to conduct these experiments on various param
States is better understood and more professionally conducted than eters as funds and time permit There is currently no equivalent
land based SAR Maritime SAR is the responsibility of a single software package or methodology available for land based SAR
federal agency the US Coast Guard USCG a branch of the US 2 2 2 Statistics and Discussion
Dept of Homeland Security By contrast land based SAR is man
aged in an ad hoc manner by local law enforcement authorities Mr Allen shared some of the US Coast Guard s statistics on
Our experience with USCG was altogether positive not only were Search and Rescue which we include in Table 1 Most cases
they eminently good at their jobs they were technically sophisti occur very close to shore and many involve land based vehicles
cated and encouraging of our often naive ideas providing advice going into the ocean The opportunities for technologists to assist
and coordination despite their own limited time and resources In with maritime SAR are modest In the US less than 1 000 lives
the United States at least maritime settings are a good incubator are lost in boating accidents each year and roughly 600 people
for SAR technology development and the USCG is a promising per year remain unaccounted for Among fatalities only 200 300
research partner As of now it is quite modestly funded so syn deaths occur after the Coast Guard is notified Some fraction of the
ergies and advocacy from well funded computer science projects unaccounted for cases include suicides Relative to other opportu
would likely be welcome nities to save lives with technology the margins for improvement
In hindsight the clearest lessons for the volunteer search team in maritime SAR are relatively small This reality frames the rest
were 1 the ocean is enormous and 2 the USCG has a sophisti of the discussion in this document with a focus on learning lessons
cated and effective maritime SAR program The meeting in Berke from this experience that apply both to SAR and hopefully to other
ley opened with a briefing from Arthur Allen an Oceanographer at important settings as well
the USCG Headquarters Office of Search and Rescue That office
oversees all US Coast Guard searches with an area of responsibil 3 COMMUNICATION AND COORDINATION
ity covering most of the Pacific half of the Atlantic and half of the As in many situations involving groups of people sharing a com
Arctic Oceans In the remainder of this section we review some of mon goal communication and coordination became major efforts
the main points Allen raised at the Berkeley meeting in the volunteer search for Jim Gray The efforts to organize these
Lives Lost
Lives Before CG After CG Unaccounted
Year Cases Saved Notified Notified Total for
2003 31 562 5 104 409 246 655 481
2004 32 517 5 555 502 277 779 676
2005 29 780 5 648 521 320 841 606
2006 28 316 5 260 476 310 786 667
Table 1 US Coast Guard Search and Rescue statistics
back office tasks were ad hoc and evolving In retrospect some emerged was to do task brokering between skilled or well resourced
interesting patterns emerged around various themes related to So volunteers and people who could take advantage of those assets
cial Computing including organizational development brokering This started happening in an ad hoc broadcast mode someone
of volunteers and know how and communication with the media would make a suggestion to the whole team and a volunteer would
and general public Many of these issues could be improved by broadcast an intention to follow up For example one member of
better software the team volunteered to raise funds to pay for pilots and search
planes if someone else would coordinate the operational aspects
But as the search progressed publicly offers of help came in from
3 1 Experience various unexpected quarters and coordination and task brokering
The volunteer effort began via overlapping email threads among became more complex Volunteers with science and military back
Gray s colleagues and friends in the hours and days following his grounds emerged with offers of specific technical expertise and
disappearance Various people exchanged ideas about getting ac suggestions for acquiring and analyzing particular satellite imagery
cess to satellite imagery hiring planes putting up missing persons Other volunteers offered to search in private planes and boats some
posters and so on Many of these tasks involved reaching out in times at serious danger to their own lives Many of these were
a coordinated and thoughtful manner to third parties But it was dangerous and discouraged by the team and the USCG Many
unclear who was hearing what information and who might be con volunteers offered to post Missing Sailor posters at marinas this
tacting third parties To solve that problem a blog called Tenacious required coordination There were a number of offers of psychic
Search was set up to allow a broadcast style of communication assistance Each offer took time from the communications coor
among the initial group of participants 11 Initially author dinator to effectively and diplomatically pursue and route onward
ship rights on the blog were left wide open This simple blog or deflect As subteams emerged within the organization this be
as bulletin board worked well for a day or two to coordinate the came easier the communications coordinator could skim an in
initial group involved in the search loosely documenting various bound message and route it to one of the other volunteer coordina
people s questions efforts skills and interests in a way that helped tors for follow up
define the group s efforts and organization Similar information brokering challenges arose in handling thou
Within a few days the story of Jim Gray s disappearance was sands of messages from the general public who were encouraged
widely known and the blog transitioned from being an in group by the media to keep their eyes open for boats and debris and report
communication medium to a widely read publishing venue for sta to a public email address This included a variety of information
tus reports on the search effort It served this role for the remainder along with many prayers and thoughts from well wishers Some of
of the volunteer search This function was quickly taken seriously the tips were clearly useful some clearly bogus or irrelevant Many
so authorship on the blog was closed to additional members and a were somewhere in between and given the sense of urgency at
separate Friends of Jim mailing list was set up for internal team the time it was often hard to decide whether to bring them to the
communications This transition led to an increased sense of orga attention of busy people the Coast Guard police Jim Gray s fam
nizational and social structure within the core group of volunteers ily and technical experts in image analysis and oceanography In
Over the next few days various individuals stepped into unoffi some cases tipsters got in contact repeatedly and it became neces
cial central roles for reasons of expediency unique skills or both sary to assemble conversations over the period of days to establish
The blog administrator evolved into a general communications co a tipster s credibility This became burdensome as the volume of
ordinator handling messages sent to a public email box for tips email grew
brokering skill matching for volunteers and serving as a point of The choice to involve the media in the effort generated work as
contact with outside parties Another volunteer emerged as a air well The goal of the media outreach was to keep the story of the
craft coordinator managing efforts to find pilot and route private disappearance in the news in hopes of maintaining public vigi
planes and boats to search for Gray A third volunteer took on the lance But not all media outlets were equally respectful of this goal
role of analysis coordinator organizing various teams working and some attempted to dig for dirt among members of the team
on image analysis and ocean drift modeling at a variety of orga After a short time Jim Gray s family asked that all media contact
nizations around the United States A fourth person was chosen be handled through a single volunteer with some advice from a
by Gray s family to serve as a central media coordinator the professional in public relations The team was quite disciplined in
sole contact for press and public relations These coordinator roles respecting this request particularly since it came from the family
were identified in retrospect and the role names were coined for
this paper simply to clarify discussion Individuals with manage
ment experience in the business world provided guidance along the 3 2 Discussion
way but much of the organizational development happened in an On reflection the organization s evolution was one of its most
organic bottom up mode interesting aspects Leadership roles emerged fairly organically
On the communications front an important task that quickly and subgroups formed with little discussion or contention over the
process or outcomes Some of the people had certain baseline com spondingly narrow field of view i e they can t see too far either
petencies e g the aircraft coordinator was a recreational pilot and side of straight down So if they don t just happen to be overhead
the analysis coordinator had both management experience and con when you need them you may have a long wait before they show
tacts with image processing experts in industry and government In up again A t this resolution it s strictly target of opportunity
general though leadership developed by one person stepping up to Undeterred the team pursued multiple avenues to acquire remote
take on a responsibility and others stepping back to let them do the imagery via connections at NASA and other government agencies
job and then jumping in to help as needed The grace with which and at various commercial satellite imagery providers The satellite
this happened is a bit surprising given the vigorously ambitious data teams at both Google and Microsoft directed us to their com
people that surrounded Jim Gray and the fact that the organization mercial provider Digital Globe Table 2 outlines the data sources
evolved largely over email The evolution of the team seems wor that were considered during the search for Jim Gray Figure 5
thy of a case study in ad hoc organizational development during is taken from the website that was put together at the time of the
crisis search to geo position and catalog the imagery that was eventually
It became clear that better software is needed to facilitate group made available 1 It shows the geographic boundaries of both
communication and coordination during crises By the end of the the data captures in the imagery and the Coast Guard s airborne
search for Tenacious a variety of standard communication meth search
ods were in use point to point email and telephony broadcast via As we discovered distribution of satellite data is governed by
blogs and web pages and multicast via conference calls wikis and national and international law We attempted from the very begin
mailing lists This mix of technologies was natural and expedient in ning of our discussion to get data from the SPOT 5 satellite but
the moment but made communication and coordination challeng this effort was halted by the US State Department which invoked
ing It was hard to work with the information being exchanged the International Charter on Space and Major Disasters to claim ex
which was represented in natural language text and stored in mul clusive access to the data over the study area retroactive to the day
tiple separate repositories As a matter of expediency in the early before our request We also learned when getting data from Digi
days the communications cordinator relied upon mental models of tal Globe s QuickBird satellite that full resolution imagery is only
basic information like who knew what information and who was available after a government mandated 24 hour delay before that
working on what tasks The emphasis on mental note taking made time Digital Globe could only provide reduced resolution images
sense in the short term but limited the coordinator s ability to share The first data acquired from Digital Globe s QuickBird satel
responsibility with others as the crisis watch extended from hours lite was focused well south of San Francisco near Catalina Island
to days to weeks Much of the coordination effort was probably There were very small odds that Tenacious would be found in that
similar to command and control challenges in military or emer region On the other hand it seemed important to begin exper
gency response teams with the key difference that it was being imenting with real data and seeing how the team could process
staffed by amateurs who were learning the process on the fly using it This early start turned out to be critical for getting the various
whatever tools were at hand pieces of the image processing pipeline in place and tested Even
Various pieces of this problem are addressable with well known data acquisition was challenging and Digital Globe was extremely
information management techniques But using current communi generous with their time and resources very aggressively produc
cation software and online services it remains difficult to manage ing and sharing their data As the search progressed Digital Globe
an evolving discussion that includes individuals restricted groups was able to acquire imagery solidly within the primary search area
and public announcements especially in a quickly changing cri and the image captures provided to our team formed some of the
sis mode of operation Identifying people and their relationships is biggest data products Digital Globe had ever generated Even so
challenging across multiple communication tools and multiple re the areas covered by the satellite captures were dwarfed by the air
cipients endpoints Standard search and visualization metaphors borne search conducted by the Coast Guard immediately after Gray
folders tags threads are not well matched to group coordination went missing Figure 5
The problem of brokering volunteers and tasks brings up further We were able to establish contacts at NASA regarding planned
problems In Appendix D 1 we list some specific challenges in this flights of the ER 2 flying laboratory aircraft over the California
domain in more detail In any software approach to these problems coast The ER 2 is typically booked on scientific missions and of
one issue is critical in an emergency people do not reach for new course requires resources to launch under any circumstances fuel
software tools So it is important to attack these challenges in a way airport time staffing wear and tear etc As it happened the ER 2
that augments popular tools rather than seeking to replace them was scheduled for training flights in the area where Tenacious dis
appeared Our contacts were able to arrange flight plans to pass
4 IMAGERY ACQUISITION over specific areas of interest and record various forms of digi
tal imagery a combination of fortunate circumstance and a well
When the volunteer search began our hope was to use our unique
connected social network Unfortunately a camera failure early in
skills and resources to augment the search efforts of the USCG
the ER 2 flight limited the data collection
with satellite imagery and private planes As we learned real time
In addition to these relatively rare imaging resources we char
search for boats at sea is not as simple as getting a satellite feed
tered private planes to fly over the ocean enabling volunteer spot
from a mapping service or borrowing a private jet
ters to look for Tenacious with naked eyes and record digital im
4 1 Experience agery This effort ended up being more limited than we originally
The day after Tenacious went missing Jim Gray s friends and
As we learned one cannot simply charter or borrow a private jet
colleagues began efforts to access satellite imagery and planes One
and fly it out over the ocean Light planes are not safe to fly far off
of the first connections made was to colleagues in Earth Science
shore Very few individuals maintain planes equipped for deep sea
who had expertise in remote sensing In an email message in the
search and flights over deep sea can only be undertaken by pilots
first few days one of the Earth scientists explained the difficulty
with appropriate maritime survival training Finally aircraft of any
of using satellite imagery to find Tenacious The problem is that
size require a flightplan to be filed and approved with a US Flight
the kind of sensors that can see a 40ft 12m boat have a corre
RADARSAT 1 A commercial Earth Observing Satellite EOS from Canada whose products are distributed by MDA Geospatial Ser
vices NASA has access to RADARSAT 1 Data in exchange for having provided a rocket to launch the satellite
http en wikipedia org wiki RADARSAT 1
Ikonos A commercial EOS operated by GeoEye USA http en wikipedia org wiki IKONOS
QuickBird A commercial EOS owned and operated by Digital Globe USA It was in use at the time by Google Earth and MS
Virtual Earth http en wikipedia org wiki QuickBird
ER 2 A high altitude aircraft operated by NASA similar to the US Air Force s U2 S reconnaissance platform http www
nasa gov centers dryden research AirSci ER 2 index html
SPOT 5 A commercial EOS operated by SPOT Image France http en wikipedia org wiki SPOT satellites
Envisat A commercial EOS launched by the European Space Agency Data products are distributed by the SARCOM consortium
created and led by SPOT Image http en wikipedia org wiki Envisat
Table 2 Some of the remote imagery sources considered during the search for Jim Gray An extensive list of Earth Observing
Satellites is kept at http en wikipedia org wiki List of Earth observation satellites
Service Station in order to cross the US Air Defense Identification image release based on context
Zone ADIZ that begins a few miles offshore As a result of these
limitations and many days of bad weather the number of private
overflights we arranged was small and all but one were close to
shore In this section we discuss the processing pipeline s and coordi
Another source of imagery that was considered was land based nation mechanisms used to reduce the raw image data to qualified
video cameras These could have more accurately established a search coordinates locations to which planes were dispatched for
time of departure for Tenacious beyond what we knew from Gray s a closer look This aspect of the search was largely data driven
mobile phone calls to family members on his way out The Coast and involved significant technical expertise It required much more
Guard operates a camera on the San Francisco waterfront pointed structured and tool intensive processes than those described in Sec
out toward the Golden Gate and the ocean but much of the tape for tion 3 On the other hand since time was short and the relevant
that day was in a state of white out rather than useful imagery expertise was quite specialized it also led to simple interfaces be
perhaps due to weather tween teams and their software The resulting amalgam of software
was not the result of a specific architecture in the usual sense of the
word archi chief techton builder A more apt term for the
4 2 Discussion software and workflow described here might be a polytechture the
The volunteer search effort was predicated on quick access to kind of system that emerges from the design efforts of many inde
satellite imagery This was surprisingly successful over 87 Gi pendent actors Though our experience is limited certain poly
gapixels of satellite imagery were acquired from Digital Globe alone techtural aspects of the effort appear interesting in retrospect and
within about four days of being captured Yet in retrospect we we describe them with the hope that the processes that emerged in
would have liked to get much more data with fewer delays In our effort may inform work in other contexts
Appendix refsec imageacq we review some of the limitations we
encountered and some ideas and challenges in improving the abil 5 1 Overview
ity to acquire imagery in life threatening emergencies Figure 1 is a simplified illustration of the critical path data and
Policy concerns come up naturally when discussing large vol control flow that emerged This diagram and the ensuing discussion
umes of remote imagery Various members of the amateur team in this section depicts the ad hoc pipeline that was developed for
voiced concerns about personal privacy during the process In ad Digital Globe s satellite imagery In Sections 5 3 1 and 5 3 2 we
dition national security issues arose including our inability to ac describe the Mechanical Turk pipeline that was primarily used for
cess SPOT 5 data While we were not given reasons for limitation NASA ER 2 overflight imagery RadarSat data was handled in
on access we speculate that certain obvious maritime features like a much less structured way a few members of the imagery team
military fleet movements are classified for good reason Mean looked at it and quickly decided it would not be helpful
while popular media sharing websites already provide widespread Before diving into the details it is instructive to work upstream
access to crowdsourced imagery and recent work has demonstrated through the pipeline from the final qualified targets back to the ini
the power of aggregating individual images into richer data prod tial imagery The objective of the exercise was to identify one or
ucts like panoramas and 3 d views While aggregation applications more sets of qualified search coordinates to which aircraft could
to date have largely confined themselves to benign settings like be dispatched lower right of Figure 1 To do so it was not suffi
tourism and ornithology maritime SAR applications e g monitor cient to simply identify the coordinates of qualified targets on the
ing marinas and shipping lanes seem closer to pure surveillance imagery we had to apply a mapping function to those coordinates
This raises understandable concerns and the policy issues are not to compensate for drift of the target from the time of image capture
simple Perhaps our main observation on this front was the need for to the flight time In our case this mapping function was provided
a contextual treatment of policy 8 balancing general case social by two independent drift teams of volunteer oceanographers one
concerns against specific circumstances for using the data in our based at the Monterey Bay Aquarium Institute and Naval Research
case for trying to rescue a friend While the search for Tenacious Lab MBARI NRL and another at NASA Ames Ocean Drift
was uniquely urgent for us similar life and death scenarios occur Modeling in Figure 1 described in more detail in Section 5 5
on a national scale with some frequency So it seems natural to The careful qualification of search coordinates was particularly
think about technical solutions that can both aggressively harvest important It was quickly realized that many of the potential search
and process imagery while provably respecting policies that limit coordinates would be far out at sea as mentioned in Section 4
Staging Common Operating Picture
FTP Server Georeferencing Map
Digital Globe SDSC U Texas
Image Batch Preprocessing
Expert Self Serve Website Novice
Image Image
Review Image
Review Review
Image Scoring Johns
Image Review
Naval Naval
Modeling NRL
Expert Expert
Target Drift
Declaration Modeling NASA
Target Qualification Ocean Drift Modeling
Figure 1 The rough dataflow for image processing Bold arrows represent images other arrows represent metadata
they would require special aircraft and crews Furthermore flying stitutions The teams also had to quickly craft interfaces to stitch
low altitude search patterns offshore and in single engine aircraft together the end to end workflow with minimal friction An inter
implied a degree of risk to the search team Thus it was incumbent esting variety of design styles emerged depending on a variety of
on the analysis team to carefully weigh this risk before declaring factors In the following subsections we describe the above com
a target to be qualified A key step in the process was the review ponents in greater detail this time from start to finish Since our
of targets by naval experts prior to their final qualification Target processing pipeline increased in sophistication during the course
Qualification in Figure 1 Section 5 4 of the search we typically only describe the state of the evolved
Prior to target qualification an enormous set of images had to be pipeline stages unless their evolution itself is of interest
reviewed and winnowed down to a small set of candidates that ap
peared to contain boats To our surprise and disappointment there 5 2 Pre processing
were no computer vision algorithms at hand that were well suited Once the image providers had data and the clearance to send it
to this task so this was done manually At first the image anal they typically sent notification of the availability via email to the
ysis effort was achieved by using the Mechanical Turk infrastruc image analysis coordinator together with an FTP address and the
ture at Amazon to coordinate volunteers from around the world header file that described the collected imagery the collection
mostly novices in the task of image analysis Subsequently a dis Immediately upon notification the pre processing team at Johns
tributed team of volunteer experts in image analysis including as Hopkins began copying the data to their cluster Meanwhile the
tronomers radiologists geologists machine vision researchers and common storage repository at the San Diego Supercomputer Cen
former military personnel performed the review function Image ter SDSC began ftp ing the data to ensure its availability and a
Review in Figure 1 Section 5 3 copy of the header was passed to a separate geo coordination team
Shifting to the start of the pipeline each image dataset required at the University of Texas which mapped the location covered by
a degree of pre processing prior to the human analysis of the im the collection and added it to a website The Texas website pro
agery this step was performed by astronomers at Johns Hopkins vided the overall shared picture of imagery collected and analyses
At the same time a separate team at the University of Texas took completed and was used by many parts of the search team to track
the image file headers and georeferenced them onto a map which progress and solicit further collections
was included in a web interface for tracking the progress of image
analysis Image Preprocessing Common Operating Picture and 5 3 Analysis Tasking and Result Processing
Staging in Figure 1 Section 5 2 During the course of the search two approaches to the parallel
In sum the eventual workflow was a distributed multi party pro processing of the tiled images were used In both of these ap
cess The pieces of this workflow were designed and built individ proaches small image tiles or smaller sub tiles had to be farmed
ually bottom up by independent volunteer teams at various in out to human analysts and the results of their analysis collated and
further filtered in order to avoid a deluge of false positives
5 3 1 Tasking Novices with Mechanical Turk graphic of the collection that was in turn georeferenced and super
The initial tasking approach was to use Amazon s Mechanical imposed on a map This allowed participants to prioritize their time
Turk service 6 to solicit and task a very large pool of anonymous by working on the most promising tiles first e g those that were
reviewers whose credentials and expertise were not known to us not heavily obscured by cloud cover
Mechanical Turk is a crowdsourcing marketplace for coordi The filtering component of this pipeline also operated some
nating the efforts of humans in performing simple tasks from their what more collaboratively and synchronously The expert analysts
own computers Given that the connectivity and display quality worked in shifts and at the end of each shift the sub team lead
available to these users was unknown the Mechanical Turk was ers gathered together and scored the most promising targets Al
configured to supply users with HITs work items that consisted though the scoring of extremely promising targets was performed
of a few 300 300 pixel image sub tiles each Using a template immediately this periodic and collective review promoted discus
image we provided of what we were looking for the volunteer was sion among the analysts allowing them to learn from each other
asked to score each sub tile for evidence of similar features and and adjust their individual standards of reporting
provide comments on artifacts of interest This was an exception
ally slow process because of the number of HITs required to pro 5 4 Target Qualification
cess a collection The analysis coordinator examined reports from the analysis pipelines
In addition to handling the partitioning of the imagery across to identify targets for submission to the qualification step In the
volunteers Mechanical Turk bookkeeping was used to ensure that Mechanical Turk case this involved spending a few hours sifting
each sub tile was redundantly viewed by multiple volunteers prior through the output of the second Mechanical Turk stage Once the
to declaring the pipeline complete At completion and at check expert pipeline was in place the coordinator was examining only a
points along the way the system also generated reports aggregating few filtered and scored targets per shift
the results received concerning each sub tile Promising targets were then submitted to a panel of two review
False positives were a significant concern even in the early stages ers each of whom had expertise in identifying engineered artifacts
of processing So a virtual team of individuals who identified them in marine imagery The analysis coordinator isolated these review
selves as having some degree of familiarity with image analysis ers from each other in part to avoid cross contamination but also
though typically astronomical or medical imagery rather than satel to isolate them from individually carrying the weight of a poten
lite imagery was assembled to perform this filtering In order to tially risky decision to initiate a search mission this would avoid
distribute the high scoring sub tiles to them the image analysis overly biasing them in a negative direction Having discussed their
team configured an iterative application of Mechanical Turk acces findings with each of the reviewers the coordinator made the final
sible only to this sub team with the high scoring sub titles from decision to designate a target as qualified
the first pipeline fed into it The reports generated by this second Given the dangers of deep sea flights this review step included
pipeline were then used by the coordinator to drive the target qual an intentional bias it imposed less rigorous constraints on targets
ification process This design pattern of an expertise hierarchy that had likely drifted close to shore than on those further at sea
seems likely to have uses in other crowdsourcing settings
5 5 Drift Modeling
5 3 2 Tasking Experienced Users Relatively early in the analysis process an individual with ma
A significant cluster of our image reviewers were co located at rine expertise recognized that should a target be qualified it would
the astronomy research center at Johns Hopkins These volunteers be necessary to estimate its movement since the time of image cap
who had ample expertise bandwidth high quality displays and a ture They formed a drift modeling team that ultimately consisted
sense of personal urgency realized they could process the im of two sub teams of oceanographers with access to two alternative
agery much faster than novices scheduled by Mechanical Turk could drift models As the image processing proceeded these sub teams
This led to two modifications in the granularity of tasking worked in the background to parameterize their models on an ongo
These individuals were accustomed to looking for anomalies in ing basis with weather and ocean surface data during the course of
astronomical imagery and were typically able to rapidly display the search Thus once targets were identified the sub teams were
scan and discard sub tiles that were 3 4 larger than those pre able to rapidly estimate the likely drift patterns
sented to amateurs This led to an individual processing rate of The drift models utilized a particle filtering approach of virtual
approximately one larger sub tile every 4 seconds including the buoys that could be released at an arbitrary time and location and
tiles that required detailed examination and the entry of commen for which the model would then produce a projected track and
tary as compared to the 20 30 second turnaround for each of the likely endpoint at a specified end time In practice one releases a
smaller sub tiles in the Mechanical Turk HITs The overall im string of adjacent virtual buoys to account for the uncertainty in the
provement in productivity over Mechanical Turk was in fact con initial location and the models sensitivity to local effects that can
siderably better than these numbers indicate because the analysts have fairly large impacts on buoy dispersion Figure 2 illustrates
experience reduced the overhead of redundant analysis and their the results of one such model The availability of two independent
physical proximity facilitated communication and cross training models with multiple virtual buoys per model greatly increased
A further improvement was that the 256 sub tiles within each our confidence in the prediction of regions to search
8k 8k pixel tile were packaged into a single zip file Volunteers It is worth noting that although these drift models were devel
could then use their favorite image browsing tools to page from oped by leading scientists in the field the results often had signifi
one sub tile to the next with a single mouse click To automate the cant uncertainty This was particularly true in the early part of the
tasking and results collection this team used scripting tools to cre search when drift modeling was used to provide a search box
ate a web based visual interface through which they and similarly for Gray s boat and had to account for many scenarios including
equipped volunteers worldwide could visually identify individual whether the boat was under sail or with engines running These sce
tiles that required work download them and then submit their re narios had very large uncertainty and led to large search boxes By
ports the time the image processing and weather allowed for target qual
In this interface tiles were super imposed on a low resolution ification the plausible scenario was reduced to a boat adrift from
Figure 3 Cropped version of an ER 2 satellite image flagged
Figure 2 A drift model output showing an eventual landing by an inexperienced Mechanical Turk volunteer
point in Monterey Bay
nical level this provided some validation of the targeting process
a relatively recent starting point Our colleagues in Oceanography
and the USCG tell us that the problem of ocean drift modeling mer 5 7 Discussion
its more research and funding it would seem to be a good area for The image processing effort was the most structured and techni
collaboration with Computer Science cal aspect of the volunteer search In trying to cull lessons from it
The drift modeling team developed its own wiki based workflow we highlight three rough topics the polytechtural design style
interface for their tasks The analysis coordinator was given a web networked approaches to search and the need for civilian com
site where he could enter a request to release virtual drifters near puter vision research targeted at disaster response applications In
a particular geolocation at a particular time Requests were pro Appendix C 2 we also discuss organizational issues that arose in
cessed by the two trajectory modeling teams and the resulting anal this more structured aspect of the search
ysis including maps of the likely drift patterns were posted back to
the coordinator via the website Geolocations in latitude longitude 5 7 1 Polytechture
are hard to transcribe accurately over the phone so this helped en The software development and deployment process that emerged
sure correct inputs to the modeling process was based on groups of experts working independently Some of
the more sophisticated software depended on pre existing expertise
5 6 Analysis Results and componentry e g parallelized image processing pipelines
The goal of the analysis team was to identify qualified search and sophisticated drift modeling software By contrast some soft
coordinates During the course of the entire search numerous tar ware was ginned up for the occasion building on now standard
gets were identified but only two were qualified One was in ER web tools like wikis scripting languages and public geocoding in
2 flyover imagery near Monterey and was originally flagged by terfaces Not too many years earlier these tools were typically
Mechanical Turk volunteers Figure 3 The other was in Digi used for low throughput rapid prototypes They are now increas
tal Globe imagery near the Farallon Islands and was identified by ingly used for production systems and in our case for a hybrid
a member of the more experienced image processing team Fig rapid production system a system that had to be built quickly
ure 4 Although the low number might suggest our filtering of and also get a real job done It is encouraging to see how much
targets was overly aggressive we have no reason to believe that was enabled by these lightweight tools
potential targets were missed Our conclusion is simply that the Redundancy was an important theme in sensitive steps of the
ocean surface is not only very large but also very empty process Redundant ftp sites ensured availability redundant drift
Once qualified these two targets were then drift modeled so that modeling teams increased confidence in predictions and redundant
coordinates for search boxes could be identified For the first of target qualification by experts provided both increased confidence
these targets the drift models indicated that the target should have and limits on responsibility bias
washed up on shore in Monterey Bay Because this was a region Perhaps the most interesting aspect of this process was the vari
very close to shore it was relatively easy to send a private plane to ety of interfaces that emerged to couple these independent compo
the region which we did nents A number of these are described above the cascaded Me
The second target was initially not far from the Farallon Island chanical Turk interface for hierarchical expertise in image analysis
and both models predicted it to have remained relatively close by the ftp email scheme for data transfer and staging the web based
Given our knowledge of Gray s intended course for the day this Common Operating Picture for geolocation and coarse grained
was a very promising target so we arranged a private off shore task tracking the self service checkin checkout interface for ex
search and rescue flight Although we did not find Tenacious we pert image analysis the decoupling of image file access from im
did observe a few fishing vessels of approximately Tenacious size age browsing software and the transactional workflow interface for
in the area It is possible that the target we identified was one of drift modeling The variations in these interfaces seemed to arise
those vessels Though the goal of the search was not met at a tech from both the tasks at hand and the people involved Sometimes a
Figure 4 A Digital Globe satellite image flagged by an experienced volunteer On the left it is in its original format gray with clouds
and hard to reproduce on paper On the right is an enhanced cropped version of the upper center of left image The white spot was
confirmed by naval imagery experts to very likely be a boat of a size consistent with Tenacious
structured process seemed necessary e g the Mechanical Turk in 5 7 2 Networked Search
terface for tasking amateurs and sometimes it was desired by one It is useful to reflect on the relative efficacy of this componen
party e g the web interface to the drift team Other tasks evolved tized polytechtural design approach compared to more traditional
ad hoc e g data staging and the bulk of the initial communication and deliberate strategies The amateur effort was forced to rely
and coordination tasks on loosely coupled resources and management operating asyn
The evolution of the web over the last decade enabled this poly chronously at a distance By contrast the Coast Guard operates
techtural design One key aspect of this is the standardization of in much more prepared and tightly coupled manner performing
software tools and design patterns the commonality of web based nearly all the search steps at once in real time after a planning
interfaces to remote systems and services the availability of easy phase maps out the maximum radius that a boat can travel trained
to use scripting languages and the widespread standardization of officers fly planes in carefully plotted flight patterns over the rele
image formats and viewers Perhaps most remarkable were the vant area using real time imaging equipment and their naked eyes
interactions between public data and global communications The to search for targets This allows the Coast Guard searches to be
manufacturer s specifications for Tenacious were found on the web nimble and focused But a componentized approach may offer cer
aerial images of Tenacious in its berth were found in publicly avail tain advantages in scaling and evolution since it does not rely upon
able sources Google Earth Microsoft Virtual Earth and a former tightly integrated and relatively scarce human and equipment re
owner of Tenacious discovered the blog in the early days and pro sources
vided additional photos of Tenacious under sail These details were A compromise architecture is what might be called Networked
helpful for parameterizing drift models and for providing tem Search Infrastructure in which the relevant components of the search
plate pictures of what analysts should look for in their imagery process are decoupled in a manner akin to our volunteer search but
Despite its inefficiencies the use of Mechanical Turk volunteers to more patiently architected evolved and integrated As a simple ex
bootstrap the image analysis process was remarkable particularly ample Coast Guard imagery experts need not be available to board
the benefit of having many people redundantly doing data analysis search planes nationwide a remote image analysis team could ex
Beyond the Turk pipeline an interesting and important data clean amine streaming and archived footage from multiple planes in
ing anecdote occurred on the blog in building the template picture different locales In fact weather hazards and other issues sug
for Tenacious Initially a family member identified Tenacious in gest removing the need for people to board planes entirely im
the satellite image geographically by identifying Gray s boat slip agery could be acquired via satellites and unmanned aerial vehi
in the marina In subsequent discussion one of the analysts no cles UAVs which are improving rapidly A componentized ap
ticed that the boat in that image did not match Tenacious online proach takes advantage of the independent evolution of technolo
specifications and after some reflection the family member con gies and the ability to train domain experts on each component
firmed that Gray had swapped boat slips some years earlier and more quickly image analysis tools can improve separately from
the online satellite image predated the swap Few if any of these imaging equipment which can evolve separately from the devices
activities would have been possible 10 15 years ago not because flying the equipment This kind of networked componentry and
of the march of technology per se but because of the enormous expertise is becoming relatively common in military settings and
volume and variety of information that is placed online and the in medical imaging in the public sector It seems useful to explore
rapidly increasingly subset of the population that is habituated to these ideas further for other civilian settings like SAR especially
using that information
when considering applications to widespread disaster response for Tenacious and its skipper in addition to incredible work by
the USCG there were other quieter efforts among Jim Gray s col
5 7 3 Automated Image Analysis leagues and family that happened outside the public eye Although
The volunteer search team included experts in image processing we were frustrated in achieving our goal the work that was done
for astronomy and consulted with experts in computer vision as along the way was remarkable in many ways and the tools and sys
well The consensus early on was that off the shelf image recog tems that were developed so quickly by the amateur team actually
nition software would not be sufficiently accurate for the urgent worked startlingly well This was due in part to the incredible show
task of identifying boats in satellite imagery of the ocean Dur of heart and hard work from the volunteers for which many people
ing the course of the search a number of machine vision experts will always be grateful It is also due to the quickly maturing con
examined the available datasets and concluded that it was not of vergence of people communication computation and sensing on
sufficient quality for automated processing though that may have the Internet Jim Gray was a shrewd observer of technology trends
been because we did not have access to the raw bits obtained by and what they suggest for the next important steps in research We
the satelite based sensors Although some experts attempted a sim hope that the search for Tenacious sheds some light in those direc
ple form of automated screening by looking for clusters of adjacent tions as well
pixels that stood out from the background even those efforts were
relatively unsuccessful 7 REFERENCES
It would be good to know if this problem is inherently hard or
1 Center for Space Research University of Texas Austin
simply requires more focused attention from computer vision re
Emergency response 2007
searchers The problem of using remote imagery for search and
http magic csr utexas edu tenacious htm
rescue is a topic where computer vision would seem to have a lot
to offer especially at sea where obstructions are few 2 Joshua Goldstein and Juliana Rotich Digitally networked
technology in Kenya s 2007 2008 post election crisis
Technical Report 2008 09 Berkman Center for Internet and
6 REFLECTION Society at Harvard University September 2008
Having described the amateur SAR processes cobbled together 3 U S Coast Guard Search and Rescue Optimal Planning
to find Tenacious we can return to some of the issues we set out to System SAROPS 2009 http www uscg mil
discuss initially when we the group of searchers met in Berkeley in acquisition international sarops asp
2008 4 Jessica Heinzelman and Carol Waters Crowdsourcing crisis
Some of the lessons learned along the way are mentioned above information in disaster affected Haiti Technical Report
more specific challenge problems are presented in Appendix D As Special report 252 United States Institute of Peace October
to whether this kind of effort could be democratized there is rea 2010
son to be optimistic on the computational front Computer hard
5 KDD Cup Center 2009
ware has continued to shrink in price since early 2007 and cloud
http www sigkdd org kddcup index php
services are commoditizing access to large computational clusters
6 Mechanical Turk 2007 http www mturk com
it is is now affordable to get rapid access to enormous comput
ing resources without any social connections or up front costs By 7 Claire Cain Miller and Jenna Wortham Technology aside
contrast custom software pipelines to perform tasks like image most people still decline to be located The New York Times
processing drift modeling and command and control coordina August 29 2010
tion are not widely available This is not an inherent problem it 8 Helen Nissenbaum Privacy in Context Technology Policy
is an area where small teams of open source developers and soft and the Integrity of Social Life Stanford Law Books 2009
ware researchers could have significant impact The key barrier to 9 E Saade Search survey for S V Tenacious Gulf of
democratization of these efforts may be access to data It is un Farallones and approaches to San Francisco Bay ACM
clear whether data providers like satellite imagery companies or SIGMOD Record 37 2 June 2008
plane owners can support large scale near real time feeds of public 10 D Song and K Goldberg Networked robotic cameras for
safety imagery And it is also not clear from a policy perspective collaborative observation of natural environments In
whether this is an agreed upon social good This topic deserves International Symposium of Robotics Research ISRR
more public discussion and technical investigation This discus October 2005
sion can be accelerated by low fidelity open source prototypes that 11 Tenacious Search blog 2007
make the best of the data that is publicly available e g by aggre http www openphi net tenacious
gating volunteer webcams as we outline in Appendix D Some
times the best way to democratize access to resources is to build APPENDIX
disruptive low fidelity prototypes
The volunteer search team s experience reinforces the need and A PREVENTION
opportunity for technical advances in Social Computing In the In Section 2 2 we noted that the best way to improve the chances
end the team exploited technology for many uses not just the of Rescue and Recovery are to provide Notification of distress early
high profile task of locating Tenacious in images from space Mod taking the Search and Planning phases out of SAR In addition to
ern networked technologies enabled a group of acquaintances and improving the ability to locate individuals a major focus of the
strangers to quickly self organize coordinate build complex work US Coast Guard is to shrink the time to notification which can
ing systems and attack problems in a data driven manner Along radically decrease the complexity of Planning Search and Rescue
the way we found plenty of room for technological improvement Recovery In this section we discuss some standard practices in
including the limitations of standard email and blogging tools and boating safety and ways that these can be improved via both policy
design patterns in architecting workflows to coordinate volunteers changes and technology
with differing skills More work is needed here
The efforts documented here are not the whole story of the search A 1 Background and Experience
cost effective approach to improve marine safety by a large margin
Radio beaconing devices such as the Emergency Positioning Ra Another major frustration during the search for Tenacious was
dio Beacon EPIRB and Personal Locator Beacon PLB are avail that Gray s cellphone signal was available to his cellular telephone
able to sailors producing signals on satellite channels monitored provider but that information was not available to the volunteer
by agencies including the US Coast Guard Many of these de search team Now just a few years after Gray disappeared there
vices are activated when a hydrostatic switch hits the water when are many services for sharing or acquiring the geolocation of GPS
mounted properly they will automatically send out a radio beacon enabled smartphones location sharing services like Foursquare and
that is detected via a geo positioning satellite known as SARSAT Google Latitude or on demand phone location services like Ap
Search and Rescue Satellite Aided Tracking Cheaper models ple s Find My Iphone tool However these are still relatively
have a manual switch In the United States EPIRBs are legally primitive in their access control policies which would discourage
required of most commercial vessels They are not mandatory for reliable use We return to this issue in Appendix D 3
recreational boats though they are strongly encouraged Many
recreational boaters do not own EPIRBs or if they do own one
it may be mounted incorrectly or not kept in working order
In addition to emergency communication devices most boaters The satellite data we received from Digital Globe had already
today carry cell phones as a matter of course and keep them in been partially processed into visual form i e it was not raw sensor
working condition Cell phones are useful not only for communica data However it was processed by a pipeline tuned for land based
tion but also for geolocation by third parties tracking their signals imagery and was not easily usable by humans due to the lack of
Also many smartphones connect periodically to Internet services contrast in marine imagery Thus the pre processing team applied
like mail and location servers signalling that the device is still processing steps to stretch the contrast within the pixels across a
functioning wider range of color values
Tenacious was outfitted with an EPIRB but no signals were re The imagery also had to be divided up into manageable chunks
ceived from it the day that Jim Gray went missing There are many Each Digital Globe collection was delivered to us as a longitudi
possible explanations for this failure It may simply have malfunc nal strip sometimes divided into sub strips that was divided into
tioned or it may not have been properly maintained In addition 8k 8K tiles Each tile was in turn divided into either 256 sub
the location of the EPIRB on Tenacious was not fail safe it could tiles of 512 512 pixels for processing by experienced users as
possibly have become trapped in the hold as the boat went un described in Section 5 3 2 or a larger number of 300 300 pixel
der By contrast Gray s smartphone continued to connect to email sub tiles for crowdsourced processing using Mechanical Turk as
servers at his employer for a few hours after he was last in verbal in Section 5 3 1 These collections were delivered as either 0 82
contact with family and friends m pixel panchromatic images or 0 6 m pixel pan sharpened natural
We do not know whether Gray s cellular telephony provider had color images the latter only after a 24 hour hold The pixel depth
information useful for constraining the possible location of Tena was either 8 or 16 bits pixel but the useful depth was typically only
cious they were unable or unwilling to share the required informa a few bits per pixel due to the low contrast of over the water im
tion with private parties likely because of legal guidelines about agery commercial satelite image processing is typically optimized
customer privacy In a late phase of the search the volunteer search for land based imagery
team discovered that there are mobile cell tower devices that can The Nasa ER 2 collection was captured using a Cirrus Digi
be placed into planes and flown over a region to listen for signals tal camera system and consisted of 34 tiles of 4500 4500 pix
from cellular handsets Apparently such devices were used in the els 8 8 8 bit RGB which included a 10 overlap between adja
aftermath of Hurricane Katrina Extraordinary efforts were made cent images These were divided into sub tiles of 300 300 pixels
to get one of these devices deployed over the region where Tena for crowdsourced processing using Mechanical Turk The nominal
cious had disappeared but the devices are apparently quite rare resolution was 2 2m though the actual resolution was substantially
and the amateur team did not succeed in getting access to one in better due to a difference between the aircraft s actual altitude and
time despite the involvement of highly influential people Cellular the nominal altitude cited for the sensor
telephony was one of the only technology industries where even Ten days after Gray s disappearance SPOT Image approached us
Jim Gray s circle of colleagues was unable to pull sufficient strings with the possibility of our purchasing Envisat data of the California
to make things happen in time coast for relevant dates This was not deemed worthwhile by that
A 2 Discussion
EPIRBs and similar technology can make it easy to search for C FURTHER DISCUSSION
missing boats where they often fail is in proper deployment This
raises a natural policy direction which was one of the clearest C 1 Image Acquisition
lessons that came out of the 2008 Berkeley meeting of the volunteer During our acquisition of imagery delays arose as the result of
search team mandate the installation of maritime safety technol three main factors scheduling latencies image processing over
ogy in a failsafe way as we have done with other technologies like heads and policy challenges In addition it was clear that more
automobile airbags It is both possible and inexpensive relative nimble techniques could be developed for remote sensing during
to the cost of a boat to require EPIRB like technology to be inte emergencies We briefly summarize these data acquisition issues
grated into boat construction Better solutions could also be devised and then discuss some lessons and challenges for future efforts sim
to enforce effective EPIRB deployment for pre existing boats and ilar to ours
to enforce the use of PLBs in small planes backcountry hiking
registration and related settings In the US in particular there are C 1 1 Scheduling
objections in some quarters to mandatory tracking technologies As we noted satellites and planes have specific hardware and
focusing on individual liberties and electronic privacy issues so workload constraints that introduce inherent latencies in position
this issue will need significant advocacy It is however the most ing their sensors Another factor is weather as it happened this
was the biggest source of delay in searching for Tenacious since like a promising avenue for improving this picture Of course the
foul weather grounded search planes for many days and cloud Coast Guard maintains planes and crews for maritime SAR So as a
cover obscured much of the satellite imagery In addition remote matter of national policy one desirable outcome would be to better
imaging hardware tends to get very high utilization since the large equip the national air fleet for SAR This may require lobbying on
fixed cost of the equipment is typically amortized across many behalf of USCG or other agencies
tasks In our case scarcity of resources did not impede our progress Regardless of the capabilities of the government it is hard for
substantially thanks to generous donations of time and resources private citizens to get access to government operated remote imag
from organizations like Digital Globe and a number of volunteer ing assets Most such assets are managed by the military and even
flight crews However this is a case where democratization of our in a high profile case like this one it was not possible to get direct
experience would require new approaches even with the best of civilian contact with the military on this topic2 Meanwhile we
intentions Digital Globe could not provide this level of effort on a expect that many of the military s key assets are not traditionally
regular basis pointed at domestic targets nor at empty ocean so in the Tena
Latency and workload management are familiar topics in com cious case the military equipment may not have been easy to utilize
puter science and the natural solution is to enable more informed Some government imaging assets are managed by non strategic
global scheduling to assign imaging assets to worthy tasks Given agencies like NASA and the National Guard In these cases it is
that various entities own such assets some kind of clearinghouse possible to take advantage of the assets if the costs are sufficiently
perhaps akin to travel reservation services would be useful to pro low For example we got lucky in our effort to access an ER 2
vide better access to remote imaging equipment Realistically the due to the training mission mentioned in Section 4 It would be nice
market for this service may currently be too small to motivate par to see this kind of opportunity systematized for the public good in
ticipation by satellite providers A crowdsourced bootstrapping ef a less ad hoc manner
fort might be more fruitful as we outline in Appendix D Another option to consider is the use of unmanned aerial ve
hicles UAVs including planes helicopters and blimps which
C 1 2 Image Processing Overhead may be cheaper and safer to operate at a large scale than human
The second source of delay image processing was a software piloted planes This seems particularly useful for fine grained sens
issue that could be easily improved if not substantively avoided ing in time and space during widespread disaster response This
As we discuss in Appendix B satellite image providers typically raises challenges in robotics and computer vision in addition to the
perform significant image processing before they release data In scheduling and policy issues mentioned above According to con
our case this had multiple negative impacts First computational tacts at the USCG the use of UAVs near shore in densely populated
bottlenecks at the provider introduced delays that could have been areas is highly problematic due to air traffic but there are other set
avoided by using the much larger computational facilities avail tings e g in the Pacific Islands where they could be very useful
able to the volunteer team In the last three years cloud comput both for SAR and for other USCG tasks like fisheries enforcement
ing services have made access to such resources much easier so Independent of whether the sensors are carried on UAVs or a
custom image processing should be an increasingly affordable ser larger number of smaller aircraft there is the opportunity to de
vice decoupled from data acquisition Second the provider s effort couple the sensors themselves from the processing of the sensor
to convert the data to human readable imagery was tuned to land derived information as discussed in Section 5 7 2
imagery so the volunteer team had to reprocess the images to im
prove the contrast within the color range applicable to at sea im C 2 Organization of the Analysis Teams
ages Finally much of the underlying contrast range had been lost While the entire volunteer search effort was a case study in rapid
in the initial conversion at the provider access to the raw bits that organizational development the technical efforts employed by the
would have facilitated machine vision processing were lost in the imagery and drift modeling teams were somewhat more structured
provider s image processing This was clearly a case where a very due to the technical nature of their tasks The team structure emerged
well intentioned effort to make things easier for the volunteer team relatively organically bottom up via domain expertise In partic
went astray due to the lack of a prior working relationship amongst ular Jim Gray s Astronomy colleagues had expertise equipment
the parties i e the provider and the volunteers lacked knowledge and software experience with bulk image processing and organized
of each other s working assumptions and capabilities This would themselves Gray s Oceanography colleagues had expertise and soft
be alleviated by the availability of widely known open source soft ware experience in drift modeling and again formed their own or
ware ganization Other parties that were contacted to help exhibited ex
treme generosity The staff at Digital Globe and Amazon Mechan
C 1 3 More Sensing ical Turk were remarkably giving of their time and resources and
As Figure 5 demonstrates the coverage of the remote imagery trusting of the volunteers efforts 3
acquired by the volunteer team did not compare in scope to the The teams were very loosely guided by the analysis coordina
overflight area covered by the Coast Guard This fact only became tor who undertook a number of tasks He coordinated the flow
clear to the amateur team towards the very end of the process and of information between teams e g satelite imagery provider pre
was both heartening and humbling we were happy to learn more processing team at JHU analysis teams at JHU and elsewhere geo
about the effectiveness of the Coast Guard even as we realized that referencing and drift modeling provided a common interface to
our high tech effort covered a smaller area On the other hand the individuals supporting higher layer functions of the search
the Coast Guard was encouraging of our efforts to leverage remote 2
sensing data and there is general agreement that there are lessons In the search for Tenacious efforts were made to directly connect
the relevant Coast Guard officers with military contacts known to
to be learned from the experience the volunteer team It remains unknown whether those contacts
One obvious challenge raised by our effort is to enable more occurred and if so whether they affected the case
and more targeted image capture by making more hardware avail 3
We stress that this list is not exhaustive Many other groups of
able Planes are cheaper nimbler and more plentiful than satellites fered help but were not called upon to the same degree And many
and they can often fly below obscuring cloud cover so they seem individuals at institutions not listed here were critical to the effort
described earlier e g organization of search aircraft management Visualization and search in communications Develop visu
of the public web site communication with the Coast Guard etc alization and search interfaces appropriate to crisis manage
and attempted to anticipate future needs e g the recruiting of ex ment Standard interfaces for visualization and search in com
perts with naval imagery experience who could provide a final re munications media email blogs mailing lists etc are a
view of the results Due to urgency of the crisis and the loose poor match for the activities described above Rather than
organization the search coordinator used a relatively hands off focusing on folders tags and message threads group coordi
non prescriptive approach to team management and tasking This nation calls for a focus on principals people organizations
seemed to work well in part due to a diversity and excess of human roles topics and events In the Gray SAR search common
capital When a volunteer team announced to the coordinator that query patterns included mnemonic expansions like who
they were pursuing a particular task there was little reason to dis was that person who sent the email with all the complex satel
courage their efforts many such activities were afoot in the early lite terminology aggregate analyses of messages e g who
days and the ones described here gelled from a subset of those is at the center of the discussion about putting up posters
A key example of this was the image geocoding process at Texas temporal analyses like when was the last message discussing
Although the relative value of that effort was not clear to the coor red dye reputation analyses like has this person demon
dinator at first it quickly became essential to coordinating all the strated expertise on this topic previously and so on Inter
imaging efforts and was very helpful in bringing new volunteers faces need to be very efficient in our experience the effort to
up to speed construct longer sequences of low level queries was stymied
by the latency of interaction with modern mail and web tools
C 3 A Note on Uncertainty often painful seconds per click
Much of the data in the search process was noisy or uncertain
In some cases this was due to properties of sensing equipment In
D 2 Image Acquisition and Analysis
others it was due to the use of statistical methods like drift modeling The specific technical challenges we encountered in image ac
to predict or bound unknown information And as is to be expected quisition and analysis can be combined into a larger scale chal
there were also cases of human error in managing and interpreting lenge the development of a publicly available system of similar
the available data some of which were caught and rectified some capabilities targeted for use in both personal and large scale crises
of which were managed as part of the design e g the redundancy by citizens of modest means From a software engineering per
in crowdsourced image analysis and very likely some of which spective it would be helpful to compare a patiently architected net
were unaccounted for In retrospect it seems natural that the inputs worked SAR infrastructure like this to the rapidly developed poly
and processes in a volunteer crisis response setting will have a high techture used to search for Tenacious
degree of noise which should be modeled and mitigated as much In full the goal is to develop a cost effective and reliable end
as possible to end solution that includes all the steps of our target qualification
pipeline image acquisition via a variety of manned and unmanned
equipment staging and processing of image streams interfaces for
D TECHNICAL CHALLENGES human analysis markup and scoring dashboards to monitor work
flow progress and outstanding tasks To ensure rapid cost effective
D 1 Social Computing spin up of the service in times of crisis it should be deployed in
Three specific technical challenges distill many of the issues that a hosted environment and be able to quickly scale up In times
arose for us in communication and coordination Solutions to these when it is not needed it should maintain an operational standby
would have broad applicability in SAR and emergency response mode that regularly drives data through the entire pipeline ensur
and in general use as well ing that all the components are in a constant state of readiness This
includes not only technical readiness but also regular checks on
Ad hoc integration of communications software Develop tools potential organizational barriers like bureaucratic authorization to
that break down barriers between popular software packages access relevant information and resources
for email mailing lists bulletin boards and blogs to provide The San Francisco Marina from which Tenacious embarked is
features useful for crisis management and group coordination an example environment where a system of this sort could be boot
Desired features include search entity extraction and reso strapped The marina is surrounded by affluent tech savvy home
lution people organizations tasks timeline and informa owners with views of boat traffic who would likely be willing to
tion flow analyses and social network features like centrality participate and contribute in a crowdsourced public safety effort
and reputation assessment Ideally it should be possible to Such a community of volunteers could contribute to and host an ar
quickly integrate multiple endpoints as well e g multiple ray of motorized webcams monitoring the marina keeping track of
remote email inboxes with contextual controls over privacy boats leaving and returning to their slips Data feeds would likely
for each endpoint be sent to a cloud storage utility where they could be analyzed in
batch or even in near real time
Bootstrapping task brokers from communications Provide a
Some of the component research challenges in building such a
rich swap board website to match tasks and volunteers in
system could include the following
an evolving crisis In the absence of a pre defined lexicon of
tasks and expertise matching must be done from contextual Crowdsourced Robotic Arrays of Webcams Foster commu
information e g text in communications and external data nities of boaters and shoreline homeowners to provide In
sources or communication patterns in a social network his ternet access to control motorized pan tilt zoom web cam
tory with other team members Even shallow analysis of text eras akin to what is being done for ornithology 10 Provide
can help assess general topics the level of technical discourse an online clearinghouse to register such cameras and design
bona fides mentioned by volunteers etc The dynamics of this mechanisms to schedule coverage of regions of interest for
information over time is also important as roles and efforts public safety applications
shift sometimes quickly Crowdsourced UAV Image Harvesting Develop a toolkit to

Related Books

Today's Vision: Tomorrow's Reality


(October 2000, March 2001, June 2001) where communities of interest, dental organizations and ADA con-stituent dental societies could share their views on the project. Drafts of the report-in-progress were dis-tributed prior to each conference. This final report, almost two years in the making, is meant to serve the entire dental community and the

Continue Reading...
Dental Therapists 2000-1 - University of Adelaide

Dental Therapists 2000-1 - University of Adelaide

Most therapists practise in the school dental service (73.4%). In Western Australia, the only State in 2000 to allow therapists to practice in the private sector, 40.8% of therapists reported that their main practice location type was private practice (Table 4). Only 1.6% of therapists reported working in public hospitals.

Continue Reading...


! the Dental Practice Board. Issues covered in my Report 2 This report records the results of my audit of these summarised accounts, and the overall findings from the audits of the underlying health organisations by auditors appointed by the Audit Commission for England and Wales (paragraphs 2.1 to 2.21).The separate NHS summarised accounts for ...

Continue Reading...
2001 Legislation Report -

2001 Legislation Report -

ORAL HEALTH LEGISLATION IN THE STATES 2001 This report is a joint project of the Children’s Dental Health Project and the MCHB National Oral Health Policy Center. Oral Health Legislation in the States 2001 The following report summarizes introduced and enacted oral health legislation in the states during calendar year 2001. This document and its supporting charts contain information that has ...

Continue Reading...
Recruitment and Practice of Dental Therapists

Recruitment and Practice of Dental Therapists

Recruitment and Practice of Dental Therapists 1 1 Review of Future Organisation of Dental Therapy Practice The report of the Dental Therapy Technical Advisory Group 2001, Future Organisation of Dental Therapy Practice, was developed at the early stages of detailed policy development for the

Continue Reading...
NHSScotland: Sterile Services Provision Review Group ...

NHSScotland: Sterile Services Provision Review Group ...

Report (2001), and produced the Glennie Framework in 2001. Whilst dental procedures are categorised as low risk for vCJD transmission, there remains a risk for transmission of bloodborne viruses (e.g. HIV, hepatitis B and C) and other bacterial, viral and fungal infections. It is estimated that in excess of 180 million instruments are re-processed in Scottish general dental practice per annum ...

Continue Reading...
Contents - Honda

Contents - Honda

use, the manual contains a table of contents, a detailed list of topics at the beginning of each section, and an index at the back of the book. As you read this manual, you will find information that is preceded by a symbol. This information is intended to help you avoid damage to your Honda, other property, or the environment.

Continue Reading...
Crf450 Service Manual -

Crf450 Service Manual -

Honda CRF450R service repair manual 2005 -2006 CRF450 | - Instant Download. Instant download of a repair manual for the 2005-2006 Honda CRF450R four-stroke bike. Covers complete tear down and rebuild, pictures and part 2009 2010 HONDA CRF450R CRF 450 R Service Repair Shop Manual - 2009-2010 Honda CRF450R factory service manual. Full of ...

Continue Reading...
Honda Crf450r Service Repair Manual 2002 2003 Download!!!

Honda Crf450r Service Repair Manual 2002 2003 Download!!!

Honda Crf450r Service Repair Manual 2002 2003 Download!!! DOWNLOAD HERE This is a COMPLETE Service Repair Manual for the HONDA CRF450R MOTORCYCLE. Production model years 2002 2003. It is a complete manual similar to a factory shop manuals or CDROM manuals which are used in repair shops. Simple to complicated repairs can be completed ...

Continue Reading...
Honda CRF 450 R (2003) Slip-On Exhaust System

Honda CRF 450 R (2003) Slip-On Exhaust System

The Akrapovic SLIP-ON exhaust system for the HONDA CRF 450 R 2003 conforms to the motocross regulations for exhaust system noise limits and is therefore suitable for MX competition. It is also suitable for Supermoto competition under the condition that it conforms to the requirements for exhaust system noise limits given in

Continue Reading...