7 120 586

7,120,586 Title:

Method and system for segmenting and identifying events in images using spoken annotations

A method for automatically organizing digitized photographic images into events based on spoken annotations comprises the steps of: providing natural-language text based on spoken annotations corresponding to at least some of the photographic images; extracting predetermined information from the natural-language text that characterizes the annotations of the images; segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and identifying each event by assembling the categories of information into event descriptions. The invention further comprises the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner, such as in a photographic album.

What is claimed is:

1. A method for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums,said method comprising the steps of: providing natural-language text based on spoken annotations corresponding to a plurality of frames of photographic images; extracting predetermined information from the natural-language text that characterizes theannotations of the images; segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and identifying each event by assembling the categoriesof information into event descriptions, wherein the step of segmenting the images into events comprises the steps of: assigning a strength value for the certain categories of information which are indicative of a boundary between events; computing theevidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to a current event; and allocating the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

2. The method as claimed in claim 1 further comprising the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner.

3. The method as claimed in claim 2 wherein the event descriptions are selected and arranged in a suitable manner as a photographic album.

4. The method as claimed in claim 1 wherein the step of providing natural-language text comprises the steps of: acquiring spoken annotations; and converting the spoken annotations to natural-language text.

5. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of: segmenting the natural-language text into words and sentences; and applying a plurality ofdictionaries or gazetteers to the words and sentences to classify important words signifying possible events.

6. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of: segmenting the natural-language text into words and sentences; identifying elements ofnumerical expression that may help to define events; and identifying expressions signifying at least one of date, time, money and percentage that may further define events.

7. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of: segmenting the natural-language text into words and sentences; and identifying references topeople, location, events and objects of interest in relation to possible events.

8. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the steps of: segmenting the natural-language text into words and sentences; and identifying noun and verbphrases that may relate to possible events.

9. The method as claimed in claim 1 wherein the step of extracting predetermined information from the natural-language text comprises the step of extracting the natural-language text according to an XML specification.

10. The method as claimed in claim 1 wherein the steps of computing the evidence and allocating the frame are taken with regard to an adjacent frame of the current frame.

11. The method as claimed in claim 1 wherein the steps of computing the evidence and allocating the frame are taken with regard to a non-adjacent frame of the current frame, and wherein the allocation of the intervening frames are made on thebasis of the current frame.

12. The method as claimed in claim 11 wherein the steps of computing the evidence and allocating the frame are taken with regard to a frame that is separated by one frame from the current frame.

13. The method of claim 1 wherein the step of extracting predetermined information from the natural-language text comprises: segmenting the natural-language text into words and sentences; and applying a gazetteer to the words and sentences toclassify important words signifying possible events, said gazetteer comprising a collection of indices including commonly-used proper names, place names including typical tourist spots and celebration places, currency names, function or stop words,irregular verb forms, regular verbs, college and university names and typical events.

14. A computer program product for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said computer program product comprising acomputer readable storage medium having a computer program stored thereon for performing the steps of: providing natural-language text based on spoken annotations corresponding to at least some of the photographic images; extracting predeterminedinformation from the natural-language text that characterizes the annotations of the images; segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary betweenevents; and identifying each event by assembling the categories of information into event descriptions; wherein the step of segmenting the images into events comprises the steps of: assigning a strength value for the certain categories of informationwhich are indicative of a boundary between events; computing the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current framerelative to a preceding frame already allocated to a current event; and allocating the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the currentevent.

15. The computer program product as claimed in claim 14 further comprising the step of summarizing each event by selecting and arranging the event descriptions in a suitable manner.

16. A system for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said system comprising: an input for receiving natural-language textbased on spoken annotations corresponding to a plurality of frames of photographic images; an event extraction stage for extracting predetermined information from the natural-language text that characterizes the annotations of the images; an eventsegmentation stage for segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and an event identification stage for identifying each eventby assembling the categories of information into event descriptions, wherein the event segmentation stage comprises a processor that: assigns a strength value for the certain categories of information which are indicative of a boundary between events; computes the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to acurrent event; and allocates the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

17. The system as claimed in claim 16 further comprising an event summarization stage for summarizing each event by selecting and arranging the event descriptions in a suitable manner.

18. The system as claimed in claim 16 wherein the event extraction stage comprises a natural language processor.

19. The system as claimed in claim 18 wherein the natural language processor comprises a plurality of finite state machines.

20. The system as claimed in claim 16 wherein the processor computes the evidence and allocates the frame with regard to an adjacent frame of the current frame.

21. The system as claimed in claim 20 wherein the processor computes the evidence and allocates the frame with regard to a non-adjacent frame of the current frame, and wherein the allocation of the intervening frames are made on the basis ofthe current frame.

22. A method for automatically organizing digitized photographic images into events, said method comprising the steps of: providing spoken annotations corresponding to a plurality of frames of photographic images; segmenting the images intoevents by examining each annotation for the presence of certain categories of information which are indicative of a boundary between events; and identifying each event by assembling the categories of information into event descriptions, wherein the stepof segmenting the images into events comprises the steps of: assigning a strength value for the certain categories of information which are indicative of a boundary between events; computing the evidence in favor of and against an event break withregard to a current frame by summing the strength values from the certain categories of information present for the current frame relative to a preceding frame already allocated to a current event; and allocating the frame to a new event when thesummarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating the frame to the current event.

23. A system for automatically organizing digitized photographic images into events based on spoken annotations, where the events are useful in organizing photographic albums, said system comprising: an input for receiving spoken annotationscorresponding to a plurality of frames of photographic images; an event segmentation stage for segmenting the images into events by examining each annotation for the presence of certain categories of information which are indicative of a boundarybetween events; and an event identification stage for identifying each event by assembling the categories of information into event descriptions, wherein the event segmentation stage comprises a processor that: assigns a strength value for the certaincategories of information which are indicative of a boundary between events; computes the evidence in favor of and against an event break with regard to a current frame by summing the strength values from the certain categories of information presentfor the current frame relative to a preceding frame already allocated to a current event; and allocates the frame to a new event when the summarized strength values in favor of an event break exceed a predetermined threshold, otherwise allocating theframe to the current event.