LabelMe

Key takeaways

LabelMe belongs to computer science; place it in that map before memorising details.
Learn the definition first, then one example that makes the definition concrete.
Connect LabelMe to a quantity you can measure, compute or draw — that is where exam questions come from.
Reproduce the core statement of LabelMe from memory before moving on to harder problems.

Reference excerpt

LabelMe is a project created by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) that provides a dataset of digital images with annotations. The dataset is dynamic, free to use, and open to public contribution. The most applicable use of LabelMe is in computer vision research. As of October 31, 2010, LabelMe has 187,240 images, 62,197 annotated images, and 658,992 labeled objects.

Motivation The motivation behind creating LabelMe comes from the history of publicly available data for computer vision researchers. Most available data was tailored to a specific research group's problems and caused new researchers to have to collect additional data to solve their own problems. LabelMe was created to solve several common shortcomings of available data. The following is a list of qualities that distinguish LabelMe from previous work.

Designed for recognition of a class of objects instead of single instances of an object. For example, a traditional dataset may have contained images of dogs, each of the same size and orientation. In contrast, LabelMe contains images of dogs in multiple angles, sizes, and orientations. Designed for recognizing objects embedded in arbitrary scenes instead of images that are cropped, normalized, and/or resized to display a single object. Complex annotation: Instead of labeling an entire image (which also limits each image to containing a single object), LabelMe allows annotation of multiple objects within an image by specifying a polygon bounding box that contains the object. Contains a large number of object classes and allows the creation of new classes easily. Diverse images: LabelMe contains images from many different scenes. Provides non-copyrighted images and allows public additions to the annotations. This creates a free environment.

Annotation Tool The LabelMe annotation tool provides a means for users to contribute to the project. The tool can be accessed anonymously or by logging into a free account. To access the tool, users must have a compatible web browser with JavaScript support. When the tool is loaded, it chooses a random image from the LabelMe dataset and displays it on the screen. If the image already has object labels associated with it, they will be overlaid on top of the image in polygon format. Each distinct object label is displayed in a different color. If the image is not completely labeled, the user can use the mouse to draw a polygon containing an object in the image. For example, in the adjacent image, if a person was standing in front of the building, the user could click on a point on the border of the person, and continue clicking along the outside edge until returning to the starting point. Once the polygon is closed, a bubble pops up on the screen which allows the user to enter a label for the object. The user can choose whatever label the user thinks best describes the object. If the user disagrees with the previous labeling of the image, the user can click on the outline polygon of an object and either delete the polygon completely or edit the text label to give it a new name. As soon as changes are made to the image by the user, they are saved and openly available for anyone to download from the LabelMe dataset. In this way, the data is always changing due to contributions by the community of users who use the tool. Once the user is finished with an image, the Show me another image link can be clicked and another random image will be selected to display to the user.

Problems with the data The LabelMe dataset has some problems. Some are inherent in the data, such as the objects in the images not being uniformly distributed with respect to size and image location. This is due to the images being primarily taken by humans who tend to focus the camera on interesting objects in a scene. However, cropping and rescaling the images randomly can simulate a uniform distribution. Other problems are caused by the amount of freedom given to the users of the annotation tool. Some problems that arise are:

The user can choose which objects in the scene to outline. Should an occluded person be labeled? Should an occluded part of an object be included when outlining the object? Should the sky be labeled? The user has to describe the shape of the object themselves by outlining a polygon. Should the fingers of a hand on a person be outlined with detail? How much precision must be used when outlining objects? The user chooses what text to enter as the label for the object. Should the label be person, man, or pedestrian? The creators of LabelMe decided to leave these decisions up to the annotator. The reason for this is that they believe people will tend to annotate the images according to what they think is the natural labeling of the images. This also provides some variability in the data, which can help researchers tune their algorithms to account for this variability.

Extending the data

… excerpt ends here. Continue reading the full article.

Worked examples

Example 1 — a first encounter with LabelMe

Start with the simplest possible case. Write down what LabelMe claims or describes in one sentence, then invent the smallest concrete situation in which that sentence is true. In computer science, the smallest case is usually a single object, a single equation or a single measurement. Check that every symbol or term in your sentence has a meaning in that case.

Example 2 — changing one variable

Take the situation from Example 1 and change exactly one quantity: double it, halve it, or set it to zero. Predict what should happen to LabelMe before you calculate. Comparing your prediction with the result is the fastest way to find out whether you understand the idea or only the words.

Example 3 — an exam-style question

Typical questions about LabelMe ask you to (a) state it precisely, (b) apply it to given data, and (c) explain a limitation. Practise writing all three answers in under five minutes; the third part is what separates a full-mark answer from an average one.

Applications of LabelMe

In research: LabelMe appears in computer science research whenever the underlying quantities have to be modelled precisely. Papers usually cite it as a starting assumption and then explore where it breaks down.
In technology and industry: Engineering practice reuses LabelMe in design rules, simulations and safety margins. Knowing the idea lets you read a specification sheet and understand why the numbers look the way they do.
In the classroom: LabelMe is common in secondary-school and first-year university syllabi. It links to neighbouring topics Datasets in computer vision, Object recognition and categorization, so understanding it makes those chapters shorter.
In everyday life: Look for LabelMe outside the textbook — in sport, cooking, traffic, electronics or the sky above you. An example you found yourself is remembered far longer than one you were given.

How to study LabelMe in 20 minutes

Read the reference excerpt below once, without taking notes.
Close the page and write down what LabelMe means in your own words.
Compare your version with the excerpt and mark what you missed.
Work through the three examples above with pen and paper.
Explain LabelMe out loud to somebody else — or to Teacher Smith in the lgStudy chat.

Frequently asked questions

What is LabelMe in simple terms?

LabelMe is a project created by the MIT Computer Science and Artificial Intelligence Laboratory (CSAIL) that provides a dataset of digital images with annotations. The dataset is dynamic, free to use, and open to public contribution.

Why does LabelMe matter?

Because it connects several computer science ideas at once: it gives you a definition you can apply, a quantity you can calculate, and a way to check whether a result is plausible.

How should I study LabelMe?

Read the excerpt, restate it from memory, then work through the examples and applications listed on this page. The five-step study plan above takes about twenty minutes.

What does this page cover?

It gives you a compact reference excerpt plus original lgStudy explanations, examples, applications and study material on LabelMe.