The MNIST database (Modified National Institute of Standards and Technology database) is a large database of handwritten digits that is commonly used for training various image processing systems. The database is also widely used for training and testing in the field of machine learning. It was created by "re-mixing" the samples from NIST's original datasets. The creators felt that since NIST's training dataset was taken from American Census Bureau employees, while the testing dataset was taken from American high school students, it was not well-suited for machine learning experiments. Furthermore, the black and white images from NIST were normalized to fit into a 28x28 pixel bounding box and anti-aliased, which introduced grayscale levels. The MNIST database contains 60,000 training images and 10,000 testing images. Half of the training set and half of the test set were taken from NIST's training dataset, while the other half of the training set and the other half of the test set were taken from NIST's testing dataset. The original creators of the database keep a list of some of the methods tested on it. In their original paper, they use a support-vector machine to get an error rate of 0.8%. The original MNIST dataset contains at least 4 wrong labels.
History
USPS database In 1988, a dataset of digits from the US Postal Service was constructed. It contained 16×16 grayscale images digitized from handwritten zip codes that appeared on U.S. mail passing through the Buffalo, New York post office. The training set had 7291 images, and test set had 2007, making a total of 9298. Both training and test set contained ambiguous, unclassifiable, and misclassified data. The dataset was used to train and benchmark the 1989 LeNet. The task is rather difficult. On the test set, two humans made errors at an average rate of 2.5%.
Special Database
In the late 1980s, the Census Bureau was interested in automatic digitization of handwritten census forms, so it enlisted the Image Recognition Group (IRG) at NIST to evaluate OCR systems. Several years of work resulted in several "Special Databases" and benchmarks. Of particular importance to MNIST are Special Database 1 (SD-1), released in May 1990, Special Database 3 (SD-3), released in February 1992, and Special Database 7 (SD-7), or NIST Test Data 1 (TD-1), released in April 1992. They were released on ISO-9660 CD-ROMs. They were obtained by asking people to write on "Handwriting Sample Forms" (HSFs), then digitizing the HSFs, then segmenting out the alphanumerical characters. Each writer wrote a single HSF. Each HSF contains multiple entry fields, wherein people were asked to write. There are 34 fields: name and date entries, a city/state field, 28 digit fields, one upper-case field, one lower-case field, and an unconstrained Constitution text paragraph. Each HSF was scanned at resolution 300 dots per inch (11.8 dots per millimeter). SD-1 and SD-3 were constructed from the same set of HSFs by 2100 out of 3400 permanent census field workers as part of the 1990 United States census. SD-1 contained the segmented data entry fields, but not the segmented alphanumericals. SD-3 contained binary 128×128 images digitized from segmented alphanumericals, with 223,125 digits, 44,951 upper-case letters, and 45,313 lower case letters. SD-7 or TD-1 was the test set, and it contained 58,646 128×128 binary images written by 500 high school students in Bethesda, Maryland. They were described as "math and science students in a high school as a short exercise during class". Each image is accompanied by a unique integer ID for the identity of its writer. SD-7 was released without labels on CD-ROMs, and the labels were later released on floppy drives. It did not contain the HSFs. SD-7 was difficult enough that the human error rate on it was 1.5%. SD-3 was much cleaner and easier to recognize than images in SD-7. The European crossed seven (7) is far more abundant in SD-7 than in SD-3. It was suspected that SD-3 was produced by people more motivated than those who produced SD-7. Also, the character segmenter for SD-3 was an older design than that of SD-7, and failed more often. It was suspected that the harder instances were filtered out of the construction of SD-3, since the hard instances failed to even pass the segmenter. It was found that machine learning systems trained and validated on SD-3 suffered significant drops in performance on SD-7, from an error rate of less than 1% to ~10%. In 1992, NIST and the Census Bureau sponsored a competition and a conference to determine the state of the art in this industry. In the competition, teams were given SD-3 as the training set before March 23, SD-7 as the test set before April 13, and would submit one or more systems for classifying SD-7 before April 27. A total of 45 algorithms were submitted from 26 companies from 7 different countries. On May 27 and 28, all parties that submitted results convened in Gaithersburg, Maryland at the First Census OCR Systems Conference. Observers from FBI, IRS, and USPS were in attendance. The winning entry did not use SD-3 for training, but a much larger proprietary training set, thus was not affected by the distribution shift. Among the 25 entries that did use SD-3 for training, the winning entry was a nearest-neighbor classifier using a handcrafted metric that is invariant to Euclidean transforms. SD-19 was published in 1995, as a compilation of SD-1, SD-3, SD-7 and some further data. It contained 814,255 binary images of alphanumericals and binary images of 4169 HSFs, including those 500 HSFs that were used to generate SD-7. It was updated in 2016.
… excerpt ends here. Continue reading the full article.
