Google Ngram Viewer

From Wikipedia, the free encyclopedia
(Redirected from Google Books Ngram)
Example of an Ngram query

The Google Ngram Viewer or Google Books Ngram Viewer is an online search engine that charts the frequencies of any set of search strings using a yearly count of n-grams found in printed sources published between 1500 and 2019[1][2][3][4] in Google's text corpora in English, Chinese (simplified), French, German, Hebrew, Italian, Russian, or Spanish.[1][2][5] There are also some specialized English corpora, such as American English, British English, and English Fiction.[6]

The program can search for a word or a phrase, including misspellings or gibberish.[5] The n-grams are matched with the text within the selected corpus, and if found in 40 or more books, are then displayed as a graph.[6] The Google Ngram Viewer supports searches for parts of speech and wildcards.[6] It is routinely used in research.[7][8]

History[edit]

The program was released with little fanfare on December 16, 2010 in the hope of opening a new window to quantitative research in the humanities field, and its database contained 500 billion words from 5.2 million books publicly available from the very beginning.[2][3][9] In the development processes, Google had teamed up with two Harvard researchers, Jean-Baptiste Michel and Erez Lieberman Aiden, aiming to provide even children with the ability to browse cultural trends throughout history, related to which Aiden called the method of high-volume data analysis in digitalized texts "culturomics".[9]

Operation and restrictions[edit]

Commas delimit user-entered search terms, indicating each separate word or phrase to find.[6] The Ngram Viewer returns a plotted line chart.

As an adjustment for more books having been published during some years, the data are normalized, as a relative level, by the number of books published in each year.[6]

Due to limitations on the size of the Ngram database, only matches found in at least 40 books are indexed in the database.[6]

Limitations[edit]

The data set has been criticized for its reliance upon inaccurate OCR, an overabundance of scientific literature, and for including large numbers of incorrectly dated and categorized texts.[10][11] Because of these errors, and because it is uncontrolled for bias[12] (such as the increasing amount of scientific literature, which causes other terms to appear to decline in popularity), it is risky to use this corpus to study language or test theories.[13] Since the data set does not include metadata, it may not reflect general linguistic or cultural change[14] and can only hint at such an effect.

Guidelines for doing research with data from Google Ngram have been proposed that address many of the issues discussed above.[15]

OCR issues[edit]

Optical character recognition, or OCR, is not always reliable, and some characters may not be scanned correctly. In particular, systemic errors like the confusion of s and f in pre-19th century texts (due to the use of ſ, the long s, which is similar in appearance to f) can cause systemic bias. Although Google Ngram Viewer claims that the results are reliable from 1800 onwards, poor OCR and insufficient data mean that frequencies given for languages such as Chinese may only be accurate from 1970 onward, with earlier parts of the corpus showing no results at all for common terms, and data for some years containing more than 50% noise.[16][17]

See also[edit]

References[edit]

  1. ^ a b Michael, Jean-Baptiste; Shen, Yuan K.; Aiden, Aviva P.; Veres, Adrian; Gray, Matthew K.; The Google Books Team; Pickett, Joseph P.; Hoiberg, Dale; Clancy, Dan; Norvig, Peter; Orwant, Jon; Pinker, Steven; Nowak, Martin A.; Aiden, Erez L. (2010). "Quantitative Analysis of Culture Using Millions of Digitized Books". Science. 331 (6014): 176–182.
  2. ^ a b c Bosker, Bianca (2010-12-17). "Google Ngram Database Tracks Popularity Of 500 Billion Words". The Huffington Post. Retrieved 2012-05-31.
  3. ^ a b Lance Whitney (2010-12-17). "Google's Ngram Viewer: A time machine for wordplay". Cnet.com. Archived from the original on 2014-01-23. Retrieved 2012-05-31.
  4. ^ @searchliaison (July 13, 2020). "The Google Books Ngram Viewer has now been updated with fresh data through 2019" (Tweet). Retrieved 2020-08-11 – via Twitter.
  5. ^ a b "Google Books Ngram Viewer - University at Buffalo Libraries". Lib.Buffalo.edu. 2011-08-22. Archived from the original on 2013-07-02. Retrieved 2012-05-31.
  6. ^ a b c d e f "Google Books Ngram Viewer - Information". Retrieved 2024-06-01.
  7. ^ Greenfield, Patricia M. (2013). "The Changing Psychology of Culture From 1800 Through 2000". Psychological Science. 24 (9): 1722–1731. doi:10.1177/0956797613479387. ISSN 0956-7976. PMID 23925305. S2CID 6123553.
  8. ^ Younes, Nadja; Reips, Ulf-Dietrich (2018). "The changing psychology of culture in German-speaking countries: A Google Ngram study: THE CHANGING PSYCHOLOGY OF CULTURE". International Journal of Psychology. 53: 53–62. doi:10.1002/ijop.12428. PMID 28474338. S2CID 7440938.
  9. ^ a b "In 500 Billion Words, New Window on Culture". The New York Times. 2010-12-16. Retrieved 2024-06-01.
  10. ^ Google Ngrams: OCR and Metadata Archived 2016-04-27 at the Wayback Machine. ResourceShelf, 19 December 2010
  11. ^ Nunberg, Geoff (16 December 2010). "Humanities research with the Google Books corpus". Archived from the original on 10 March 2016.
  12. ^ Pechenick, Eitan Adam; Danforth, Christopher M.; Dodds, Peter Sheridan; Barrat, Alain (7 October 2015). "Characterizing the Google Books Corpus: Strong Limits to Inferences of Socio-Cultural and Linguistic Evolution". PLOS ONE. 10 (10): e0137041. arXiv:1501.00960. Bibcode:2015PLoSO..1037041P. doi:10.1371/journal.pone.0137041. PMC 4596490. PMID 26445406.
  13. ^ Zhang, Sarah. "The Pitfalls of Using Google Ngram to Study Language". WIRED. Retrieved 2017-05-24.
  14. ^ Koplenig, Alexander (2015-09-02). "The impact of lacking metadata for the measurement of cultural and linguistic change using the Google Ngram data sets—Reconstructing the composition of the German corpus in times of WWII". Digital Scholarship in the Humanities. 32 (1) (published 2017-04-01): 169–188. doi:10.1093/llc/fqv037. ISSN 2055-7671.
  15. ^ Younes, Nadja; Reips, Ulf-Dietrich (2019-03-22). "Guideline for improving the reliability of Google Ngram studies: Evidence from religious terms". PLOS ONE. 14 (3): e0213554. Bibcode:2019PLoSO..1413554Y. doi:10.1371/journal.pone.0213554. ISSN 1932-6203. PMC 6430395. PMID 30901329.
  16. ^ "Google n-grams and pre-modern Chinese". digitalsinology.org.
  17. ^ "When n-grams go bad". digitalsinology.org.

Bibliography[edit]

External links[edit]

Retrieved from "https://en.wikipedia.org/w/index.php?title=Google_Ngram_Viewer&oldid=1226750918"