Keepin’ it real: Linguistic models of authenticity judgments for artificially generated rap lyrics

English version

Autoři: Folgert Karsdorp ^aff001; Enrique Manjavacas ^aff002; Mike Kestemont ^aff002
Působiště autorů: Meertens Institute, Royal Netherlands Academy of Arts and Sciences, Amsterdam, The Netherlands ^aff001; Department of Literature, University of Antwerp, Antwerp, Belgium ^aff002
Vyšlo v časopise: PLoS ONE 14(10)
Kategorie: Research Article
prolekare.web.journal.doi_sk: https://doi.org/10.1371/journal.pone.0224152

Souhrn

Through advances in neural language modeling, it has become possible to generate artificial texts in a variety of genres and styles. While the semantic coherence of such texts should not be over-estimated, the grammatical correctness and stylistic qualities of these artificial texts are at times remarkably convincing. In this paper, we report a study into crowd-sourced authenticity judgments for such artificially generated texts. As a case study, we have turned to rap lyrics, an established sub-genre of present-day popular music, known for its explicit content and unique rhythmical delivery of lyrics. The empirical basis of our study is an experiment carried out in the context of a large, mainstream contemporary music festival in the Netherlands. Apart from more generic factors, we model a diverse set of linguistic characteristics of the input that might have functioned as authenticity cues. It is shown that participants are only marginally capable of distinguishing between authentic and generated materials. By scrutinizing the linguistic features that influence the participants’ authenticity judgments, it is shown that linguistic properties such as ‘syntactic complexity’, ‘lexical diversity’ and ‘rhyme density’ add to the user’s perception of texts being authentic. This research contributes to the improvement of the quality and credibility of generated text. Additionally, it enhances our understanding of the perception of authentic and artificial art.

Klíčová slova:

Games – Language – Syntax – Semantics – Syllables – Natural language – Phonology – Vocabulary

Zdroje

1. Gatt A, Krahmer E. Survey of the State of the Art in Natural Language Generation: Core Tasks, Applications and Evaluation. Journal of Artificial Intelligence Research. 2018;61(1):65–170. doi: 10.1613/jair.5477

2. Dale R. The return of the chatbots. Natural Language Engineering. 2016;22(5):811–817. doi: 10.1017/S1351324916000243

3. Ficler J, Goldberg Y. Controlling Linguistic Style Aspects in Neural Language Generation. In: Proceedings of the Workshop on Stylistic Variation. Copenhagen, Denmark: Association for Computational Linguistics; 2017. p. 94–104.

4. Borji A. Pros and Cons of GAN Evaluation Measures. Computer Vision and Image Understanding. 2019;179:41–65. doi: 10.1016/j.cviu.2018.10.009

5. Potash P, Romanov A, Rumshisky A. GhostWriter: Using an LSTM for Automatic Rap Lyric Generation. In: Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing. Lisbon, Portugal: Association for Computational Linguistics; 2015. p. 1919–1924. Available from: http://www.aclweb.org/anthology/D15-1221.

6. Tikhonov A, Yamshchikov IP. Guess who? Multilingual approach for the automated generation of author-stylized poetry. arXiv e-prints. 2018;.

7. Turing AM. Computing Machinery and Intelligence. Mind. 1950;49:433–460. doi: 10.1093/mind/LIX.236.433

8. Radziwill NM, Benton MC. Evaluating Quality of Chatbots and Intelligent Conversational Agents. CoRR. 2017;abs/1704.04579.

9. Rose T. Black noise: Rap music and black culture in contemporary America. Hanover, NH: University Press of New England; 1994.

10. Nelson G. Hip Hop America. USA: Viking; 1998.

11. Chang J. Can’t stop won’t stop: A history of the hip-hop generation; introduction by D.J. Kool Herc. New York: St. Martin’s Press; 2005.

12. Williams JA. Rhymin’ and Stealin’: Musical Borrowing in Hip-hop. University of Michigan Press; 2013.

13. Bradley A, DuBois A, D C, Gates HL. The Anthology of Rap. Yale University Press; 2010.

14. Krims A. Rap music and the poetics of identity. Cambridge: Cambridge University Press; 2000.

15. Adams K. On the Metrical Techniques of Flow in Rap Music. Music Theory Online. 2009;15(5). doi: 10.30535/mto.15.5.1

16. Adams K. The musical analysis of hip-hop. In: Williams J, editor. The Cambridge Companion to Hip-Hop. New York: Cambridge University Press; 2015. p. 118–134.

17. Kautny O. Lyrics and flow in rap music. In: Williams J, editor. The Cambridge Companion to Hip-Hop. New York: Cambridge University Press; 2015. p. 118–134.

18. Ohriner M. Metric ambiguity and flow in Rap music: A corpus-assisted study of Outkast’s ‘mainstream’ (1996). EMR. 2017;11(2). doi: 10.18061/emr.v11i2.4896

19. Condit-Schultz N. MCFlow: A digital corpus of rap transcriptions. Empirical Musicology Review. 2017;11(2):124–147. doi: 10.18061/emr.v11i2.4961

20. Oliveira HG. A Survey on Intelligent Poetry Generation: Languages, Features, Techniques, Reutilisation and Evaluation. In: Proceedings of The 10th International Natural Language Generation conference. Santiago de Compostela, Spain; 2017. p. 11–20.

21. Manjavacas E, Karsdorp F, Burtenshaw B, Kestemont M. Synthetic literature: Writing science fiction in a co-creative process. In: Proceedings of the Workshop on Computational Creativity in Natural Language Generation (CC-NLG 2017); 2017. p. 29–37.

22. Lippi M, Montemurro MA, Esposti MD, Cristadoro G. Natural Language Statistical Features of LSTM-generated Texts. arXiv preprint arXiv:180404087. 2018;.

23. Manjavacas E, Gussem JD, Daelemans W, Kestemont M. Assessing the Stylistic Properties of Neurally Generated Text in Authorship Attribution. In: Proceedings of the Workshop on Stylistic Variation. Copenhagen, Denmark: Association for Computational Linguistics; 2017. p. 116–125.

24. Carlson M. The robotic reporter. Digital Journalism. 2015;3(3):416–431. doi: 10.1080/21670811.2014.976412

25. Lemelshtrich Latar N. The robot journalist in the age of social physics: The end of human journalism? In: Einav G, editor. The New World of Transitioned Media. Heidelberg: Springer International Publishing; 2015. p. 65–80.

26. Van der Kaa H, Krahmer E. Journalist versus news consumer: The perceived credibility of machine written news. In: Proceedings of the Computation+ Journalism Conference, Columbia University, New York. vol. 24; 2014. p. 25.

27. Clerwall C. Enter the robot journalist. Journalism Practice. 2014;8(5):519–531. doi: 10.1080/17512786.2014.883116

28. Jung J, Song H, Kim Y, Im H, Oh S. Intrusion of software robots into journalism: The public’s and journalists’ perceptions of news written by algorithms and human journalists. Computers in human behavior. 2017;71:291–298. doi: 10.1016/j.chb.2017.02.022

29. Graefe A, Haim M, Haarmann B, Brosius HB. Readers’ perception of computer-generated news: Credibility, expertise, and readability. Journalism. 2018;19(5):595–610. doi: 10.1177/1464884916641269

30. Weide RL. The CMU pronouncing dictionary; 1998. http://www.speech.cs.cmu.edu/cgibin/cmudict.

31. Park K, Kim J. g2pE; 2019. https://github.com/Kyubyong/g2p.

32. Merity S, Keskar NS, Socher R. Regularizing and optimizing LSTM language models. arXiv preprint arXiv:170802182. 2017;.

33. Jozefowicz R, Zaremba W, Sutskever I. An Empirical Exploration of Recurrent Network Architectures. In: Proceedings of the 32Nd International Conference on International Conference on Machine Learning—Volume 37. ICML’15. JMLR.org; 2015. p. 2342–2350. Available from: http://dl.acm.org/citation.cfm?id=3045118.3045367.

34. Chung J, Ahn S, Bengio Y. Hierarchical multiscale recurrent neural networks. arXiv preprint arXiv:160901704. 2016;.

35. Bürkner PC. brms: An R package for Bayesian multilevel models using Stan. Journal of Statistical Software. 2017;80(1):1–28.

36. Hoffman MD, Gelman A. The No-U-turn sampler: adaptively setting path lengths in Hamiltonian Monte Carlo. Journal of Machine Learning Research. 2014;15(1):1593–1623.

37. Gelman A, Rubin DB. Inference from iterative simulation using multiple sequences. Statistical science. 1992;7(4):457–472. doi: 10.1214/ss/1177011136

38. McElreath R. Statistical Rethinking: A Bayesian Course with Examples in R and Stan. London, New York: CRC Press, Taylor & Francis Group; 2016.

39. Bürkner PC, Charpentier E. Monotonic Effects: A Principled Approach for Including Ordinal Predictors in Regression Models; 2018. Available from: psyarxiv.com/9qkhj.

40. Eager C. Standardize: Tools for standardizing variables for regression in R; 2017.

41. Van Deemter K, Krahmer E, Theune M. Real versus template-based Natural Language Generation: a false opposition? Computational linguistics. 2005;31(1):15–23. doi: 10.1162/0891201053630291

42. Caswell D, Dörr K. Automated Journalism 2.0: Event-driven narratives: From simple descriptions to real stories. Journalism practice. 2018;12(4):477–496. doi: 10.1080/17512786.2017.1320773

43. Everett RM, Nurse JRC, Erola A. The anatomy of online deception: what makes automated text convincing? In: Association for Computing Machinery; 2016. p. 1115–1120.

44. Roemmele M, Gordon AS. Creative help: a story writing assistant. In: International Conference on Interactive Digital Storytelling. Springer; 2015. p. 81–92.

45. Roemmele M. Writing stories with help from recurrent neural networks. In: Thirtieth AAAI Conference on Artificial Intelligence; 2016. p. 4311–4312.

46. Yi X, Li R, Sun M. Generating chinese classical poems with rnn encoder-decoder. In: Chinese Computational Linguistics and Natural Language Processing Based on Naturally Annotated Big Data. Springer; 2017. p. 211–223.

Keepin’ it real: Linguistic models of authenticity judgments for artificially generated rap lyrics

Souhrn

Klíčová slova:

Zdroje

PLOS One

Aktuální možnosti diagnostiky a léčby litiáz