Prejudices behind Algorithms: Automated Surveillance Systems as Tools of Segregation and Discrimination

Author: Maciej Gurtowski
Institution: Nicolaus Copernicus University, Toruń, Poland
Author: Jan Waszewski
Institution: War Studies University, Warsaw, Poland
Year of publication: 2018
Source: Show
Pages: 94-109
DOI Address:
PDF: kie/122/kie12205.pdf

The aim of this paper is to describe the emerging phenomenon of new racism brought by the development of automated surveillance tools. We would like to show that this new system of discrimination is a byproduct of some general transformations in the field of social control. For this purpose, we will refer to invisible hand type of explanation, and the concept of perverse effect in particular. We will also relate widely to some cases and concepts from the “new surveillance” and social control literature and we will try to summarize it.


  • Angwin, J. (2016). Facebook Says It Will Stop Allowing Some Advertisers to Exclude Users by Race. ProPublica, November 11. Retrieved from:
  • Angwin, J., Larson, J., Mattu, S., & Kirchner, L. (2016). Machine Bias. ProPublica, May 23. Retrieved from:
  • Angwin, J., & Parris Jr., T. (2016). Facebook Lets Advertisers Exclude Users by Race. Pro-Publica, October 28. Retrieved from:­tisers-exclude-users-by-race.
  • Backman, C. (2012). Mandatory Criminal Records Checks in Sweden: Scandals and Function Creep. Surveillance & Society, 10(3/4), pp. 276–291. DOI: 10.24908/ss.v10i3/4.4206.
  • Bauman, Z., & Lyon, D. (2013). Płynna inwigilacja. Rozmowy. Kraków: Wydawnictwo Lite­rackie.
  • Bauman, Z., Bigo, D., Esteves, P., Guild, E., Jabri, V., Lyon, D., & Walker, R.B.J. (2014). After Snowden: Rethinking the Impact of Surveillance. International Political Sociology, 8(2), pp. 121–144. DOI: 10.1111/ips.12048.
  • Boudon, R. (1982). The Unintended Consequences of Social Action. Berlin, New York: Springer.
  • Brennan, M. (2015). Can Computers Be Racist? Big Data, Inequality, and Discrimination. Equals Change Blog, November 18. Retrieved from:­crimination/.
  • Burdziej, S. (2014). Sociological and Theological Imagination in a Post-secular Society. Polish Sociological Review, 2(186), pp. 179–193.
  • Campbell, N.D. (2004). Technologies of Suspicion: Coercion and Compassion in Post-discipli­nary Surveillance Regimes. Surveillance & Society, 2(1), pp. 78–92.
  • Ceyhan, A. (2008). Technologization of Security: Management of Uncertainty and Risk in the Age of Biometrics. Surveillance & Society, 5(2), pp. 102–123.
  • Chriss, J.J. (2013). Social Control: An Introduction. 2nd Ed. Cambridge: Polity Press.
  • Clarke, R. (1988). Information Technology and Dataveillance. Communications of the ACM, 31(5), pp. 498–512. DOI: 10.1145/42411.42413.
  • Curry, M.R. (2003). The Profiler’s Question and the Treacherous Traveler: Narratives of Belonging in Commercial Aviation. Surveillance & Society, 1(4), pp. 475–499. DOI: 10.24908/ss.v1i4.3332.
  • Duhigg, C. (2013). The Power of Habit: Why We Do What We Do, and How to Change. London: Random House.
  • Durkheim, É. (1893/2014). The Division of Labor in Society. New York: Simon and Schuster.
  • Ferguson, A.G. (2017). The Rise of Big Data Policing: Surveillance, Race, and the Future of Law Enforcement. New York: NYU Press.
  • Foucault, M. (1977). Discipline and Punish: The Birth of the Prison. New York: Vintage Books.
  • Galetta, A. (2013). The Changing Nature of the Presumption of Innocence in Today’s Surveil­lance Societies: Rewrite Human Rights or Regulate the Use of Surveillance Technolo­gies? European Journal of Law and Technology, 4(2).
  • Gandomi, A., & Haider, M. (2015). Beyond the Hype: Big Data Concepts, Methods, and Analytics. International Journal of Information Management, 35(2), pp. 137–144. DOI: 10.1016/j.ijinfomgt.2014.10.007.
  • Gunkel, D.J. (2014). Social Contract 2.0: Terms of Service Agreements and Political Theory. Journal of Media Critiques, 2, pp. 145–167. DOI: 10.17349/jmc114208.
  • Guzik, K. (2009). Discrimination by Design: Data Mining in the United States’ ‘War on Ter­rorism’. Surveillance & Society, 7(1), pp. 1–17.
  • Haggerty, K.D., & Ericson, R.V. (2000). The Surveillant Assemblage. The British Journal of Sociology, 51(4), pp. 605–622. DOI: 10.1080/00071310020015280.
  • Hardt, M. (2014). How Big Data Is Unfair. Medium, September 26. Retrieved from:
  • Introna, L.D., & Wood, D. (2004). Picturing Algorithmic Surveillance: The Politics of Facial Recognition Systems. Surveillance & Society, 2(2/3), pp. 177–198.
  • Kang, J., Bennett, M., Carbado, D., Casey, P., Dasgupta, N., Faigman, D., Godsil, R., Green­wald, A.G., Levinson, J., & Mnookin, J. (2012). Implicit Bias in the Courtroom. UCLA Law Review, 59, pp. 1124–1186.
  • Kenk, V.S., Križaj, J., Štruc, V., & Dobrišek, S. (2013). Smart Surveillance Technologies in Border Control. European Journal of Law and Technology, 4(2).
  • Klauser, F.R., & Albrechtslund, A. (2014). From Self-tracking to Smart Urban Infrastructures: Towards an Interdisciplinary Research Agenda on Big Data. Surveillance & Society, 12(2), pp. 273–286. DOI: 10.24908/ss.v12i2.4605.
  • Kolańczyk, A. (2009). Procesy świadome a automatyzmy w poznaniu społecznym. In: M. Kofta, & M. Kossowska (Eds.), Psychologia poznania społecznego. Nowe idee (pp. 31–57). Warszawa: Wydawnictwo Naukowe PWN.
  • Lahav, G. (2008). Mobility and Border Security: The US Aviation System, the State, and the Rise of Public-Private Partnerships. In: M.B. Salter (Ed.), Politics at the Airport (pp. 77–103). Minneapolis: University of Minnesota Press.
  • Latour, B. (1987). Science in Action: How to Follow Scientists and Engineers through Society. Cambridge, Mass: Harvard University Press.
  • Latour, B. (1999). Pandora’s Hope: Essays on the Reality of Science Studies. Cambridge, Mass: Harvard University Press.
  • Lyon, D. (2003). Surveillance After September 11. Cambridge: Polity.
  • Lyon, D. (2009). Identifying Citizens: ID Cards as Surveillance. Cambridge: Polity Press.
  • Lyon, D. (Ed.). (2011). Theorizing Surveillance: The Panopticon and Beyond. London and New York: Routledge Taylor and Francis Group.
  • Lyon, D. (2015). The Snowden Stakes: Challenges for Understanding Surveillance Today. Sur­veillance & Society, 13(2), pp. 139–152.
  • Maki, K. (2011). Neoliberal Deviants and Surveillance: Welfare Recipients under the Watch­ful Eye of Ontario Works. Surveillance & Society, 9(1/2), pp. 47–63.
  • Marx, G.T. (2002). What’s New About the ‘New Surveillance’? Classifying for Change and Continuity. Surveillance & Society, 1(1), pp. 9–29. DOI: 10.24908/ss.v1i1.3391.
  • Marx, G.T. (2005). Surveillance and Society. In: G. Ritzer (Ed.), Encyclopedia of Social The­ory, Vol. 2 (pp. 816–821). Thousand Oakes, London, New Delhi: Sage Publications.
  • Marx, G.T. (2009). A Tack in the Shoe and Taking Off the Shoe: Neutralization and Counter-neutralization Dynamics. Surveillance & Society, 6(3), pp. 294–306. DOI: 10.24908/ss.v6i3.3286.
  • Mattelart, A. (2010). The Globalization of Surveillance: The Origin of the Securitarian Order. Cambridge: Polity.
  • Mayer-Schönberger, V., & Cukier, K. (2013). Big Data: a Revolution That Will Transform How We Live, Work, and Think. Eamon Dolan/Mariner Books.
  • Meissner, M. (2017). China’s Social Credit System: a Big-Data Enabled Approach to Mar­ket Regulation with Broad Implications for Doing Business in China. May 24. Retrieved from:
  • Merton, R.K. (1948 /1976). Discrimination and the American Creed. In: R.K. Merton, Socio­logical Ambivalence and Other Essays (pp. 189–216), New York: The Free Press.
  • Needham, J. (2013). Disruptive Possibilities: How Big Data Changes Everything. Bejing, Cambridge, Farnham, Koln, Semastopol, Tokyo: O’Reilly Media.
  • Newkirk, A.B. (2010). The Rise of the Fusion-Intelligence Complex: a Critique of Politi­cal Surveillance after 9/11. Surveillance & Society, 8(1), pp. 43–60. DOI: 10.24908/ss.v8i1.3473.
  • Nozick, R. (1974). Anarchy, State, and Utopia. Malden, Mass: Basic Book.
  • Nozick, R. (1991). Invisible-Hand Explanations. The American Economic Review, 84(2), pp. 314–318.
  • O’Neil, C. (2016). Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy. New York: Crown Publishers.
  • Pasquale, F. (2015). The Black Box Society: The Secret Algorithms that Control Money and Information. Cambridge: Harvard University Press.
  • Salter, M.B. (Ed.). (2008). Politics at the Airport. Minneapolis: University of Minneso­ta Press.
  • Schelling, T.C. (1971). Dynamic Models of Segregation. Journal of Mathematical Sociology, 1, pp. 143–186. DOI: 10.1080/0022250X.1971.9989794.
  • Schneier, B. (2014). NSA Robots Are ‘Collecting’ Your Data, Too, and They’re Getting Away with It. The Guardian, February 27. Retrieved from:­mentisfree/2014/feb/27/nsa-robots-algorithm-surveillance-bruce-schneier.
  • Searle, R.H. (2003). Organizational Justice in E-recruiting: Issues and Controversies. Surveil­lance & Society, 1(2), pp. 227–231. DOI: 10.24908/ss.v1i2.3357.
  • Stephens-Davidowitz, S. (2017). Everybody Lies: Big Data, New Data, and What the Internet Can Tell Us About Who We Really Are. New York: Dey Street Books.
  • Sweeney, L. (2013). Discrimination in Online Ad Delivery. Communications of the ACM, 56(5), pp. 44–54.
  • Tetlock, P.E., & Mitchell, G. (2006). Antidiscrimination Law and the Perils of Mindreading. Ohio State Law Journal, 67(5), pp. 1023–1121.
  • Urry, J. (2007). Mobilities. Cambridge, Malden: Polity.
  • Van Otterlo, M. (2014). Automated Experimentation in Walden 3.0: The Next step in Profiling. Surveillance & Society, 12(2), pp. 255–272. DOI: 10.24908/ss.v12i2.4600.
  • Waszewski, J. (2015). Ewolucja systemów nadzoru. In: A. Zybertowicz, M. Gurtowski, K. Tamborska, M. Trawiński, & J. Waszewski, Samobójstwo Oświecenia? Jak neuro­nauka i nowe technologie pustoszą ludzki świat (pp. 233–284). Kraków: Wydawnictwo Kasper.
  • Wilson, D., & Weber, L. (2008). Surveillance, Risk and Preemption on the Australian Border. Surveillance & Society, 5(2), pp.124–141. DOI: 10.24908/ss.v5i2.3431.
  • Zybertowicz, A., Gurtowski, M., Tamborska, K., Trawiński, M., & Waszewski, J. (2015). Samo­bójstwo Oświecenia? Jak neuronauka i nowe technologie pustoszą ludzki świat. Kraków: Wydawnictwo Kasper.

Big Data data doubles dataveillance invisible hand new surveillance perverse effect racism surveillant assemblages

Message to:



© 2017 Adam Marszałek Publishing House. All rights reserved.

Projekt i wykonanie Pollyart