Does data science make the world a better or worse place? The Empowerment Summit explored this question, aiming to emphasize the good and also exposing details of the bad.
It was an exciting conference, from Judith Duportail’s personal journey to uncover the truth about Tinder to the thought-provoking “battle” of privacy vs. personalization. As a fan of the book “Weapons of Math Destruction,” I was especially looking forward to the closing keynote by the author, Cathy O’Neil.
Just before the final coffee break, I spotted her bright blue hair and tried to beat the rush asking her a few questions. Next thing I knew, I was defending Swiss startup Privately as Cathy pointed out the long term risks of their approach. I then went into her talk expecting a very critical view of data science and was not disappointed.
The first half was a nice review for anyone who has read her book and an eye-opener for those who haven’t, with examples of data-driven bias in hiring, crime, education, lending and insurance. In the US she claimed, “Most of the big data algorithms being used in these industries are essentially bypassing anti-discrimination laws,” with the motivation of “competitiveness.” She made the point even more human, challenging us to see technology that, “makes lucky people luckier and unlucky people unluckier.”
These feedback loops create a dystopian world that actually exists. Cathy cited a survey with the question, “Do you think the system is rigged against you?” If you say yes, it is.
In data science we are trained to ask hard questions and go to the root of a problem. Cathy’s questions are among the hardest because they require collective answers : not just a few data scientists building algorithms with exponential effects, but all of society defining ethical limits of fairness and success and translating them into code. She described the difficulty as “great news for nerds,” but if we can agree and build this into our algorithms it will be great news for everyone.
In the Q&A Cathy took the gloves off, “recognizing that people in this audience care about privacy” (it was a strongly privacy-oriented crowd, as seen during the interactive voting on privacy vs. personalization), “I don’t give a s*** about privacy,” she said, “it might make sense in Europe but probably not.”
Could she back that up before leaving the stage and getting mauled by the rabidly privacy-loving Swiss crowd?
Yes, and this was as advertised a talk about power. She pointed to the examples as systems of power, where we don’t have power or privacy rights and cannot opt out. She urged us to demand, “When are you allowed to use data against me and what can you do?”
As the other talks confirmed, technically we can do a lot. Data themselves are neither good nor bad. I trust that with people like Cathy sounding the alarm, we will use the power they give for good.
Author: Imai Jen-La Plante
Chief Innovation Officer, Berney Associés