Consumers have preferred convenience over privacy. While many say privacy matters, few have placed any real value on protecting their data. We still buy connected devices, and use services that aren't always honest about the use of private data. Companies like Google, Facebook, and Amazon remain at the center of our digital lives.
But this might be changing. A 2019 survey conducted by Cisco looked at what people do - not just what they say - when it comes to privacy. The survey shows a new group of people who say they care about privacy and have done something about it, by switching companies or providers over their policies.
According to a Harvard Business Review study, 72 percent of Americans are reluctant to share information with businesses because they “just want to maintain [their] privacy.” And Deloitte, as a part of a study about consumer trust and data protection, explains that caring about data privacy and security is not just a risk management issue, but a “potential source of competitive advantage that may be a central component of brand-building and corporate reputation.”
Policy-makers stand right behind these privacy-minded consumers. The US and EU mandate strict rules regarding the storage and exchange of personally identifiable data. Policies like GDPR or CCPA have taken center stage in recent times.
Artificial Intelligence technologies are the driving force behind the data needs. Almost all use of AI requires the collection and use of large amounts of data, in many cases personal data, to learn and make intelligent decisions.
Many of the areas where AI could be used beneficially are those where privacy is of the highest concern, and where the limited availability of data has been a hindrance; for example medical history, financial records, or private habits. In this post we’ll show how we can tap into the pool of sensitive data while simultaneously addressing the need for data protection.
Threats and solutions
Privacy risks from AI stem not just from the mass collection of personal data, but from the models that power most of today’s algorithms. Data isn’t vulnerable just from database breaches, but from “leaks” in the models that reveal the data on which they were trained.
Optimal privacy preserving systems must resist attacks against the dataset, for example they can’t reveal whether an individual is present in a given dataset and they cannot allow any extraction of characteristics of an individual from within a dataset. They must also withstand attacks on the algorithm, for instance those that try to derive information about the dataset from it.
Bellow we’ll list different threats to both data and algorithms, and technical solutions that minimize them.
Protecting the data
Threats
Data theft
Data theft is one of the most visible threats. Private data may be transferred to another party for computation and may need to be available in plaintext during parts of processing unless privacy-preserving techniques are used. This creates insider, outsider, and configuration risks that the architecture must address.
The system should define who can access input and output data, minimise that access, and verify the controls against the relevant threat model. Some use cases may require the data to remain hidden from the model provider or operator.
Privacy goes hand in hand with security. Strong controls can reduce the likelihood and impact of data exposure, but they do not by themselves guarantee regulatory compliance or prevent every incident.
Identity/membership inference
Membership inference attacks can tell whether a sample was in the training set based on the output of the model. If the sample is linked to a person, such as medical or financial data, then inferring membership is a privacy threat. For example, the attacker could learn whether someone's individual record was used to train a ML model associated with a specific disease.
Some membership-inference attacks can work with black-box model access and confidence scores. Their effectiveness depends on factors such as the model, outputs, training process, and attacker's auxiliary information.
Feature reconstruction
Reconstruction attacks try to recreate the private data from the feature vectors. As such, they need white-box access to the model.
These attacks could happen when the feature vectors used in the training phase are not deleted after building the model. In fact, some ML algorithms such as SVM or kNN store feature vectors in the model itself.
Examples of successful cases of reconstruction attacks include: fingerprint reconstruction where a fingerprint image was recreated from a minutiae template (features); and mobile device touch gesture reconstruction where touch events were recreated from gesture features such as velocity and direction. In both cases, the privacy violation was a security threat to authentication systems.
Solutions
Data anonymization
Anonymisation aims to make a person no longer identifiable, while pseudonymisation replaces direct identifiers but can remain reversible with separately held information. Whether data is truly anonymous depends on context and re-identification risk.
It is good practice to remove personal identifiers from data if you're planning to make any it public. But, even then, sophisticated attackers can identify a lot about a user. For example, Netflix released anonymized versions of their datasets, to aid contestants for its 1M$ prize to build better recommender systems for movies. Despite the anonymization, researchers were able to use this dataset along with IMDB background knowledge to identify the Netflix records of known users, and were further able to deduce the users’ apparent political preferences.
Differential privacy
Models can sometimes encode more information than necessary for the prediction task, and can inadvertently memorize individual samples. For example, a language model built to predict the next word (like those you see on smartphones) can be exploited to release information about individual samples that were used for training.
Differential privacy uses a formal privacy budget and controlled randomisation to limit what an output reveals about an individual's contribution. It introduces a privacy–utility trade-off, so usefulness must be evaluated for the specific task and parameter choices.
Protecting the algorithm
Threats
Model theft
The model is core to the a company's interests: there is little point in improving AI capabilities, if your competitors can easily copy the models.
A model can be stolen outright or can be reverse-engineered based on its outputs, see model inversion below.
Model inversion
With model inversion, the attacker's goal is to create feature vectors that are similar to those used to create the model, by looking only at the model responses. This is especially important nowadays, when ML-as-a-service is a popular approach.
Such attacks use the confidence information (e.g. class probabilities) that are sent with the response for testing samples. These attacks produce an average that represent a certain class; so, they would be most threatening to privacy when a certain class represents a single individual, as in face recognition.
Adversarial manipulation
In computer vision, small perturbations to images can cause neural networks to make mistakes such as confusing a cat with a computer. They are like optical illusions for machines.
Adversarial attacks can be designed to make ML models produce specific outputs chosen ahead of time by the attacker. More recently, researchers have shown that they can reprogram the target model to perform a task chosen by the attacker, without the attacker needing to specify or compute the desired output for each test-time input. This attack finds a single adversarial perturbation, that can be added to all test-time inputs to a machine learning model to cause the model to perform a task chosen by the adversary, even if the model was not trained to do this task. These perturbations can thus be considered a program for the new task.
Solutions
Homomorphic encryption
When using homomorphic encryption, data can be encrypted by its owner and sent to the model owner to run computation as if it was unencrypted. There are currently restrictions on the type of calculations that can be performed using homomorphic encryption, and the computation performance is still very far from traditional techniques.
Secure multi-party computation
With secure multi-party computation, the processing is done on encrypted data shares, split so that no single party can retrieve the entire data on their own.
Depending on the protocol and threat model, parties can compute a result without any one participant receiving every party's underlying data. The guarantees depend on protocol design, implementation, and assumptions about collusion.
Federated learning
Federated learning means distributing copies of a machine learning model to the devices where the data is kept, performing training iterations locally, and returning the results of the computation to a central repository to update the main algorithm.
Whereas the data remains with its owner, it doesn’t entirely guarantee security and privacy since attackers can steal personally identifiable data directly from the devices or interfere with the communication process.
Conclusion
In this post we looked at ways in which bad actors can target AI systems and sensitive information. The techniques described can mitigate particular threats, but none is a universal defence and each depends on correct implementation.
A privacy goal may include limiting whether an attacker can infer that a person's data belongs to a dataset or extract sensitive attributes. Anonymisation, pseudonymisation, and differential privacy address different parts of that problem and provide different guarantees; they are not interchangeable.
Attacks on algorithms may try to reverse engineer a model or use it in unintended ways. Limiting access and unnecessary output detail can be one mitigation, but black-box access is not sufficient on its own. The effect of rounded confidence values or class-only outputs must be tested against the relevant attack.
Not a zero-sum game
To conclude, AI can help solve many business challenges but is very dependent on large amounts of data, often sensitive. The solution to privacy concerns is not to limit the advancement and applications of the technology, but rather to find new ways to use the data in privacy-preserving ways.
Privacy isn’t just a few sentences buried in terms and conditions, it is something to consider from day one when building products or services. Designing software with privacy in mind is not a zero-sum game - it is better for everyone.
Read more: How a Private Data Moat Can Strengthen Your AI Advantage or explore our AI Strategy & Leadership service for enterprise AI governance.