A Silicon Valley company is improving lives in the developing world
Wired, April 2016
In the Butantã branch of Extra, the Brazilian supermarket chain, in the western suburbs of São Paulo, Sandra Morais, 37, is taking photos of bags of rice.
She’s not some retail Instagrammer or an obsessive foodie, but one of 25,000 data collectors that a San Francisco-based startup has sent out, armed with just an Android phone, to harvest real-time economic data as and where it happens. From the price of onions in Indian cities to delayed infrastructure projects in ruralChina, Premise data is, for the first time, giving governments, investors and NGOs an accurate glimpse of what is happening (often literally) on the ground.
For her time – it takes up to ten hours a week to capture all the items on her list – Morais earns around an extra £60 per month, enough to keep her five-year-old daughter in a good school and, as she puts it, “to have some comfort and pay off my debts”.
Premise was founded in 2012 by an American former investment analyst, David Soloff, now 46, who realised that a large amount of developing-world economic data, on which big institutions were basing their risk and funding decisions, was significantly out of date by the time it reached their desks.
Government figures were often weeks old (and sometimes manipulated). Consumer prices reflected broad trends rather than what shoppers were actually paying in the market. And, whereas satellite imagery could give a clue as to the extent of, say, an electrification project in rural Zimbabwe, there was no way of telling which houses were actually connected to the grid. “The problem was,” he says, when WIRED catches him on Skype, in transit between Europe and the US, “that the way data in these areas was being captured was 60 or 70 years old, rooted in the years just after second world war, specifically the Marshall Plan. Typically, a highly paid, specialist consultant would fly into the country from somewhere like London or Frankfurt to do some fact-finding on, maybe, a semi-annual basis. And then, they’d write a report, which would be delivered to the folks who were running the programme.
“This is an inefficient and compromised mechanism for determining whether a programme is doing what it is supposed to. It is too few visits. It is exceptionally lumpy. It is someone from the outside coming in. And it is extraordinarily expensive for the value and quantity of information coming back. It is actually siphoning programme dollars away from the site of the intervention. The consultant just goes back and spends the money on a new suit.”
And these are regions which have been developing fast during the past ten to 15 years. “So you have a really manual survey infrastructure that’s generations old,” he says, “an incredibly fast-moving set of micro economies and a hopeless gap between the two. And that was only going to widen – unless we built the infrastructure to keep tabs on what’s really going on.”
In fact, whatever sector he moves into, Soloff is never going to have much time for consultants in suits. A star student at Columbia and Berkeley, he was in a rap crew (Willard Moan) and a punk band (Halcion), before taking the Wall Street dollar. He likes to brag about having been “arrested in more than one country”. And, despite signs of middle age, he can still come over as the kid who is about to cause havoc somewhere. “Being in a band was the most fun I have ever had…” he says, quickly adding, “until now.” “It is similar to a startup. No one gets it. It’s us against the world. We’re going to show them. It’s the same sources of resentment and creative optimism.”
Premise’s idea is simple: put a smartphone app in the hands of people on the ground and get them to capture data, which is sent to a central server. There, the company’s algorithms can spot patterns and anomalies, it can request more information and, ultimately, it can publish the data in a form that can be used by Premise’s clients, from commercial organisations to NGOs. “The vast majority of our customers are not these mysterious hedge funds you read about,” Soloff says. “The people using this data are folks like the World Bank, who are trying to figure out where to build roads, for example. They are the [Bill & Melinda] Gates Foundation, trying to map where critical financial services are available in developing Africa. Or the [United Nations] World Food Programme, which needs to understand where market imbalances are happening.”
The company’s first client, when it was still in stealth mode in 2013, was Bloomberg, which streamed Premise’s food-staples indices for the US, China, India, Argentina and Brazil to analysts’ screens. From then on, the company grew fast, attracting clients such as Standard Chartered Bank, which uses data supplied by Premise to publish a food-price index for Nigeria and Ghana. Premise now has data-gatherers situated across 34 countries, has raised $66.5 million (£45m) in VC funding and, so far, has paid out $3 million to its contributors on the ground.
This money makes a difference. Almost all Premise collectors can show tangible ways the extra income has changed their lives. Sandra Morais already has a full-time job in a shop selling cosmetics, but discovered Premise through her sister, who was using the extra income to pay her way through college. (Colleges in Brazil are often heavily subsidised, but poorer students can find it hard to afford books or travel to and from campus.) She has now been working for the company for 18 months. As well as the extra income, she says, she is benefiting from learning something about the world of prices and economics. “As I go to the supermarket every day,” she says, “I am starting to understand the social situation around me and how the prices of things affect that.”
The Premise app is designed to run on the often-outdated versions of the Android OS that are common in the developing world. (The company says 30 per cent of the phones the app is running on are “vintage” – older than 2012.) Each week, collectors receive a new list of items they need to record. It is up to them when they go out and complete it, though the system is designed to understand their individual work patterns and preferences, and to hand out jobs accordingly.
On the day we meet, Morais’s app is asking for information on meat (pre-packed, not at the butcher’s counter), chocolate (branded, milk or plain, and with no fillings) and biscuits, as well as the rice she has found. The collectors then track down these items in their own time – many do this on their way home from work or while they are shopping – photographing them and keying in the price they are selling at. All of this information, including a GPS tag that is automatically added by the phone, is uploaded to Premise, either immediately or, in areas where internet coverage is more patchy, at the end of the day when the collector is able to go online. For each item uploaded, the collector is paid a few cents, via services such as PayPal or into their mobile top-up account, or sometimes via a different route. Collectors in Egypt and Venezuela, for example, prefer bitcoin.
Premise now processes about 42,000 images per day, each of which is vetted for accuracy. “We are not a big-data company,” says Joe Reisinger, 33, Premise co-founder and CTO. “In fact, we are kind of the opposite of crowdsourcing. So my job is to work out how to get the type of data that may be smaller, but with the precision that allows us to derive meaningful conclusions from it.”
First, uploads are checked for authenticity: “We have a suite of low-level algorithms to make sure the data makes sense,” Reisinger says. “That it’s not from a different city, or the collector seems to have travelled five kilometres in five minutes. It’s the same kind of check that credit-card transactions go through.”
Then, the system updates the profile of the collector themselves, which helps build up a trust rating for them. New collectors are often given tasks where the answers are already known, but once they start delivering good data they can be used to establish baselines against which other collectors’ data can be compared.
Once the system is confident that the data it is seeing is genuine, it can start looking for what Reisinger calls “the interesting stuff”. “For example, if a price is spiking I might want to change the data collection of that survey to find out how localised that is,” he says. “Is it just confined to that store? Or is it across a wide variety of chains and cities? The system can send out an automatic notification to other collectors in the area to look out for similar data. We call this ‘adaptive survey design’ and, depending on the schedules of the collectors themselves, we can turn this around in a day or two, before compiling a report for our client.”
Right now, the photographs people like Morais upload act as a kind of check, ensuring that the image is what the collector says it is. (“A banana rather than your foot,” as Reisinger puts it.) But it also reveals ways in which a survey may need to be redesigned. “We discovered, for example, that tofu in rural Indonesia is sold in a different way than we had assumed,” says Reisinger, “in buckets rather than sealed packets, so we were able to update the way we asked for information as a result.”
But the company is also developing machine-vision algorithms that can read more information from the images its collectors upload. Its systems can already read price tags on many packages, and a research project to harvest data from its archive of images – which have been uploaded to its system over months or years, and cover a huge array of products – has recently been launched. “What might you see,” Reisinger says, “if, say, you have 100,000 or 200,000 photos of maize over time, as we do from Africa? Could you have a machine vision of these that might predict, say, an impending drought?”
It’s likely, though, this kind of high-level analysis will always be a sideline to Premise’s activities. The company’s real strength is to zoom in to spot specific issues on the ground, work that has already helped the Food and Agriculture Organization of the United Nations (FAO) identify food shortages before they happen. “The FAO needs to figure out what the required amount of foodstuffs are and where they need to go to stave off famine,” he says. “When the FAO first set itself up in Rome [its HQ was transferred there in 1951], the vast amount of famine was a result of climatic failure. Obviously, that remains an important thing, but a far more common source of resource insecurity today is human-induced: people cutting off access, interrupting supply, hoarding, making people abandon farm plots and so on. The unpredictability of those scenarios makes them a more persistent and widespread problem than it has ever been before. We need a much higher-resolution read of what’s happening at all these different locations, across a big basket of commodity foodstuffs, every day. It’s no longer enough to look at satellite photos of fields and assume all is OK.”
Soloff likes to describe Premise as a “triple-bottom-line” business. As its most recent injection of funds shows – $50m, lead by SpaceXand Tesla Motorsbacker Valor Equity Partners in September 2015 – selling accurate market data is potentially a very profitable business. The company charges customers a subscription to its indices and for individual annual contracts, which are in the six-figure dollar range. (The company won’t disclose its financial figures but says its turnover for 2015 will be in the “lower double digits” of millions of dollars and that it expects this to increase fivefold in 2016.) Its commercial clients benefit from getting real-time data on their markets often weeks before official government figures are released. And its NGO clients can use the information to cut down on waste and see where resources need to be reassigned. But it is the third bottom line – the benefit to the collectors themselves and the regions they live in – that give Premise employees most pride.
Luis Garcia, 28, is Premise’s Brazil country manager, and a man energised by the transformative possibilities of the approach. A big man, he overflows with enthusiasm about the company and seems perpetually about to burst out of his dark manager’s suit. On an agonisingly slow car journey through São Paulo’s permanently snarled-up traffic, he’s barely able to sit still in his seat as he explains why he signed up to the Premise project. “I used to work in a bank, managing business accounts,” he says, settling in to a story he has obviously told with pride many times before, “and I found Premise because I was looking for side jobs. I found this amazing ad on Craigslist [posted by Premise in San Francisco] offering the chance to make money from your cellphone by doing market research, and I signed up right away. And as I got closer to Premise, I realised that the work touched my most noble values – equality, transparency and wealth redistribution. Brazil is the face of inequality and the phone is a great equaliser. It gives you access to the internet, to a social network and, through things like Premise, access to money. You could say that we are taking money from VCs in San Francisco and giving it to people in Africa.”
Garcia became one of Premise’s first collectors in Brazil and quickly set about building the company’s presence there. “On the first day I took 60 pictures in an hour,” he says. “I got home and thought, ‘This might be interesting.’ The next day, I worked an hour and did 80, then 120 and, by the fourth day, I was taking 200 in two hours. Once you get the hang of it, you do it without thinking. I say to new users, ‘The next thing you know, you’ll be doing it automatically, without your cellphone.'”
At the end of his first week, Garcia sent an email to his manager – based in Buenos Aires – full of suggestions for how the service could be improved. “I became involved in training and in localising request,” he says. “The original list had things like Oreos on them, which we didn’t have.” (Brazil prefers the coconut-based Oreo clone Negresco.) “Or [breakfast cereal] Cap’n Crunch, which I had never heard of. They’re important – the data has to be global – but it was making data collection here difficult.”
He also set out to recruit others. “I would meet people in person and explain how the system worked. It was a lot about building trust, especially because we pay people at the end of the month. There are a lot of Ponzi schemes in Brazil and I had to reassure people that they would get their money. But it also meant I could show them how good working for Premise could be for them.”
Thanks to his seemingly untiring energy, Brazil is now Premise’s second largest national network, with collectors across almost the entire country. “We have people in [far-flung locations such as] Pará,” he says, “in Manaus, in Casa Nova in Bahia, which is pretty much in the middle of nowhere, as well as teams in the big cities such as Rio de Janeiro and São Paulo.”
Garcia still tries to meet them face-to-face but mostly stays in touch via groups on Facebook and WhatsApp. “When I wake up in the morning,” he says, “I sometimes have 150 to 200 unread messages on my phone.”
For Garcia, these online groups are as important a product of Premise’s work as the data their members collect. “Many of these people were not even on the internet when I first met them,” he says. “I would show them how to use Android, how to download the app and go online. And now they have a whole network of support, sharing ideas, giving each other tips, connecting in a way they never did before. And that truly changes their lives.”
Garcia is one of those rare managers who is able to maintain personal connections with the members of his team, even as it grows. (His work day is, by his own admission, very long, but, he says, “I love it. It is for such a good cause.”) And he inspires a kind of wide-eyed respect in the people who work for him. Waiting for us, as we pull in to a supermarket car park in Socorro, a working-class suburb in south-west São Paulo, is Paulo Santos, 30, who lives in a nearby favela with his partner and their two children. Santos is one of the people Garcia introduced to the internet through Premise, and it has made significant changes to his life.
“Right now, this is my main job,” he says, “as I was recently laid off.” (He was working as a telemarketer.) “I earn about 300 reais [£50] a month from it, which is not enough to live on – I need to find another job – but it’s a good complementary income.” (Santos reckons he needs to earn about 1,800 reais per month but, as WIRED found in the taxi, one of the biggest problems for Brazil’s urban poor is the amount of time spent travelling to and from low-paid jobs. A journey of 15km took almost three hours – and that was in “normal” traffic.)
Santos is proud of the work he does for Garcia – and slightly embarrassed that he has dropped his smartphone and can’t show WIRED the app in action. But he is happy to take us on a virtual tour of his local supermarket – a colossus of a store, with 40 aisles, selling almost everything from icing sugar to car tyres. And, like any professional supermarket shopper, he has a strategy. “I look at my list and I go up and down each aisle, making a really thorough sweep,” he says. “That way, I can cover the most products in the quickest time. I usually work four days a week and within about 5km of my home.”
And, like Sandra Morais, he says he is learning more about economics. “You get a good perception of how prices rise and fall, why onions have gone from three reais [about 50p] per kilo to eight, and then down to three again. And, of course,” he grins, “you always know where the best prices are.”
Premise’s strength is the simplicity of its app and the fact that its model is almost infinitely adaptable and scalable. As well as the prices of goods, it is able to survey costs of services such as doctors and dentists, or the availability of seats on buses. And, the company says, it reserves about 20 per cent of its bandwidth for bigger things “it thinks are important”.
Last year, it asked its collectors in Venezuela to photograph the lengths of queues outside supermarkets, gaps on shelves and signage that indicated rationing, to see the true effect of falling oilprices on the country’s food supply. (The government was painting a rosier picture.) And it predicted the outcome of Brazil’s 2014 presidential election by counting the number of posters for rival candidates on the streets of the country’s cities.
But the big opportunity will come from increasing the number of data collectors and making better use of the technology in their phones. In five years’ time, Soloff says, millions of people could be submitting photos and data to Premise, and they may not have to be doing so consciously. “We could move into the passive sensors that smartphones have, such as accelerometers,” he says. “And we could equip them with other sensors that detect air or water quality.”
Reisinger agrees. “All these sensors give us a broader view of what is happening on the ground. So, if you can detect a phone’s charging state and you have enough people in a city who are all charging their phones at the same time, you can read information from this. If 2,000 people suddenly unplug all at once, maybe that’s a brownout where they are. Or take ambient noise. That’s about quality of life, but also economic activity. Users could opt in and we could take a sound recording every five minutes to measure ambient noise.”
All of this would be layered on to the other sources of information. Premise already harvests price information from about 20,000 websites. (The collectors spot instances when local prices are significantly different.) Soloff sees ways it will integrate with sources such as a satellite imagery. “We partner with some satellite-image providers to amplify their information,” he says. “For example, electrification projects in east Africa. The satellite can see the extent of electrification. But people can then go to the houses and note whether or not a drop line is present for the electrification of the home.”
And, says Reisinger, it works the other way round, too: “If I wanted to survey a large agricultural area,” he says, “it might be better using a satellite. But I would use a human operator in one field to get a baseline and then extrapolate that over a larger region.”
The company is also recruiting at the top level. Last year, Harvard economist Larry Summers and Antonio Gracias of Valor Equity Partners joined its board (“to bring in some adult supervision,” Soloff says), to work with Karim Faris of Google Ventures and former Facebook executive Chamath Palihapitiya, who is especially excited by Premise’s opportunities for growth.
However, some commentators are sceptical of Premise’s approach. “Surveys are of no value unless we can be assured that they are representative of the underlying population,” Barry Bosworth, an economist at the Brookings Institution, told Time in August 2014. “The survey will reflect all the biases of the reporter who decides what prices to report. We may use the internet more in the future to collect data, but it will have to be used with some structure to assure that the individual quotes are representative of an even larger underlying population.”
And the sector Premise is moving into may not be as broken as it claims. “[Soloff’s image of a consultant in a suit] is outdated,” says a UK-based monitoring and evaluation (M&E) specialist, who works for a large number of aid organisations, and who spoke to WIRED on the condition of anonymity. “Most M&E staff are nationals in the countries in which NGO programmes are operating… and there are plenty of NGOs using smartphones and the like to collect data from field locations. Premise sees its future, it seems, in going down a more proprietary route. If it can sell that data, then good luck to it, there must be a demand that is being met. But as to how ‘disruptive’ that will be, I would reserve judgement. I suspect that, at best, it will become another, hopefully useful, part of an already complex ecosystem.”
Soloff acknowledges that Premise will always be one of a range of data collectors, but, he says, “Premise increases the spectrum of information that can be measured on the ground – and the ability to iterate surveys quickly – without losing the rigour of how change is measured.”
Clearly, much of Premise’s data will be bought by speculators betting on commodity-price movements – no one invests tens of millions of dollars in a company that is based primarily in the aid sector. But Palihapitiya, especially, is excited by the opportunities Premise offers its collectors on the ground. “Over time,” he said on joining the company, “people will want to be part of this network for economic reasons, for social ranking, to help organisations they support and to be on the right side of social justice. Regardless of the incentives, the company can build a massive information marketplace and monetise this through companies and institutions paying for ‘ground truth’. We can then return some of that value back to the network, so they see the value in being part of the Premise community.”
Garcia agrees. “This is a cause of giving money to people, wealth distribution, empowering people,” he says.”This is a cause I want to work for, I would die to work for. I am super proud.”
