This Bloomberg article about big data pertains to trading and hedge funds and how data can help them predict trends and make the firm some profit. Firms have started to look at and use data to help predict future returns. Mangers hope to use satellite, social media and personal information to predict things like airline ticket sales and illnesses. Most mangers have not figured out a way to use such data to find specific trends and trading signals. One key example of this is Two Sigma a fund that has used big data and machine learning algorithms to find trends and they have outperformed rivals in doing so. A number of different funds are starting to invest more into data science and data sets in order to stay competitive and find new ways to try and outperform the market. Firms understand how important it is to start using data to help them in the market but there is a problem. Since there is so much data to be collected a lot of it is useless or cannot be used and the stuff than can be used needs to be cleaned and looked further into. One person, James Holloway stated “humans have created a huge amount of low-grade data recently, but have failed to realize the rule quantity does not equal quality.” What James is saying is that people are using a massive amount of data that might not be very valuable but there is a lot of it. This is difficult for firms going through all the data because there is so much of it that is useless to them.
Another problem facing the firms is that some of the vendors, as the article points out, cannot sell their data. The article uses the example of cell phones and using their locations without consumer consent. Or drones talking photos, are they flying under regulations and restrictions? Firms can go through a number of different vendors in search for certain data that they find helpful to them in the trading market. First you need to find out if where the data is coming from and if these vendors have the right to share or sell their data to you.
A third big data set than can help a firm is social media, Twitter and Facebook in particular. One manager said that they collect about 300,000 data sets spanning over the social media but with all the fake accounts posting fake news and prices that can affect a stock price there is a lot of scrubbing that goes into cleaning it. A company Augvest set out to help big firms solve the data puzzle and came up with an idea to break a wall between the fund managers and the data scientists and allow them to work together on trades rather than back office stuff for the data scientists.