Tuesday, November 22, 2016

Big Data Is a Big Mess for Hedge Funds Hunting for Trade Signals

            This Bloomberg article about big data pertains to trading and hedge funds and how data can help them predict trends and make the firm some profit.  Firms have started to look at and use data to help predict future returns. Mangers hope to use satellite, social media and personal information to predict things like airline ticket sales and illnesses. Most mangers have not figured out a way to use such data to find specific trends and trading signals. One key example of this is Two Sigma a fund that has used big data and machine learning algorithms to find trends and they have outperformed rivals in doing so.  A number of different funds are starting to invest more into data science and data sets in order to stay competitive and find new ways to try and outperform the market. Firms understand how important it is to start using data to help them in the market but there is a problem. Since there is so much data to be collected a lot of it is useless or cannot be used and the stuff than can be used needs to be cleaned and looked further into.  One person, James Holloway stated “humans have created a huge amount of low-grade data recently, but have failed to realize the rule quantity does not equal quality.” What James is saying is that people are using a massive amount of data that might not be very valuable but there is a lot of it. This is difficult for firms going through all the data because there is so much of it that is useless to them.
            Another problem facing the firms is that some of the vendors, as the article points out, cannot sell their data.  The article uses the example of cell phones and using their locations without consumer consent. Or drones talking photos, are they flying under regulations and restrictions? Firms can go through a number of different vendors in search for certain data that they find helpful to them in the trading market. First you need to find out if where the data is coming from and if these vendors have the right to share or sell their data to you.  
            A third big data set than can help a firm is social media, Twitter and Facebook in particular.  One manager said that they collect about 300,000 data sets spanning over the social media but with all the fake accounts posting fake news and prices that can affect a stock price there is a lot of scrubbing that goes into cleaning it.  A company Augvest set out to help big firms solve the data puzzle and came up with an idea to break a wall between the fund managers and the data scientists and allow them to work together on trades rather than back office stuff for the data scientists.



  1. After reading through Kieran's response as to why, "Big Data Is a Big Mess for Hedge Funds Hunting for Trade Signals", I found it extremely interesting and wanted to further investigate this topic. You would think that with the increasingly overwhelming capabilities of technology, that technology would be helping hedge funds drastically change their operations and increase potential margins. The main reason that hedge funds are using big data analytics in their operations is to try and spot trends in the market. This has been the case for Nowcast Inc. a Japanese Startup who is "providing automated earnings estimates of consumer goods makers as soon as October by analyzing millions of transactions at retail stores". This is actually a great idea, and could potentially be very valuable to hedge funds that adopt this product. But is all of this data collecting and scrubbing really worth it at the end of the day? Yes, it is worth it because in business a fraction of a second or a little bit more information could make a deal truly worth it. Nowcast will be analyzing big data on consumer prices and retail sales transactions. If hedge funds can see the trends and models of consumer products, it can really change the way of trading. If their application works it can provide hedge funds with the ability to increase net profits and get a leg up on the competition. This new idea of using big data can be an extremely helpful tool if it works properly, but I do understand the room for concern regarding the large amount of wasted information that is being mined. Will the profits resulting from this analytical work be worth it, or will hedge funds pay the analysts more money than they are actually making?


  2. This Bloomberg article was an interesting read and Kieran helped point out the main objectives of the article. I find it surprising when data scientists mention there is too much data, when you think the more data you have the better your chances of finding the next trend. However, this was not the case since the firms were scanning thru useless data. So, I do agree with quality over quantity, but the firms should figure out a way to sift thru the useless data. If they figured out a way to find keywords in the data set, then it may be easier to sift thru the searched data rather than the useless conglomerate of data.

    The market is always changing by the second, so firms will need to utilize big data in order to be competitive in their industry and outperform the markets. Social media is a great place to start for data mining; however there can be fake accounts and profiles that will provide false data and skew your end results. The social media platform, StockTwits, is a financial communication platform for investors. It’s basically Twitter for investors and traders. Firms should incorporate this platform to see what people are talking about and notice if any trends are in the process of forming. There may be too much useless data out there, but I strongly believe firms should be putting in the time to sort and clean up the data that is available. The higher quality data you have, the better are your chances of finding the next trend.


Note: Only a member of this blog may post a comment.