MyBigDataBlog

In my blog posts, I have described the ways in which BIG Data can be used to benefit organisations, help with decision making and provide solutions to problems. In this blog, I will describe some methods that can be used to apply BIG Data techniques to solve the problem of designing targeted advertising. Targeted advertising is an advertising technique used by companies where they will personalise the ads to the specific user which are tailored to each user based on the data they collect about the user.  Companies such as Google and Facebook invest an immense amount of money to make sure they are able to collect and store as much data from each user as they physically can. They will then use BIG Data analysis techniques to analyse the data they collect.    These companies utilise targeted advertising as they are able to collect massive amounts of data from each of their users which makes their advertising a lot more accurate due to the massive size of their data sets....

Today, I learned about the types of data visualisation. Data visualisation is the methods used to present data in a way that is easy to understand, usually a graphical representation such as a graph or a map of the data. Data visualisation will typically represent the data as statistics such as numbers or percentages.   Visualising data allows us to interpret and understand the data easier than trying to understand the data in its unstructured format, which is important when it comes to making informed decisions based on the insight the data can provide.  Some types of data visualisation that I have learned about are: Charts such as pie charts in which data is represented as a percentage of a circle with labels to show the percentage and index of the data.  Tables which store data in rows and fields of cells, with a field name at the top of each field to distinguish the type of data being stored in that field. For example a table could store personal details with the...

Today, I learned about data mining techniques. Data mining is a method used in BIG Data to find correlations and patterns within massive data sets. There are various data mining techniques which allow organisations to lower costs and increase their revenue.  Data mining allows you to sort through massive amounts of unstructured data to figure out which parts are relevant so you can use those parts to predict outcomes, allowing you to make informed decisions faster.  One data mining technique is to track patterns in the data set, such as data that is repeating at regular intervals or anomalies within the data. This can help organisations to recognise trends within the data which could provide information such as a spike in the sales of a product at a specific time of the year. The other data mining technique is classification which is more complicated than just tracking patterns as it combines a collection of variables into different categories which can be used to provide insi...

 Today, I learned about the types of problems that are suited to BIG Data analysis, including making it easier to measure what consumers actually want. There are problems within every industry that are suited to BIG Data analysis, such as the healthcare industry where BIG Data analysis is used to personalise healthcare plans for patients, or the shopping industry where predictive analysis is used to order inventory based on insights gained from customer and product data, and the security industry where BIG Data has a massive impact on cybersecurity protocols and monitoring due to the real-time data monitoring aspect of BIG Data.  I also learned that the longer data is collected the more accurate the data set becomes in terms of how predictive it is, so data collected for 5 years will be more accurate than data collected for only 3 years.

 Today, I learned about some strategies to limit the negative effects of BIG Data. One of these strategies could be to give the person that generates the data power over the decision for how the data can be used and who is allowed to use the data, however some people believe that this solution is not enough and that there should be stronger restrictions. The data protection act 2018, which is based on the EU's GDPR legislation, is another strategy to limit the negative effects of big data. The data protection act lists strict rules for data controllers about how the data is stored and used, such as processing personal data is only allowed if consent is given by the owner of the data.   

 Today, I learned about the implications BIG Data has on society, specifically the implications big data has on society when used in politics. BIG Data is being used by every political party to influence who people vote for, the result being that BIG Data has a massive impact on who is running governments.  By collecting massive amounts of data sets from social media posts, BIG Data allows political parties to send targeted messages to specific individuals based on keywords from social media posts to persuade their voting decisions.  Another implication of BIG Data has on society is that we are constantly being surveilled as the data from everything you do is collected and stored, yet only less than 1% is actually being analysed, which has resulted in many public cases of privacy breaches. There is also concern that all of the surveillance and data collection that is happening could result in a similar situation to china where they are under constant surveillance with eve...

Today, I learned about the implications BIG Data has on individuals, the biggest of which being the infringement of BIG Data on civil liberties.  One concern about the implications BIG Data has on civil liberties is the investigatory powers act which allows certain organisations to obtain a warrant to bug phones and computers, for which companies would be required to provide assistance as well as encryption keys to decrypt customer communications.  These organisations can collect massive datasets of communication data, such as a dataset of NHS records, and, by using BIG Data analysis, are able to analyse the data to search for keywords. However, these operations require authorisation from a panel of judges which provides a safeguard from misuse of the investigatory powers act.  Another implication of BIG Data for individuals is that since all of the data from social media posts are being collected and stored in massive data sets, then it is possible for everyone, includin...

Today, I learned about the limitations of predictive analysis. In my previous blog posts, I talked about how I learned about the use of predictive analysis within BIG Data, and how it is the use of algorithms to predict the likeliness of future events based on the analysis of data collected.  I have learned that despite the fact that big data analysis relies on massive data sets, these data sets usually don't represent the entire population which means that predictive analytics may be inaccurate when it is applied to individuals.  I also learned that while predictive analysis provides organisation with information that can be used to make important changes or decisions, it does not guarantee that the organisation makes the correct decision and it is ultimately up to the organisation to use the information provided by predictive analysis methods appropriately when they are making decisions.  

 Today, I have learned about BIG Data's technological requirements. BIG Data has three requirements which are how the data is stored, how the data is processed, and how the data is integrated. Due to the massive amounts of data collected and analysed in BIG Data as well as the exponential growth of the amount of data being gathered, it required a massive amount of storage to store all of the data. This data is mostly stored on the cloud as part of massive collections of computers known as server farms, which is made possible by the rise of cloud storage which resulted in the cost of storing data descreasing. Recent improvements within computer hardware and processing power has made it possible for software to be utilised to process massive data sets. Software tools designed to analyse and process these massive data sets are known as BIG Data Technologies and make it possible for value to be extracted from data sets storing an insane amount of data, which would not have been possibl...

Today, I have learned about the future applications of BIG Data and the controversies associated with these applications, such as the controversy over the use of BIG Data analysis to identify, predict and take pre-emptive action against crime.  One future application of BIG Data that I learned about was to train machine learning algorithms for use in the military to provide warfighters with real-time information to give them a tactical advantage. Another application of BIG Data within the military is intelligence gathering as BIG Data analysis techniques can be used to search through data sources such as text messages and social media posts for key words or patterns to provide actionable information.  Another future application of BIG Data that I learned about was the application of BIG Data within the shipping industry. Predictive analysis is used within the shipping industry to provide insight into the condition of the ship as well as information about the ship based on data...

Today, I learned about the contemporary applications of BIG Data within society and the benefits that these applications provide. Recently, many cities have began to embrace BIG Data by implementing various big data developments.    with the intention of improving the quality of life of the people that live in these cities. Some applications of BIG Data in society include water and energy, housing, mobility, and communication. These applications are accomplished using data collected from public organisations and intelligent data collection systems which collect information from smart assets and IoT devices. BIG Data also has applications within the justice system, such as predictive crime mapping which is a method involving using data statistics to identify areas which are more likely for crimes to occur in. Another application of BIG Data within the justice system is predictive analytics which uses statistics to gain information about risks associated with specific individual...

Today, I learned that BIG Data has massive applications in science, specifically healthcare.  I learned that BIG Data has many applications in healthcare, such as electronic health records and server logs from devices and search engines. These applications of BIG Data provide benefits to a number of people within the healthcare system, such as patients, staff, equipment & pharmaceutical manufacturers as well as healthcare organisations. These benefits include enhanced medical outcomes, lower costs and more efficient operations, and the ability for healthcare organisations to make more informed decisions about patient care.

 Today, I have learned about the contemporary applications of big data in business. Prior to the development of big data tools, many businesses were only able to use a small portion of the data they gathered for analytical purposes as they lacked the tools required to analyse entire data sets.  The ability to process and analyse massive data sets enables companies to extract business value from the data, which results in increased opportunities in the data mining and machine learning industries. I also learned about how big data analysis applications benefit businesses by enabling them to understand customers better, allowing the business to use predictive marketing techniques to increase revenue, as well as lowering costs and allowing for equipment maintenance to become proactive. Another benefit of big data analysis for businesses is that it provides information about products and market trends.  Another benefit of big data analysis that I learned about is that supply c...

 Today, I learned about the characteristics of BIG Data analysis. I learned that there are two perspectives for big data analysis, which are decision oriented analysis and action oriented analysis. Decision oriented analysis looks at subsets of the data set to make generalizations which can be used to make predictions and assumptions about the rest of the data set to make business decisions. Action oriented analysis looks at patterns that emerge and offers rapid response to leverage big data by making decisions proactively. I learned that big data analysis can be problematic, due to big data often dealing with raw data, it can be difficult to explore the data manually and programs must be used for any kind of exploration because of the sheer amount of data. If you have collected a huge amount of data, the analysis can be data driven as opposed to hypothesis driven big data analysis. This data driven method of data analysis can be accomplished using machine learning. I learned about...

Today, I learned about the limitations of traditional data.  Traditional data analysis has always been confined to structured data sets such as data stored in a relational database, which means that traditional data analysis techniques are extremely limited when it comes to BIG Data since big data data sets are massive and unstructured. Traditional data analysis is designed to be used for smaller data sets, and as such, is unsuitable for the data sets used in BIG Data analysis.  The massive growth of big data in recent times means that data collected and generated is extremely massive and can be complex which causes issues for normal data analysis systems and techniques are they are insufficient for dealing with this large amount of complex data. 

I learned about Statistics today, specifically descriptive and inferential statistics. Statistics, within big data, is the method used to collect, organise, analyse and present data. Descriptive statistics are the statistics which summarise data that has been analysed. Descriptive statics are used to give an overview of the set of data Inferential statistics are the statistics which allow us to make inferences from the data set about a larger population of data.  I also learned that inferential statistics allow us to make generalisations about larger sets of the population from a smaller data set.

Today, I Learned about the value of BIG Data. I learned that data can be defined using the 5 Vs which are volume, variety, veracity, value and velocity. Volume is the 4th V. The value of data comes from the value of what can be provided using the data and what companies can do using the data I also learned that while companies can use the same tools to collect big data, the tools and methods they use to extract the value from the big data will vary between companies.

 Today, I learned about the reason for the growth of data, which is the exponential increase of devices which are part of the internet of things.  The internet of things is a collection of devices which have built in sensors that can collect and process data. The internet of things is made up of devices such as cars, smart phones, kitchen appliances, watches, and there are almost 20 billion IoT devices connected across the world

 Today, I learned about the exponential growth of big data.  A report from IBM states that 2.5 quintillion bytes of data are created every day and that number is so big that it means that 90% of all of the data that currently exists in the world was created in the last two years.  A report from the International Data Corporation states that from 2013-2020 the global collection of data would grow from 4.4 zettabytes to 44 zettabytes and by 2025 the global collection of data would reach 163 zettabytes and around 220 zettabytes in 2026. Some facts that I thought were interesting are:  Since 1980, the world's capacity to store data has doubled every 40 months.     Over 100,000 apps are created on the apple app store and the google play store every month. Over 5 billion google search queries are made every day.   Over 2.2 billion people actively use Facebook every month.  Almost 300 billion emails are sent daily.