There was a time that having a solid background in math, probability & statistics, linear algebra, with some machine learning plus having great data analysis and presentation skills were enough to make it in data science. The traditional role of a business intelligence specialist has matured into data science. In recent years, however, the job description of data scientists are have been shifted from pure data analysis contains more skills from big data, machine learning, and cloud technologies rather than conventional analytics. Also, having substantial expertise in front-end and back-end software development and DevOps is becoming a necessity of a data science toolbox.
Nowadays, The ability to delivering end-to-end software powered by artificial intelligence (AI) in an Agile way is becoming the new necessity of the market, not just a nice-to-have skill. Indeed, data science is not only data analysis; it's a combination of big data, machine learning, and software engineering. Two main types of data science projects are mostly concerned with business analytics or developing AI-powered software.
For business analytics projects, data scientists usually have to be able to access big data infrastructure and create a data pipeline to deliver actionable insights from the data. Typically, the software being developed in these projects is in the form of analytics dashboards that provides actionable insights to the business decision-makers on-the-fly and assist them with data-driven decision making. For software development, data scientists need to use their big data engineering, data science, machine learning engineering, and software development skills to deploy predictive models in operational software.
The gap between market needs and the skillset of data scientists is now widening as the data science field is gaining more maturity. The company executives are not anymore satisfied with somebody who delivers the actionable insights in a PowerPoint presentation to them every quarter, but they need to access their live dashboards every day for their data-driven decision making. That is probably why the demand for analytics dashboards has been growing steadily within the last few years.
With the advancement of off-the-shelf cloud solutions for AI and big data analytics, data science daily tasks like creating machine learning models are sometimes taken over data-savvy software developers or even non-IT people. Sometimes, this combination works even better for businesses, because in the end, what they need is a deployed and operational software, not a high-accuracy model in a Jupyter notebook.
All of these factors go hand-in-hand together to make it a necessity for data scientists to grow into "full-stack" experts. Facing the pressure to deploy data science and machine learning solutions into enterprise software, and be able to work with big data and DevOps frameworks create new full-stack data scientists. If a data scientist builds a machine learning model with perfect accuracy like 99% that is not a ready-to-deploy piece of software, it is not good enough anymore for the employers!
Natural selection chooses AI algorithms that can show their faces to the world, even if those algorithms a less smart and less customized to the specific use case. Data scientists must learn most of these skills to survive the competition and maintain their position of highest-paid and most respected experts in the industry. In the future, what employers considered as unicorn data scientists will become a norm, and data scientists have to step up and meet the demands of the market to keep their competitive edge.