Understanding the Basics: Data Engineering, Data Analytics, and Data Science.

Artificial Intelligence has had more buzz than ever since Open AI released chat GTP to the public, Alexander Wang, Ceo of Scale AI called it a “Four year overnight success”. This gives an overview of the difficulties involved in building it.

ChatGPT and other AI tools reveal a fundamental truth: they heavily rely on data. This reliance explains the sudden popularity of fields such as data engineering, data science, and data analysis. It can be said that the influence and popularity of ChatGPT has extended to these domains.

However, this newfound fame has raised questions about the roles of individuals in related fields. While they all contribute to world-class technologies through their work with data, understanding their individual roles is crucial—especially for people with the desire to get into the fields. This article aims to discuss these roles in concise terms.

Firstly, let's define data—it's the collection of facts and statistics used for reference or analysis. Not clear enough? Well, let's simplify: Data is the collection of activities, engagements, or actions taken and studied to aid future actions.

In the business realm, data is a powerful tool. The top players in business recognize that delving deep into data gives them a competitive edge. This awareness further drives the demand for roles we'll explore in this article. Now, let's dive into understanding these crucial roles.

Data engineering is a field that focuses on designing and building data infrastructure. It involves building and optimizing systems for data analysis.

If this does not make sense to you, then think of it like this: before anything can be done with data, the data actually needs to exist, and it needs to exist in a way that can be used in the most expandable way possible. Data engineers are responsible for building the foundational infrastructure of data.

Consider them the architects of data—after constructing the data pipelines, they ensure it is collected, stored, and positioned for future analysis.

They use tools like SQL (for databases like MySQL, PostgreSQL) and NoSQL databases (like MongoDB, Cassandra) to get and work with data. They also use tools like Hadoop, Oracle Data Integrator, and IBM DataStage to organize, clean, and change data. These tools help them manage and change data for different purposes. The Data engineer ensures that data is available in a structured and usable format for analysis.

Data engineering is about creating data systems, which can include the development of software systems and applications. Therefore, if you're a software engineer with a good background and strong programming skills, you're in a great position to move into data engineering roles.

Now, let's shift our focus. We've talked about data engineering, the foundation builders.

The infrastructure is established, and data is at our fingertips. This is where data analysts step into the scene.

From the infrastructure built and the data obtained, data analysts can then take the reins by building dashboards, understandable reports, and visuals to provide insights for informed decision-making.

Notice the distinction now? While the former builds the foundation, the latter builds on the foundation.

Once the data infrastructure is in place, the data analysts work with the available data to generate reports, visualizations, and meaningful conclusions. They clean and organize data, analyze it to find important patterns, and explore the data to understand its main characteristics. Afterward, they create easy-to-understand reports and dashboards to help with future work and decision-making.

Data engineers set up the data infrastructure, and data analysts use it to create reports and visualizations. Engineers build the foundation, and analysts build on it by cleaning and organizing data, finding patterns, and creating easy-to-understand reports and dashboards for decision-making.

Data analysts employ various tools like Excel for data cleaning, filtering, and charting. They also use advanced visualization tools such as Power BI and Tableau to present data in a clear manner accessible to everyone. This is important because data analysts often engage in soft skills, including presentation in fields like sales, marketing, and finance, requiring effective communication and ensuring understanding to both technical and non-technical audiences.

If all this doesn't make sense, then let me tell you a story...

Steve Nutjobs sells fashion items online. He's been doing it for a year now and has good data on what he sold. Enter his cousin, Bill, who knows a thing or two about being a data analyst. While helping out, Bill dives into the year's data using tools like Excel.

As he works, Bill discovers that certain products sell better during specific times or with promotions on social media. Armed with this info, Bill plans targeted promotions and tweaks Steve's inventory to match what customers like. He also analyzes customer feedback and reviews, learning what features and styles shoppers prefer.

Using these insights, Bill brings in more of the popular products, making customers happier and boosting sales. Steve's online store thrives, all thanks to Bill's data analytics skills.

Bill has just played his role as a Data Analyst in Steve's eCommerce venture. I hope this story has given you a better understanding, especially in the context of differentiating between the role of a Data Engineer. Although, he would have a more rigorous experience executing his role in bigger organizations, but you get this gist.

Beyond these fundamental tasks, Data Analysts delve into more sophisticated and analytical roles. They use languages like Python and R, write database queries—essentially, guiding instructions for data—and apply math concepts like statistics and probability. Understanding algebra and calculus helps them stand out.

So, data analysts do all these? Shouldn’t this be everything? What else could possibly exist? I'm glad you have these in mind. This is where the data scientist comes in.

While data analysts focus on organizing and presenting data, data scientists take it a step further. They delve deep into data mines, utilizing advanced algorithms to shape the future of data. Data scientists develop models that predict outcomes and optimize decision-making processes.

Take note of a keyword here: 'Model.' This is new, and we haven't seen this before.

So, what then is a model?

A 'Model' refers to a mathematical representation or framework created using algorithms and trained on data. The purpose of a model is to make predictions or decisions based on new, unseen data.

Now, let's utilize the data we already have, we can term them as historical data (training data). We can teach the model patterns and relationships within the data. Why is this useful? Simply because the model learns from this data to make predictions or classifications when exposed to new, unseen data. This officially brings into light the concept of automation and less human input, although in a preliminary way.

Once trained, the model can be applied to new data to make predictions, classifications, or decisions. And that is what differentiates the Data Analyst from the Data Scientist. Data Science is the extension of Data Analytics. It revolves around more math (a computational degree would be more plausible for job titles around Data Science, although not mandatory if you have enough fieldwork), more analytical thinking, and more programming.

We can now say that Data science extends beyond descriptive analytics, incorporating more advanced mathematical concepts, analytical thinking, and programming skills.

The integration of data science with data analytics amplifies the impact of insights gained from data.

A data scientist utilizes a range of tools and skills, including Python, Power BI, machine learning algorithms, SQL, and mathematical concepts such as statistics and probability.

Once a model is trained, it can be then deployed in various real-world scenarios to make predictions, classifications, or decisions on their own. Yes, now we are talking about Artificial Intelligence. I'm sure you have heard about this already. :)

Data scientists may come from fields such as computer science, statistics, physics, and engineering as some mathematical concepts are commonly used in data science (e.g., linear algebra, calculus) and programming languages (e.g., Python, R).

It's essential to note that, theoretically, the roles of a data scientist, engineer, and analyst differ. In practice, though, these roles often overlap, with blurry lines between them in many organizations, Team members may find themselves wearing multiple hats.

This situation is particularly common in small companies or startups aiming to minimize spending to meet their goals. At this stage, these organizations carefully manage their resources, leading to a blending of roles. So, Despite officially designating the position as a 'Data Scientist,' in practice, it extends to encompass the day-to-day or occasional implementation of both data engineers and analysts.

In Big companies, however, the division of labor tends to be more specialized, and roles may be more clearly defined, that is, the data scientist will do the job of the Data Scientist, the Data Analyst will do the job of the analyst, etc. This is most likely because, unlike a startup, their number one priority may not be to manage their resources and use fewer people.

Now you know the difference between a Data Engineer, Data Analyst, and Data Scientist, And if there's any part you do not fully comprehend, you can always go back to read as it has been written in an easy and digestible way.

And for the people wanting to get into any these space, its very important to consider the things that interest you.

If you are good at advanced mathematical concepts, analytical thinking, and programming, then pursuing a career in data science is the right move for you.

If you prefer storytelling, analyzing businesses, or Maths (Average level but the higher the better), Data Analytics is the right choice for you.

And lastly, if you enjoy building infrastructures or have a background in software engineering already, then pushing it further in a way that compliments your already existing skills is you opting for Data Engineering.

Ultimately, your success depends on picking what works for you and becoming a consistent doer.

If you like this content, kindly give it a like and share, it would go a long way :).

See you soon.

Good luck :)