Validate Data Mining In Tableau With A Chi-Square Test

validate validation

I have just enrolled in a Data Science course on Udemy  and I learned good stuff.

In this article we will start using statistics. Don’t worry we’ll do something simple, we’ll use the Chi-square test in a basic way. There is a special section to learn how to do statistics at an advanced level.

I’ll explain why we’re going to learn how to use the Chi-square test. The results we have with theses 2 bar charts are good. We see on theses 2 bar charts that age has a significant impact on the rate of client leaving the bank. We also see in which age groups the clients leaves the bank the most and which age groups the clients leave the bank the least. With that we have good insights.

In the A/B test « Gender », we can see that there is a correlation between the male and female sex and the choice to leave the bank. But as I said before, this A/B test is basic. The results of a basic A/B test visually shows us what is probably happenning in reality but we aren’t 100% sure of these results. To validate these results, we need do to use statistical tests like Chi-square test.

Doing a report based on basic A/B test is very risky and you can have completely false insights. I don’t advise you to do it (unless you want to leave your job). It’s for this reason that using Chi-square will help us to have strong insights.

Chi-square will allow us to know if our results are statistically significant. Our results are based on a sample of 10 000 clients and Chi-square test will tell us if these results are due to chance effects or if these results can represent all the client of the bank.

For example in our A/B test « Gender », we observed that in our sample of 10 000 clients, women are more likely to leave the bank compared to men.

tableau data mining science chi square test a/b test

Now, we aren’t sure if the results of this sample represent the behavior of all the bank’s clients.

To use basic Chi-square test, we use an online tool. Click here  .

tableau data mining science chi square test a/b test

On internet, there are plenty of websites to do a Chi-square test but we’ll use this one so that you can understand how it works. To do a Chi-square test, we need to use absolute values and in our A/B test we have percentage.

Let’s go back to Tableau. We’ll create a new tab with a version of A/B test with absolute values. In this way, we keep the A/B test with the percentages. Do a right-click on the « Gender » tab and select « Duplicate ».

tableau data mining science chi square test a/b test

Name the new tab « Gender Actual » to specify that it’s absolute values.

tableau data mining science chi square test a/b test

To have the absolute values, move « Number of Records » in « Measures » to the « Marks » area and put it over top of « SUM(Number of Records ».

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Move « Number of Records » in « Measures » to « Rows » over « SUM(Number of Records ».

tableau data mining science chi square test a/b test

Cool, we have our absolute values.

tableau data mining science chi square test a/b test

We also need total absolute values, which means the total number of men and women. There is a very fast way to get that. Right-click on the vertical axis and select « Add Reference Line ».

tableau data mining science chi square test a/b test

Then in « Value », click on the drop-down on the right and select « Sum » to have the total sum of the observations.

tableau data mining science chi square test a/b test

And in « Scope », you select « Per Cell » option to specify that you want the total sums for each category, male and female.

tableau data mining science chi square test a/b test

Now, we have the total sum at the top of the bars. We will modify labels to have the absolute values. In « Label », we will change « Computation » to « Value » and click on the « OK » button.

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Perfect, we have the total amount of observation at the top of each bar : 4543 women and 5457 men. We have what we need to use our online tool.

tableau data mining science chi square test a/b test

OK, I’ll explain how this tool works. « Sample1 » and « Sample2 » correspond to the independent variable « Gender ». You choose in which order you enter the data, « Sample1 » for men or the opposite. In our case, we use « Sample1 » for women and « Sample2 » for men.

« #success » corresponds to the result Y=1, which means in our case « yes, the client left the bank ».

« #trials » is the total number of observations, which means the total number of women in « Sample1 » and the total number of men « Sample2 ».

That’s how you enter the data :

  • For « Sample1 » in #success, you enter 1139 because there are 1139 women who left the bank. For « Sample1 » in #trials, you enter 4543 because there are 4543 women in total.

 

  • For « Sample2 » in #success, you enter 898 because there are 898 men who left the bank. For « Sample2 » in #trials, you enter 5457 because there are 5457 men in total.

tableau data mining science chi square test a/b test

Here is the verdict : « Sample1 is more successful ». « Sample1 » corresponds to women and #success is :« yes, the client left the bank ». This verdict means that of all the bank’s client, women are more likely to leave the bank than men. And look, there is something important, it’s « p<0.001 ». This means that the « p » is strictly less than 0.001.

tableau data mining science chi square test a/b test

« p » is the value that indicates whether an independent variable has a statistically significant effect on a dependent variable. In our case, the independent variable is « Gender » and the dependent variable is « Exited », which is : « yes, the client left the bank ». So « p » is strictly less than 0.001, which means that the independent variable « Gender » has a statistically significant effect on the dependent variable « Exited ». This shows us that out of the total number of bank’s clients, women are more likely to leave the bank than men.

This is how we use Chi-square test with this online tool. This is the same principle on all online tools that you can find on Google or DuckDuckGo . You can repeat these instructions that I gave you with other tools, you will get the same results.

It’s cool with the Chi-square we validated the A/B test and to specify that this A/B test is validated, we’ll color the tab in green.

Right-click on the tab, select « Color » and select « Green ».

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Perfect, now we’ll validate another A/B test. Selects « HasCreditCard » tab.

tableau data mining science chi square test a/b test

We’re going to create an A/B test « HasCreditCard » only with absolute values. To save time, right-click on « Gender Actual » tab and select « Duplicate ».

tableau data mining science chi square test a/b test

We’ll remove the green color on the tab « Gender Actual (2) ». Right-click on the tab and select « Color » and « None ».

tableau data mining science chi square test a/b test

You rename the tab « HasCreditCard Actual ».

tableau data mining science chi square test a/b test

Move the variable « HasCrCard » over « Gender » in « Columns ».

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Excellent, everything is ready to do a Chi-square test. We’ll remove « Exited » labels to better see the absolutes values. Make a click and drag out.

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Perfect, let’s go back to our online tool. In this case, « Sample1 » is « no », which means client who don’t have credit card and « Sample2 » for « yes », which means clients who have a credit card.

That’s how you enter the data :

  • For « Sample1 » in #success, you enter 613 because there are 613 clients who left the bank. For « Sample1 » in #trials, you enter 2945 because there are 2945 clients who don’t have a credit card.
  • For « Sample2 » in #success, you enter 1424 because there are 1424 clients who left the bank. For « Sample2 » in #trials, you enter 7055 because there are 7055 clients who have a credit card.

tableau data mining science chi square test a/b test

Let’s look at the verdict, it’s « No significant difference ». « p » value is very high, it’s above 5%. This confirms that the independent variable « HasCrCard » has no statistically significant effect on the dependent variable « Exited ». That was the conclusion we had made when we had done the A/B test with percentages.

We had seen that there was 21% of « Exited » (clients who left the bank) in the category « no » and 20% in the category « yes ». With these results we concluded that most likely the variable « HasCrCard » had no impact on the rate of clients who left the bank. Chi-square test confirms our conclusion and we can put the tab « HasCrCard » in green to say that it’s OK.

Right-click on the tab « HasCreditCard » => « Color » => « Green ».

tableau data mining science chi square test a/b test

tableau data mining science chi square test a/b test

Excellent, now, you can do a statistical A/B test with 2 categories. Soon, we will do statistical A/B tests with more than 2 categories.

Share this article if you think it can help someone you know. Thank you.

-Steph

Combine 2 charts

tableau chart compare paralell data mining science

I have just enrolled in a Data Science course on Udemy  and I learned good stuff.

We’ll move to the next level. We’ll work with 2 bar charts in parallel to have a more efficient data mining. In a previous article, we created 2 different bar charts. The 1st was an A/B test (actually, it’s a classification test) that told us in which age range the clients were most likely to leave the bank. The 2nd was a bar chart showing the age distribution of clients in our sample of 10 000 clients.

Let’s go. We’re going to have an A/B test with age range and we’ll add a bar chart of the client distribution below. To add a bar chart, we must start by choosing what we want to keep and what we want to add. In our case, we want to keep the columns because they’re the same in the 2 bar charts.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

And we just want to add a new line so we will add a new variable in « Rows ». As we want to add a bar chart of distribution, we will use the variable which corresponds to the number of observation « Number of Records ».

In « Measures » moves the variable « Number of Records » in « Rows » to the right of « SUM(Number of Records).

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

We have a 2nd bar chart below the 1st bar chart. As you can see, these 2 bar charts are in one column. « Columns » is « Age(bins) ». These 2 bar charts are in 2 different lines which are the lines that correspond to the 2 « SUM(Number of Records) » in « Rows ».

The space on the left has also changed. There is « All » which represents the 2 bar charts at the same time. It means, when your select « All », you make change in the 2 bar charts.

tableau chart compare paralell data mining science

Below this tab « All » we have 2 tabs. The 1st tab represents the 1st bar chart so the 1st « SUM(Number of Records) » in « Rows » and the 2nd tab represents the 2nd bar chart so the 2nd « SUM(Number of Records) » in « Rows ».

tableau chart compare paralell data mining science

Which means that if you want to make changes on the 2 bar charts at the same time, you make the changes in the tab « All ». If you want to make changes only in the first bar chart, you select the first tab below « All ». If you want to make changes only in the 2nd bar chart, you select the second tab below « All ».

So if you change the color in tab « All », our 2 bar charts will be colored by the same color.

Select the « All » tab and click on « Colors ».

tableau chart compare paralell data mining science

Click on « Edit Colors… » and select « Stayed ». Select the green color and click on the « OK » button.

tableau chart compare paralell data mining science

As you can see, the color changed in the 2 bar charts.

tableau chart compare paralell data mining science

Click on the tab of the 2nd bar chart.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

Removes the « Exited » variable from « Colors » to remove colors only in the 2nd bar chart.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

Removes the « SUM(Number of Records) » variable from « Label » to remove the labels only in the 2nd bar chart.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

We will add color on this 2nd bar chart. Click on « Colors », click on « More colors… » and select the blue color. Click on the « OK » button.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

Now, we would like to see the colors vary in intensity depending on the number of observations. Take « SUM(Number of Records) » from the 2nd line in « Rows » and holding « Ctrl » or « Command », move it to « Colors ».

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

Cool ! We will take care of the 1st bar chart. Select the tab of the 1st bar chart.

tableau chart compare paralell data mining science

Click on « Colors ». Click on « Edit Colors… ». Select « Stayed ». Select the brown color and click on the « OK » button.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

For more clarity, we will add labels in 2nd bar chart. Click on the tab of the 2nd bar chart. Take « SUM(Number of Records) » from « Colors » and holding « Ctrl » or « Command » and move it to « Labels ».

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

Perfect. Now we will change the location of the bar chart. We will put the 2nd bar chart instead of the 1st bar chart. According to the logic of « Rows » and « Columns », simply put the 2nd line « SUM(Number of Records) » to the left to pass in 1st line.

tableau chart compare paralell data mining science

tableau chart compare paralell data mining science

BOOM, the bar chart of the age distribution is going over because it’s in the 1st line in « Rows ». With these changes, tabs to change the bar charts have changed order.

Observation

What we can observe with these bar chart is that we see on the 1st bar chart that the majority of bank’s clients are in the age group of 30 to 34 years old and 35 to 39 years old. In these 2 age groups, we see on the 2nd bar chart that client of 30 to 34 years old are less likely to leave the bank than clients between 35 and 39 years old. Look at ages 30 to 34, the rate of clients leaving the bank is 8% while in the 35 to 39 age group, the number of clients leaving the bank is 13%.

In the age group of 40 to 54 years old, we see on the 2nd bar chart that the rate of clients leaving the bank is increasing and is above of the average rate of clients leaving the bank (20%). But we see in the 1st bar chart that the number of clients in the age group of 40 to 54 years old decrease with the age groups.

Do you remember the potential for anomalies in age groups 75, 85 and 90 ? We’ll check it. In the 1st bar chart we can see that there are 11 clients in the age group of 80 to 84 years old, 2 clients in the age group of 85 to 89 years old and 2 clients in the age group of 90 to 94 years old. We can conclude that these observations in age group of 80, 85 and 90 aren’t very significant from a statistical point of view because 2 clients is something negligible in this sample of 10 000 clients.

In the first age group of 15 to 19 years old, we can see that there are 49 clients, which is not very significant.

Compare these 2 bar chart in parallel allows us to have additional insights.

Subscribe to my newsletter and share this article if you think it can help someone you know. Thank you.

-Steph

Create Bins and View Distributions

tableau, bins, bar, chart, distribution, age, data, science

I have just enrolled in a Data Science course on Udemy  and I learned good stuff.

It’s cool, you finished the 1st part. Now we’re going to do more deep Data Mining analysis with this bank’s dataset.

tableau, bins, bar, chart, distribution, age, data, science

To make these analyzes more deep, we’ll create a more statistical approach.

To do that we will create a new tab.

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

For this new tab, we want to understand how client distributed according to their age. Is there a majority of young or old people ?

tableau, bins, bar, chart, distribution, age, data, science

Move the variable « Age » in « Columns ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

As we want to see the distribution of client ages, we need to use the variable « Number of Records » to see the number of observations. Move the variable « Number of Record » to « Rows ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

Boom, we have a chart but there is only one point on the top right. What happened is that Tableau took the sum of the ages of all the bank’s clients and the sum of all the « Number of Records », it means the total number of clients, 10 000 clients.

We’ll find a solution but before we’ll change the format to better see the chart. Right-click in the middle of the chart and select « Format ».

tableau, bins, bar, chart, distribution, age, data, science

For the font’s size, select « 12 ».

tableau, bins, bar, chart, distribution, age, data, science

Here you can see that the total age is 39 218 but that’s not what we’re looking for. What we want to see is the number of clients for each age.

I’ll explain what’s going on. We took the aggregated sums of our variables. Aggregate means that we took the total sum of the variable for each category. We added the ages but in fact we want to see the total number of observations for each age separately.

To have that, just click on the arrow in « SUM(Age) » in « Columns ».

tableau, bins, bar, chart, distribution, age, data, science

Then select « Dimensions »

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

You see, Tableau doesn’t take the aggregated sum of ages but it takes ages separately. We have a curve that shows us the continuous distribution of our clients ages. That is to say, for each age, the curve gives is the number of clients of this age.

We’ll look at the dataset. Right-click on « Churn Modelling » and select « View Data… ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

There is window that appears that shows us the data in detail. If you scroll to the right, you will find the column « Age ».

tableau, bins, bar, chart, distribution, age, data, science

We see that the ages rounded. As all ages rounded, Tableau is able to group clients by age. By positioning the mouse on the curve, we can see that there are 200 clients who are 26 years old.

tableau, bins, bar, chart, distribution, age, data, science

If in the dataset, ages weren’t rounded, you would have seen clients with 26.5 or 26.3 years. It would create a lot of irregularity, there would be plenty of spikes with lots of variations.

Oooooh look, there is a variation that isn’t normal.

tableau, bins, bar, chart, distribution, age, data, science

Let’s analyze it in detail. Around this peak, we see that there are 348 clients who are 29 years old.

tableau, bins, bar, chart, distribution, age, data, science

Here, 404 clients who are 31 years old.

tableau, bins, bar, chart, distribution, age, data, science

And this peak down that shows us that there are 327 clients who are 30 years old.

tableau, bins, bar, chart, distribution, age, data, science

How to explain this irregularity ? It’s possible that many people of 29 years old are about to turn 30 years old and many people of 31 years old who just had 31 years old. It’s chance that make us have inaccuracies. You may have other inaccuracies if you data isn’t precise and rounded. In our case, the ages are rounded but we want to get rid of our small irregularity that we see on our curve.

There is way to see our distribution without our irregularities, it’s « bins ». « Bins » consists of grouping the information into different categories. That is we’re going to regroup our clients in different age groups.

Right-click on « Age » in « Measures ». Select « Create » and select « Bins… ».

tableau, bins, bar, chart, distribution, age, data, science

A window appears. We’ll group our clients in 5-years increments. In « Size of bins », write « 5 » and click on the « OK » button.

tableau, bins, bar, chart, distribution, age, data, science

As you can see, the variable « Age » has remained in « Measures » but there is a new variable in « Dimensions ».This is the variable we created « Age(bins) ».

tableau, bins, bar, chart, distribution, age, data, science

Our « Age(bins) » variable was correctly placed in « Dimensions » because it is a category variable because each category corresponds to a 5-year age group.

For example, one category is 20 to 24 age group. Now we’ll create a new distribution based on « bins ».

To do that, we’ll remove the variable « Age » from « Columns » with a click and drag outside.

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

You move the variable « Age(bins) » from « Dimensions » to « Columns ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

Note

In this case, it’s not possible to directly replace « Age » by « Age(bins) » over « Age » on « Columns ». This is because « Age » is a measure and « Age(bins) is a dimension.

That’s nice distribution, it’s usually the type of distribution (chart) we see in economics or mathematics. The difference with the old chart is that this chart is discrete. This chart is discrete because the clients grouped by age group while the previous chart was continuous.

On this distribution (chart), each bar corresponds to an age range. For example, this bar corresponds to the 25-29 age group.

tableau, bins, bar, chart, distribution, age, data, science

Now, we’ll change the colors.

In « Row », move « SUM(Number of Record) » while holding down the « Ctrl » or « Command » key on your keyboard to « Colors ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

We get our distribution in blue but we’ll change the color to red. Click on « Colors » and click on « Edit Colors »

tableau, bins, bar, chart, distribution, age, data, science

In the window that appears, click on the blue square on the right to display the color pallet.

tableau, bins, bar, chart, distribution, age, data, science

Select the red color and click on the « OK » button.

tableau, bins, bar, chart, distribution, age, data, science

Click on the « OK » button of the « Edit Colors » window.

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

To facilitate the reading of the bar chart, we’ll add the number of clients in each age group. In « Row », move « SUM (Number of Record) » while holding the « Ctrl » or « Command » key on your keyboard to « Label ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

That’s it, we can see how many clients there are in each age group.

We see that the dominant bar is the 35-39 age bracket and the second dominant bar is the 30-34 age bracket. Overall, we can see that most clients are between 25 and 40 years old, which seems consistent.

On our bar chart, we have absolute values. We’ll replace that with percentages. Click in the little arrow in « SUM(Number of Records) » in « Label » and you select « Add Table Calculation… » but I’ll show you another way to do it.

tableau, bins, bar, chart, distribution, age, data, science

Instead of clicking « Add Table Calculation… », click on « Quick Table Calculation » and select « Percent of total ».

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

It’s cool, we have the exact percentage of people in each age bracket. Now, we can see that in the 25 to 40 age group, we have 20 + 23 +17= 60% of clients.

I’ll show you one last thing.You can change the size of the slices easily, just click on « Age(bins) » and select « Edit ».

tableau, bins, bar, chart, distribution, age, data, science

In the windows, you can change the size of the slices (bins). Put « 10 » instead of « 5 » to get 10-years slices. Click on the « OK » button.

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

Now, we have a distibution with fewer slices and the dominant slice is 30 to 39 years old.

Well, it was just to show you how to change the size of bins. To go back to the old distribution with the 5-years slices, click on « Back » button.

tableau, bins, bar, chart, distribution, age, data, science

tableau, bins, bar, chart, distribution, age, data, science

As you can see, the values on bars are in percentages but the values on the axis are in absolutes values. Here is an exercise that I ask you to do : « Put the values of the axis in percentage ». I’ll give you the answer the next article.

Share this article if you think if can help someone you know.Thank you.

-Steph

Work With An Alias

data science alias bar chart tableau mining

I have just enrolled in a Data Science course on Udemy and I learned good stuff.

In the last article, I showed you how to do a simple A/B test. We will continue with the result we had with the A/B test.

data science alias bar chart tableau mining

Here is the result of the A/B test. What is in orange is the percentage of men who left the bank, it’s 16%. What is in blue is the percentage of women who left the bank, it’s 25%.

With our bar chart we can quickly see that women are more likely to leave the bank than men, all the rest being equal in our sample.

I remind you that this is a basic A/B test. There are 2 type of A/B test, the basic A/B test and the statistical A/B test. The statistical A/B test is done with a statistical test like the KHI-2 test. For our case, the basic A/B test already give us good insights.

To make our bar chart even easier to read, we will work with aliases.

The first thing we will do is we will improve the format. Right-click on this space between « Gender » and the bars and select « Format… ».

data science alias bar chart tableau mining

The « Sheet » tab appears. In « Worksheet » changes the text size to « 12 ».

data science alias bar chart tableau mining

What is good with data mining is that we aren’t obligated to make a perfect chart because we don’t have to present them in a report to managers or a meeting.

For example, if I had to present this chart in a report, it would be necessary to change the vertical title. But we only make a model so this change isn’t necessary.

Now, look at this rectangle. We can see « Exited », « 0 » and « 1 ».

data science alias bar chart tableau mining

« 0 » means that the client stayed in the bank and « 1 » means that the client left the bank. We can also see that client who left the bank are in orange so 25% for women and 16% for men. And the client who stayed in the bank are blue so 75% for women and 84% for men.

We did an excellent basic A/B test but it would be much easier to read if we replace « 0 » with « Stayed » and « 1 » with « Exited ».

With aliases we can do that. An alias is to replace the binary results « 0 » and « 1 » with « Stayed » and « Exited » because it’s not easy to remember the meaning of « 0 » and « 1 ».

There are 2 ways to do it : create a calculated field or use aliases.

We will use aliases. Know that aliases are not going to change the « 0 » and « 1 » in the dataset, this change is only in Tableau.

In « Dimensions », right-click on « Exited » and select « Aliases… ».

data science alias bar chart tableau mining

data science alias bar chart tableau mining

A small window appears. In this small window, you can create an alias for each value contained in the « Exited » variable.

The variable « Exited » contains the value « 0 » and « 1 ». For the value « 0 », we will create the alias « Stayed » to say that the client stayed in the bank. For the value « 1 », we will create the alias « Exited » to say that the client left the bank. Then click on the « OK » button.

data science alias bar chart tableau mining

Look, we can see the new values in the rectangle.

data science alias bar chart tableau mining

The values « 0 » and « 1 » have been replaced by « Stayed » and « Exited ».

Now that the aliases saved, we will take the variable « Exited » in « Dimensions » and move it to « Label ».

data science alias bar chart tableau mining

data science alias bar chart tableau mining

Look, we have our aliases « Stayed » and « Exited » on the bar chart.

In this ways, it’s easier for people to read the bar chart without asking what meaning of « 0 » of « 1 » values. « Stayed » and « Exited » are clearer.

Now you know how to use aliases so that people can easily read the binary values of a chart.

Share this article if you think it can help someone you know. Thank you.

-Steph

Dataset For Data Mining

dataset data mining

I have just enrolled in a Data Science course on Udemy  and I learned good stuff.

To have the dataset to do Data Mining, you need to go to the superdatascience website . In « Part.1 Visualization », you see the section « How to use Tableau for Data Mining ». Click on « Churn Modeling » to download the file.

dataset data mining

Once you have downloaded the file, move the file to the directory you created for the course. In this directory, create a new directory (unless you already do it) named « 2.Chunk investigation ».

dataset data mining

dataset data mining

Open this fiel with Excel or with other spreadsheet software.

dataset data mining

Know that we use this dataset for the visualization part but we will also use this dataset for the modeling part.

Let’s analyze the data of this dataset.

This dataset is quite large because it contains 10 000 lines and a few columns. This is the list of a bank’s client. The client information is :

  • Customer id (login)

  • Surname (last name)

  • Credit score ( is the measure that indicates the client’s ability to borrow)

  • Geography (client’s country)

  • Gender (male or female)

  • Age

  • Tenure -(the number of years the client is in the bank)

  • Balance (balance of the client’s bank account)

  • NumOfProduct (number of product that the client has in the bank – credit card, contract, account)

  • HasCrCard (does the client have a credit card ?)

  • IsActiveMember (did the client use his/her credit card during the last month ?)

  • EstimatedSalary (the bank’s estimate of the client’s annual salary)

  • Exited (did the client leave the bank ?)

Now, I will explain the context related to this dataset. This bank has branches in several countries like Germany, Spain and France. This bank noticed that lately there were many clients who left the bank. The bank has a report called « churn rate » which is the customers rate who leave the bank and for a few months the « churn rate » is really higher than usual. It’s for this reason that the bank needs a data scientist (you) to find the problem and propose solutions.

This dataset is a small sample of clients bank. These are 10 000 randomly selected client.

The column « Exited » is a column that didn’t exist before. This column has created when the bank realized that there was an abnormal number of client who were leaving the bank.

dataset data mining

Then the bank observed these clients for 6 months to see which client left the bank.

dataset data mining

In the « Exited » column, the number « 1 » means that the client left the bank and the number « 0 » means that the client stayed in the bank.

To analyze this dataset, you’ll need to do A/B Tests. For exemple, a classic A/B Test is to see if women are more likely to left the bank than men. That’s means, see the number of men who left the bank, see the number of women who left the bank and then normalize by the total number of clients. It’s important to normalize the number of clients because there are not the same proportions of women as men. Next, based on the last column « Exited », you’ll find out if it’s the men or women who are likely to left the bank.

Once you have relevant results, you can show your report to the bank. And with this report you should be able to propose solutions to the bank. For example, if the report says that women leave the bank in bulk, it’s because there is a problem and it’s necessary to see whether the bank is offering women something right. Or it’s possible that another bank offers a much more attractive offer for women or something else.

You will learn how to investigate in the dataset and find answer through client information with A/B tests.

Share this article if you think it can help someone you know. Thank you.

-Steph

Data Science Domains

matrix

I have just enrolled in a Data Science course on Udemy and I learned good stuff.

In Science, there are several domains. In Data Science, it’s the same.

data science domain

Data Science is composed of 3 fields : computer science, math and statistics and domain knowledge. But for some years, this changed a bit. Data Scientists need to have other skills than programming and statistics.

Look at this new diagram :

data science domain

Let’s look at these skills in detail.

Statistics

Data is the basis of the Data Scientists so they must be able to filter the data to have relevant data that will provide them with insights. This allow Data Scientist to build models to classify the population and make reliable forecasts of future events.

Visualization

Do you know the computers langage ? Do you know bytecode like « 00100010100101010110 » ? No and it’s the same for me. It’s for this reason that Data Scientists must have the ability to see through the data and especially show them to others. This is why visualization is an important skill to show the data.

Data Mining

This is the part of the work where the Data Scientist has to make the detective like Sherlock Holmes. It’s in this phase that we must look in the data for insights and abnormalities.

Database and Data process

It’s simple, the Data Scientist cleans the data, stores and processes the data in the database.

Pattern recognition, Machine learning et Neurocomputing

These 3 disciplines help explain to computers how to learn do to a specific task on its own. There are not things I’ll learn but these are interesting disciplines for some business problems.

In our world where competition is increasingly aggressive, technical skills are no longer enough. Here are other skills that Data Scientist need to have.

Communication

communication

Data Scientists need to interact with people everyday. They have to do that because the insights are not just in the data. There are insights that we can only find by talking to people. That’s why it’s important to not afraid to talk to people to ask them questions on a daily basis.

Presentation

This is another type of communication . In this case the Data Scientist doesn’t try to extract information but to explain what he/she found to the people. This is a very important skill because the Data Scientist is the intermediary between insights and people. It’s a bit the data translator, it’s simply explain the content of data.

Domain knowledge

Data Science can be used in any industry. One day you can do research to find fraudulent transactions and another day you can build a compensation model for employees of a medical establishment.

That is why, in what industry you work, you must do research and know quickly the necessary part of the industry. The rest will come naturally. Quickly learn the basics of the industry where you work.

Practice in real situations

Proverb : « It’s by forging that one you become a blacksmith » says everything. This concept is extremely applicable in Data Science.

Programmation

The 2nd basic domain of Data Scientists. The better you talk to your computer and the more efficient you are, the more successful you will be. If you don’t know how to program, learn this from today. Programming has to become a hobby, something you like to do.

Creativity

This is what make the difference between Data Scientist and Data Analysts. To become an excellent Data Scientist, you need to work your creativity. Be curious and you will find insights that nobody would never have found.

Now you know the skills needed to become an excellent Data Scientist. As you see I have a lot to do.

Share this article if you think it can help someone you know. Thank you.

-Steph