GRcade

site23 wrote:

Jenuall wrote:I love Pandas, but I'm not sure that will be much help!

Why? It seems like a good fit for the stated problem to me.

It was an attempted joke, don't worry!

Oh OK - my bad, sorry!

Thanks for the offers of help, much appreciated.

I did do a little research on Python this morning. I have dabbled in it, but I am a novice on it, whilst very confident with Excel. I'm going to try merging data sets in Excel now and see if it can handle the task. My understanding is that pivot tables display all data in a query, even if Excel doesnt display the whole query. I hate pivot tables, but I think there may be a solution there. At least that would take the pressure away from having to cram Python study in to a two week period.

I am 50/50 on whether this task is achievable within Excel, which is disappointing for something which I picked because it was described as the task which could be completed within Excel. What's really frustrating is that I wouldn't expect to have any issues with the tasks if it was a dataset which actually fits in Excel, but I expect that half of my time will be sat waiting for Excel to calculate / load just due to the volume of data. I've already had it crash on me, just tinkering around and exploring the data.

Pandas is looking more and more likely at this point.

Edit: I have managed to get all of the data in to one combined query. To do this, I had to query all 6 CSV files and append the queries as one. As I normally query databases, this isn't something I've done before, but it was fairly intuitive. The issue was though, that Excel couldn't cope and crashed out, moaning about not having enough memory. Turns out, Office automatically installs the 32 bit version, so I've literally spent the whole day trying to fit this data in to one workbook and have finally achieved it, through reinstalling Office as 64 bit. I'm glad I've looked at this in advance, as this would have been an entire day of my 5 day allocation wasted on pulling the data in to a workbook.

I can still only view the maximum 1,048,541 rows as allowed by Excel, but all 16 million rows can be manipulated within pivot tables, so at very least I know that I can achieve something with the data now...

So, I'm now on day two of my project and I've managed to quite a lot within Python. I've managed to load the CSV files in to dataframes, merge them in to one, create additional fields and manipulate the data by creating pivot tables in Python. I am really pleased that I stuck with Python. Trying to learn it (or at least, learn the bits I needed) in less than two weeks was a big ask, but I've achieved a decent level of data analysis within Python already and hopefully this will be accepted as evidence that I can use a variety of tools. Even if it doesn't, Python is the program I expect to be using in the years to come. I expect to move to Excel for the final output towards the end of the week, giving me 3 or 4 days hands on with Python.

Whilst it has been and will continue to be difficult, I can only imagine how frustrating this task would have been in Excel, where I would likely have spent most of my week faced with system crashes due to the size of the files.

Found out today that my application to study abroad for a year has been accepted. I'm going to be in Hong Kong! I'm so excited!

That's exciting Outrunner!

And well done on becoming a Python Pro Sprouty!

Jenuall wrote:That's exciting Outrunner!

And well done on becoming a Python Pro Sprouty!

Thanks! I'm a long way from considering myself a pro at this point, but I am at least evidencing that I have skills beyond my normal toolset, which may well secure me a better grade.

My original intention was to do most of the grunt work within Python, then visualise the data output within Excel. I've come to the conclusion that Excel isn't the right tool for that either, so I am now attempting to visualise within Python. Once again, not my original intention, but it only cements my side goal of proving my ability to use a range of systems. Only downside is that again, I need to learn how to actually do it.

I have managed to put together a whisker chart within Python, which was one of the two graph types that I did not evidence in my portfolio. I am slightly panicked at this point as I doubt the final output will look as smart as my normal work, but I am hoping the exam board will see my work as a whole and appreciate that whilst it may not look as smart, I have gone to great efforts to do something outside of my comfort zone.

On a personal note, I am pretty pleased that I have actually put together some analysis within Python and look forward to using it further in the future.

Well done! Re: graphics, which plotting library are you using? If you have an hour or so spare you can usually dig into the visual options and make the plots a bit nicer. I use Matplotlib and that can look a bit ragged by default but it has a lot of settings to play with!

site23 wrote:Well done! Re: graphics, which plotting library are you using? If you have an hour or so spare you can usually dig into the visual options and make the plots a bit nicer. I use Matplotlib and that can look a bit ragged by default but it has a lot of settings to play with!

These are what I have imported, though most of the work is within Pandas. I have used numpy to replace NaN values.

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

Some of the graphs I have used:

Task2b.plot(kind='bar');

Task3.boxplot();

Task5b.plot(kind='bar',xlabel='Actual Cost')

I don't think I have used matplotlib yet, but I obviously had some intention to at some point...

I have no idea how to amend the graphs - colours, formatting, amending labeling etc!

Edit: Chucking a scatter chart into the mix. I know that's matplotlib as I had to import the function.

plt.scatter((Task4b['Actual Cost']),(Task4b['Quantity']),marker='.')
plt.xlabel('Actual Cost (Millions)')
plt.ylabel('Quantity (Millions)');

Sprouty wrote:I have no idea how to amend the graphs - colours, formatting, amending labeling etc!

Here's a Colab notebook with some examples that might be helpful.

https://colab.research.google.com/drive/1EL31gYprKHGf-JIV4bxYrTlEUXqcIeIV?usp=sharing

site23 wrote:

Sprouty wrote:I have no idea how to amend the graphs - colours, formatting, amending labeling etc!

Here's a Colab notebook with some examples that might be helpful.

https://colab.research.google.com/drive/1EL31gYprKHGf-JIV4bxYrTlEUXqcIeIV?usp=sharing

Appreciated, thank you. I have progressed loads tonight, but the chart type featured in this article is one that needs some tidying, so that looks really helpful.

I looked at this for the first proper time just now. I have a Bachelor of Arts in Social Science a 2:2 gotten from Nottingham Trent University, never mentioned it much as I have always been very very sour about getting a 2:2 was 2 marks off a 2:1 and have been bitterly annoyed with myself ever since, but for some reason I think I finally hit a bit of a curve with it and don't feel quiet as cruddy over it

I also have Early Years Professional Status (EYPS) which is a professional status for Early Years Foundation Stage educators (ages 0 – 5), which is intended to be broadly equivalent to Qualified Teacher Status (ages 5 – 18) but targeted to teaching younger kids.

Yet I am a glorified shelf stacker lol

I have now got to the point that I can produce a presentation from my analysis, all completed within Python.

There are a couple of things that I would like to add, but if I don't get them in, it wont matter (essentially, adding additional insight beyond the brief). I've put together a merged dataset (15 million records), added fields, created various views using pivot tables and produced nine graphs in the format of pie, scatter, bar, boxplot & line charts.

I would like to add in histograms and a heatmap before putting the final work together in a powerpoint presentation, both are things I think I will be able to achieve.

Thanks for the encouragement & words of advice, much appreciated!

Many congrats dude, that's brilliant news - sounds like it was very well deserved, well done!

Nicely done Sprouty!

Was he okay with the project using entirely green text?

aayl1 wrote:Nicely done Sprouty!

Was he okay with the project using entirely green text?

I actually used blue throughout my full portfoilio and the final project and there was a note on my score report criticising my use of an all blue capital font format on my presentation!

Should have used green!

I've had my results back for last semesters modules and really happy with what I got. I'm particularly pleased with my Contemporary Japanese Society module which I got 74% on. I loved that module and the tutor was great. I'm now much less worried about keeping my grade average above 60% so I can go to HK.

Been a while since I posted.

I managed to graduate with a first in Molecular and Cellular Biology MSci (Hons) from The University of Glasgow in 2020.

I'm now 6 months in to a PhD at The University of Strathclyde and all is going well so far.