Using cloud tools like Bigquery and SaturnCloud.io make data scientists, data engineers and analysts work much easier. Both tools remove the need to manage as much infrastructure on your companies side and instead you can use AWS, Google cloud and Azure to run your analysis and manage your data only when needed. Today we wanted to use discuss using these cloud tools that are available to everyone to analyze a medical data set. In particular will be using the Kaggle data set for medicare providers. This has information on diagnosis related groups average costs, hospital locations and interesting facts about providers and their service quality. These data sets provide you the ability to answer questions about which states have the highest quality of service (based on the metrics they provide), the highest average costs as well as look into demographics of the patient bases. Since healthcare costs tend to be one of the largest focuses, we are going to look at which states have the largest percentage of costs that are above average. We will be using Saturn.io as a platform for our Jupyter notebook and BigQuery in order to query the data. Tools We Will Use Saturn.io For those unfamiliar with tools like Saturn. Saturn.io allows you to easily spin up a VM that has already has Jupyter notebook installed. But it is really much more than just that. Using Saturn.io makes it easy to share your work with other teammates without having to deal with the hassle of making sure they have all the correct libraries installed. Also, for those of you who have had the issue of running Jupyter Notebooks locally and run out of memory, it also allows you to spin up VMs with the memory and Ram required and only pay for what you use. There are also a few other bells and whistles that really mitigate some of the complexities of work typically done from a DevOps perspective. That’s what makes tools like this great. In many ways, until you work in a corporate environment where even sharing a basic Excel document can become cumbersome, it is almost hard to explain to younger data scientists why this is so great! Big Query Due to where the data is being stored. We will also need to interact with Big Query. This is a Google Cloud service that allows you to query data. In this case, the medicare data is open to everyone but is stored on Google Cloud. As like most cloud services, this makes it convenient to manage as long as Google keeps this data public. It is really easy to pull the data and run queries without having to download a csv and load it into your own database. Connecting to Big Query For step one we will be retrieving the cms_medicare public data set. In order to do so you will need to set up Google API credentials (see instructions here). You will notice ... https://www.youtube.com/watch?v=ZalsNP6bMFg

What is Palantir? And does it have a place in the data engineering world? Thanks to @Codestrap I got the opportunity to poke at Palantirs Foundry In this first video I connect to a BigQuery dataset and pull it into Foundry. If you enjoyed this video, check out some of my other top videos. Top Courses To Become A Data Engineer In 2022 https://www.youtube.com/watch?v=kW8_l57w74g What Is The Modern Data Stack - Intro To Data Infrastructure Part 1 https://www.youtube.com/watch?v=-ClWgwC0Sbw If you would like to learn more about data engineering, then check out Googles GCP certificate https://bit.ly/3NQVn7V If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here. https://seattledataguy.substack.com/​​ Or check out my blog https://www.theseattledataguy.com/ And if you want to support the channel, then you can become a paid member of my newsletter https://seattledataguy.substack.com/subscribe FTC Disclosure: I am working in partnership with Palantir as a Foundry ambassador!.I have access to a free Foundry Stack. I don't work for Palantir. I just kept seeing videos about Snowflake vs Palantir and I wanted to dig into Palantir so I could understand the difference. Foundry screenshots are Copyright © 2022 Palantir Technologies Inc. and/or affiliates (“Palantir”). All rights reserved. All data is notional or open source. Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio _____________________________________________________________ Subscribe: https://www.youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmation=1 _____________________________________________________________ About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data. *I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. ... https://www.youtube.com/watch?v=Qa5KlRdk5UE