Data engineers have a lot of different resources they can learn from. One great example is books. There are tons of great data engineering books that you can read to learn about spark, streaming, redshift, snowflake and more. Here are the books from this video The Data Warehouse Toolkit https://aatinegar.com/wp-content/uploads/2016/05/Kimball_The-Data-Warehouse-Toolkit-3rd-Edition.pdf Learning PySpark https://amzn.to/31MQqun Transaction Processing https://amzn.to/3oGIm6W Streaming Systems https://amzn.to/3IHh7RF Data Pipelines Pocket Guide https://amzn.to/3oQHk8N If you enjoyed this video, check out some of my other top videos. What Skills Do Data Engineers Need? https://www.youtube.com/watch?v=LgSHaOvNodA Data Engineering Project Ideas https://youtu.be/LJkVvNWlO0g If you want to learn more about machine learning, check out DataCamps Machine Learining Course https://bit.ly/3BeLEml If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here. https://seattledataguy.substack.com/​​ Or check out my blog https://www.theseattledataguy.com/ Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio _____________________________________________________________ Subscribe: https://www.youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmation=1 _____________________________________________________________ About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consults on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data. *I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. ... https://www.youtube.com/watch?v=xtfuO7kGJeY

SQL is becoming a common skill requirement across industries. Companies like Amazon and Google often will require their data analysts, data scientists and product managers to at least be familiar with SQL. This is because SQL remains the language of data. So, in order to be data driven, people need to know how to access and analyze data. With so many people looking at, slicing, manipulating and analyzing data with SQL we wanted to provide some tips to help improve your SQL. These tips and tricks we have picked up along the way while writing SQL. Some of them are do’s and don’ts others are just best practices. Overall we hope that they will help bring your SQL to the next level. Some of the tips will be things you shouldn’t do, even when you might be tempted and others are best practices that will help ensure that you can trust your data. Overall, they are both meant to be informative as well as reduce possible future headaches. Here are 4 tips and tricks to help you improve your SQL game as a data scientist, engineer or analyst - Use A Case Statement With A Sum() Function - Don’t Use Avg() on an Average It’s Not the Same - Learn How To Use The Lead() and Lag() Functions - Learn How To Use Arrays And Maps These are just 4 of a whole list of tips we want to share with you! Let us know if you have any further questions and feel free to comment and reach out to our team. You can find us at: https://www.theseattledataguy.com/ https://medium.com/@SeattleDataGuy https://twitter.com/SeattleDataGuy Also read more about Postgres Arrays Here https://www.postgresql.org/docs/9.1/arrays.html Thanks again! ... https://www.youtube.com/watch?v=kSt9NV-qZkc

If you're interested in learning more about consulting, then sign up for the next cohort of my course! https://forms.gle/Ufd9pMsZxo1isLtW9 If you enjoyed this video, check out some of my other top videos. Top Courses To Become A Data Engineer In 2022 https://www.youtube.com/watch?v=kW8_l57w74g What Is The Modern Data Stack - Intro To Data Infrastructure Part 1 https://www.youtube.com/watch?v=-ClWgwC0Sbw If you would like to learn more about data engineering, then check out Googles GCP certificate https://bit.ly/3NQVn7V If you'd like to read up on my updates about the data field, then you can sign up for our newsletter here. https://seattledataguy.substack.com/​​ Or check out my blog https://www.theseattledataguy.com/ And if you want to support the channel, then you can become a paid member of my newsletter https://seattledataguy.substack.com/subscribe Tags: Data engineering projects, Data engineer project ideas, data project sources, data analytics project sources, data project portfolio _____________________________________________________________ Subscribe: https://www.youtube.com/channel/UCmLGJ3VYBcfRaWbP6JLJcpA?sub_confirmation=1 _____________________________________________________________ About me: I have spent my career focused on all forms of data. I have focused on developing algorithms to detect fraud, reduce patient readmission and redesign insurance provider policy to help reduce the overall cost of healthcare. I have also helped develop analytics for marketing and IT operations in order to optimize limited resources such as employees and budget. I privately consult on data science and engineering problems both solo as well as with a company called Acheron Analytics. I have experience both working hands-on with technical problems as well as helping leadership teams develop strategies to maximize their data. *I do participate in affiliate programs, if a link has an "*" by it, then I may receive a small portion of the proceeds at no extra cost to you. ... https://www.youtube.com/watch?v=XcwY-HRktRA

Using cloud tools like Bigquery and SaturnCloud.io make data scientists, data engineers and analysts work much easier. Both tools remove the need to manage as much infrastructure on your companies side and instead you can use AWS, Google cloud and Azure to run your analysis and manage your data only when needed. Today we wanted to use discuss using these cloud tools that are available to everyone to analyze a medical data set. In particular will be using the Kaggle data set for medicare providers. This has information on diagnosis related groups average costs, hospital locations and interesting facts about providers and their service quality. These data sets provide you the ability to answer questions about which states have the highest quality of service (based on the metrics they provide), the highest average costs as well as look into demographics of the patient bases. Since healthcare costs tend to be one of the largest focuses, we are going to look at which states have the largest percentage of costs that are above average. We will be using Saturn.io as a platform for our Jupyter notebook and BigQuery in order to query the data. Tools We Will Use Saturn.io For those unfamiliar with tools like Saturn. Saturn.io allows you to easily spin up a VM that has already has Jupyter notebook installed. But it is really much more than just that. Using Saturn.io makes it easy to share your work with other teammates without having to deal with the hassle of making sure they have all the correct libraries installed. Also, for those of you who have had the issue of running Jupyter Notebooks locally and run out of memory, it also allows you to spin up VMs with the memory and Ram required and only pay for what you use. There are also a few other bells and whistles that really mitigate some of the complexities of work typically done from a DevOps perspective. That’s what makes tools like this great. In many ways, until you work in a corporate environment where even sharing a basic Excel document can become cumbersome, it is almost hard to explain to younger data scientists why this is so great! Big Query Due to where the data is being stored. We will also need to interact with Big Query. This is a Google Cloud service that allows you to query data. In this case, the medicare data is open to everyone but is stored on Google Cloud. As like most cloud services, this makes it convenient to manage as long as Google keeps this data public. It is really easy to pull the data and run queries without having to download a csv and load it into your own database. Connecting to Big Query For step one we will be retrieving the cms_medicare public data set. In order to do so you will need to set up Google API credentials (see instructions here). You will notice ... https://www.youtube.com/watch?v=ZalsNP6bMFg