Skip to main content

Posts

Showing posts from December, 2016

A Data Engineer's Guide To Non-Traditional Data Storages

With the rise of big data and data science, many engineering roles are being challenged and expanded. One new-age role is data engineering . Originally, the purpose of data engineering was the loading of external data sources and the designing of databases (designing and developing pipelines to collect, manipulate, store, and analyze data). It has since grown to support the volume and complexity of big data. So data engineering now encapsulates a wide range of skills, from web-crawling, data cleansing, distributed computing, and data storage and retrieval. For data engineering and data engineers, data storage and retrieval is the critical component of the pipeline together with how the data can be used and analyzed. In recent times, many new and different data storage technologies have emerged. However, which one is best suited and has the most appropriate features for data engineering? Most engineers are familiar with SQL databases, such as PostgreSQL, MSSQL, and MySQL, whi