Ray is a general purpose distributed computing framework. At a low level, Ray provides fault-tolerant primitives that support applications running across multiple processors. At a higher level, Ray supports scalable reinforcement learning, including the common problem of hyperparameter tuning. In a previous episode, we explored the primitives of Ray as well as Anyscale, the business The post Ray Applications with Richard Liaw appeared first on Software Engineering Daily.

Data – Software Engineering Daily

Ray Applications with Richard Liaw

Pandas is a Python data analysis library, and an essential tool in data science. Pandas allows users to load large quantities of data into a data structure called a dataframe, over which the user can call mathematical operations. When the data fits entirely into memory this works well, but sometimes there is too much data The post Modin: Pandas Scalability with Devin Petersohn appeared first on Software Engineering Daily.

Modin: Pandas Scalability with Devin Petersohn

A large codebase cannot be searched with naive indexing algorithms. In order to search through a codebase the size of Uber’s it is necessary to build a much more sophisticated indexing system than simple pure text search. SourceGraph is a system for universal code search. It allows developers to more easily onboard to a new The post SourceGraph: Code Search and Intelligence with Beyang Liu appeared first on Software Engineering Daily.

SourceGraph: Code Search and Intelligence with Beyang Liu

ADP has been around for more than 70 years, fulfilling payroll and other human resources services. Payroll processing is a complex business, involving the movement of money in accordance with regulatory and legal strictures.  From an engineering point of view, ADP has decades of software behind it, and a bright future of a platform company The post ADP Engineering with Tim Halbur appeared first on Software Engineering Daily.

ADP Engineering with Tim Halbur

M3 is a scalable metrics database originally built to host Uber’s rapidly growing data storage from Prometheus. When Rob Skillington was at Uber, he helped design, implement, and deploy M3. Since leaving Uber, he has co-founded a company around a hosted version of M3 called Chronosphere. If you have access to a scalable metrics database, The post Chronosphere: Scalable Metrics Database with Rob Skillington appeared first on Software Engineering Daily.

Chronosphere: Scalable Metrics Database with Rob Skillington

DynamoDB is a managed NoSQL database service from AWS. It is widely used as a transactional database to fulfill key-value and wide-column data models. In a previous show with Rick Houlihan, we explored how to build a data model and optimize the query patterns for a NoSQL database.  Today’s show is about DynamoDB specifically: partitioning, The post DynamoDB with Alex DeBrie appeared first on Software Engineering Daily.

DynamoDB with Alex DeBrie

As a user browses a webpage, that browser session generates events that need to be recorded, validated, enriched, and stored. This data is sometimes called customer data infrastructure, or CDI. This data requires a full stack of different tools: a system on the frontend to collect the data, middleware to transport the data, and backend The post Snowplow Analytics: Data Collection Platform with Alex Dean appeared first on Software Engineering Daily.

Snowplow Analytics: Data Collection Platform with Alex Dean

A software company manages and interacts with hundreds of APIs. These APIs require testing, performance analysis, authorization management, and release management. In a word, APIs require collaboration. Postman is a system for API collaboration. It allows users to test APIs with collections of requests, monitor the API responses, and visualize the query results. Users of The post Postman: API Development with Abhinav Asthana appeared first on Software Engineering Daily.

Postman: API Development with Abhinav Asthana

Originally published May 2, 2017. We are taking a few weeks off. We’ll be back soon with new episodes. A new programmer learns to build applications using data structures like a queue, a cache, or a database. Modern cloud applications are built using more sophisticated tools like Redis, Kafka, or Amazon S3. These tools do The post Data Intensive Applications with Martin Kleppman (Summer Break Repeat) appeared first on Software Engineering Daily.

Data Intensive Applications with Martin Kleppman (Summer Break Repeat)

Originally published October 24, 2019. We are taking a few weeks off. We’ll be back soon with new episodes. Redis is an in-memory database that persists to disk. Redis is commonly used as an object cache for web applications. Applications are composed of caches and databases. A cache typically stores the data in memory, and The post Redis with Alvin Richards (Summer Break Repeat) appeared first on Software Engineering Daily.

Redis with Alvin Richards (Summer Break Repeat)

Apache Airflow was released in 2015, introducing the first popular open source solution to data pipeline orchestration. Since that time, Airflow has been widely adopted for dependency-based data workflows. A developer might orchestrate a pipeline with hundreds of tasks, with dependencies between jobs in Spark, Hadoop, and Snowflake. Since Airflow’s creation, it has powered the The post Apache Airflow with Maxime Beauchemin, Vikram Koka, and Ash Berlin-Taylor appeared first on Software Engineering Daily.

Apache Airflow with Maxime Beauchemin, Vikram Koka, and Ash Berlin-Taylor

The life cycle of data management includes data cleaning, extraction, integration, analysis and exploration, and machine learning models. It would be great if all of this data management could be handled with automation, but unfortunately that is not an option. For most applications, data management requires a human in the loop. A human in the The post Human in the Loop Data Analytics with Aditya Parameswaran appeared first on Software Engineering Daily.

Human in the Loop Data Analytics with Aditya Parameswaran

Uber needs to visualize data on a range of different surfaces. A smartphone user sees cars moving around on a map as they wait for their ride to arrive. Data scientists and operations researchers within Uber study the renderings of traffic moving throughout a city. Data visualization is core to Uber, and the company has The post Uber’s Data Visualization Tools with Ib Green appeared first on Software Engineering Daily.

Uber’s Data Visualization Tools with Ib Green

A frontend developer issuing a query to a backend server typically requires the developer to issue that query through an ORM or a raw database query. Prisma is an alternative to both of these data access patterns, allowing for easier database access through auto-generated, type-safe query building tailored to an existing database schema. By integrating The post Prisma: Modern Database Tooling with Johannes Schickling appeared first on Software Engineering Daily.

Prisma: Modern Database Tooling with Johannes Schickling

Many data sources produce new data points at a very high rate. With so much data, the issue of data quality emerges. Low quality data can degrade the accuracy of machine learning models that are built around those data sources. Ideally, we would have completely clean data sources, but that’s not very realistic. One alternative The post HoloClean: Data Quality Management with Theodoros Rekatsinas appeared first on Software Engineering Daily.

HoloClean: Data Quality Management with Theodoros Rekatsinas

Databases and data engineering episodes of Software Engineering Daily

hubhopper

Data – Software Engineering Daily

hubhopper