And leverage Durable Functions to reduce the burden when managing long-living serverless workloads

Photo by Larisa Birta on Unsplash

Microservices architectures have been widely used by enterprise organizations to split their applications into smaller parts. Experience shows that, along with benefits, microservices also bring challenges for software development teams. Integrating related parts is among the most common issues and there’s no silver bullet to tackle the problem. Microservices Choreography vs Orchestration: The Benefits of Choreography brings reasonable thoughts on this matter.

Detached events and functions (choreography) fit concise workloads quite well but may bring unwanted complexity to the overall solution when it comes to workflow-like processing. This is why I believe there’s either room for orchestration. …


A sequel to “WebSockets and AsyncIO: Beyond 5-line Samples”

sports court with lines and numbers marked
sports court with lines and numbers marked
Photo by Pocky Lee on Unsplash

In this sequel to a previous article, I discuss design, libraries, and quality assurance in the scope of WebSockets, asynchronous programming, and concurrency with Python.

In the first article, we took a look at the WebSockets technology and how it fits Python. I’ve also proposed a fictitious use case to add more context to the discussion.

Now it’s time to shift attention to AsyncIO, a built-in library to write concurrent code using the async/await syntax. (I recommend taking a look at this tutorial if you’re not familiar with it.) …


Thoughts on async and concurrent programming with Python

A man and a woman at a computer
A man and a woman at a computer
Photo by heylagostechie on Unsplash

My team was recently assigned a task that comprises writing a Python client to interact with a specific WebSockets API. It’s led us to learn new things and the goal of this article is to share our progress. We believe there’s always room for improvement, so please feel free to comment and bring your expertise to the table.

When we say “beyond 5-line samples,” we mean that we found guidance when talking to our coworkers and searching the internet, but we missed references that bring related technology, design tips, programming libraries, and quality assurance matters together. …


Leverage Data Catalog to discover & annotate Qlik Sense assets

Background photo by Lauren Mancke on Unsplash

In this blog post, I’m going to share thoughts and knowledge that drove the development of a new sample connector for Google Data Catalog: the Qlik Sense connector.

This is the third open-source connector that allows customers to ingest metadata from BI Systems into Data Catalog: the ones for Looker and Tableau have been available for a while.

Disclaimer: Google and/or Google Cloud do not officially support any tool to connect Data Catalog to non-GCP systems at the time this article has been written down (January 2021). …


A custom model

More and more companies migrate their data workloads to the cloud every day. At the same time, new data protection regulation programs become effective around the globe. Such simultaneous events lead companies to enforce their Data Governance standards intending to avoid legal charges due to inappropriate data management.

Data Governance frameworks are hence becoming more common, but providers not necessarily speak the same language — what any of us could certainly expect as each player has their own strategies to tackle problems… On the other hand, customers have distinct requirements that are not fulfilled by a single platform. …


Hints and tips to create your own driver from scratch

Photo by Aron Visuals on Unsplash

GitLab is rich and scalable when it comes to software architecture, and Runner is one of the key components of such an ecosystem. It is responsible for running CI/CD jobs and deployed decoupled from the GitLab instance.

Runners come in multiple flavors: Docker, Kubernetes, Shell, SSH, VirtualBox, and others. Technically speaking, each “flavor” is implemented as an Executor. But what if you want to run the CI/CD jobs in an infrastructure that is not supported by the native executors or to scale your Runners fleet in a custom strategy that better fits your needs? …


How to leverage CSV, Cloud Storage, and Data Studio to support fast data-driven decision making

Photo by Luke Chesser on Unsplash

What comes to mind when you hear “data-driven”? Expensive BI system licenses? Complex ETL processes that prepare data to populate dashboards? Corporate or technical constraints that prevent you from accessing relevant data to make decisions? Or affordable tools that are available whenever you need them, even to guide simple decisions? I hope you answered “The last option!”, but I bring good news for you in case you didn’t.

Well, “data-driven” has been a trending topic for a long time and it is not difficult to understand why. As all of us know, people, gadgets, and devices — aka “things” —…


Leverage Data Catalog to discover & annotate Tableau assets

Background photo by Lauren Mancke on Unsplash

The Google Cloud Data Catalog Team has recently announced its product is now GA and ready to accept custom (aka user-defined) entries! This brand new feature opens up scope for integrations and now users can leverage Data Catalog’s well-known potential to manage metadata from almost any kind of data asset.

To demonstrate how it works, I’ll share design thoughts and sample code to connect Data Catalog to market-leader Business Intelligence/Data Visualization tools, covering Tableau metadata integration in this blog post. They come from the experience of participating in the development of fully operational sample connectors, publicly available on GitHub.

Disclaimer…


Leverage Data Catalog to discover & annotate Looker assets

Background photo by Lauren Mancke on Unsplash

The Google Cloud Data Catalog Team has recently announced its product is now GA and ready to accept custom (aka user-defined) entries! This brand new feature opens up scope for integrations and now users can leverage Data Catalog’s well-known potential to manage metadata from almost any kind of data asset.

To demonstrate how it works, I’ll share design thoughts and sample code to connect Data Catalog to market-leader Business Intelligence/Data Visualization tools, covering Looker metadata integration in this blog post. They come from the experience of participating in the development of fully operational sample connectors, publicly available on GitHub.

Disclaimer…


How the most important soft skill may help you to succeed

Photo by Power Digital Marketing on Unsplash

Some of my co-workers have been encouraging me to write something about soft skills for a while, so let me try!

Soft skills are a set of competencies a person should develop in order to empathize and do a better job, especially when working with teams. They’re no more or less important than technical (aka hard) skills. They are peers — they make relationships more human.

I’ll start with some insights from Gordon Haff (Red Hat). …

Ricardo Mendes

data engineering lead @ ciandt.com • hobbyist tech writer • dad • birder

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store