Sequin is a tool for capturing changes and streaming data out of your Postgres database, guaranteeing exactly once processing. What does that mean?

Sequin leverages Postgres's Write-Ahead Log (WAL) to transform it into a stream or message queue, providing a more efficient alternative to Kafka and Simple Queue Service (SQS). This means that you can:

• Replicate data from your existing tables to other apps, databases, caches, materialized views, or frontend clients
  
  
• Build event processing workflows such as triggering side effects when data in Postgres changes

It achieves this by monitoring the Postgres WAL to track changes in your table's rows. This can be done at a granular level, capturing every create, update,nd delete operation, or more efficiently, by simply retrieving the latest version of each row.

In the first case Sequin acts as a replacement for

• Postgres triggers
• SQS
• ORM hooks
• WAL events

and in the latter as a replacement for

• Kafka
• COPY
• Manual syncs

In either case it manages to replace many cases of DIY or difficult setups, like those involving Kafka or Debezium, with a single all encompassing service.

To stream data using Sequin, you create "sequences" for each table you want to stream so that on the other end you can consume those rows from the sequence using clients that work with HTTP Push like Webhooks or HTTP Pull like SQS. Sequin takes care of the clients too since it lets you create them from withing the platform, guaranteeing exactly once processing.

On that matter, while no system can guarantee exactly-once delivery , at-least-once delivery and exactly-once processing can work together:

Processing is the the full message life-cycle: the message was delivered to the receiver, the receiver did its job, and then the receiver acknowledged the message. 

With that definition, SQS, Kafka, and Sequin are all systems that guarantee exactly-once processing. The term processing captures both the delivery of the message and the successful acknowledgment of the message; "exactly-once processing" indicates it's a messaging system with at-least-once delivery and acknowledgments.

Practically with Sequin in place :

• You never miss a row or change: Sequin ensures all database changes are delivered and processed exactly once by consumers
  
  
• You get SQL-based routing: Filter and route messages to consumers using SQL where conditions
  
  
• You can rewind consumers to any row on your table. Or republish select messages that match a SQL query
  
  
• Can start from anywhere: Consumers can start processing rows from any point in a table
  
  
• Application writing wise, you don't have to use stored procedures and triggers to define your app's business logic, since the consumers can be written in the language of your choice

Diverging from the usual practice of extending Postgres's functionality with extensions, Sequin offers a Docker image coming with a web console/UI for configuration that you can run next to your Postgres database. It works with any Postgres database version 12+ onwards.

Of course it is open source, therefore the easiest way to get started is to grab the docker/docker-compose. yaml file from the repository which bundles a Postgres and Redis instance as well as a Sequin server. Note that Sequin is also offered as a cloud-managed service.

Since Sequin is written in Elixir, there's also an accompanying lightweight Elixir SDK for sending, receiving, and acknowledging messages in Sequin streams. Utilizing it , an example of sending , receiving, and acknowledging messages in Sequin streams looks like : 

I left the most exciting features yet to be announced, Transforms; that is you can transform message payloads by writing functions in Lua, JavaScript, or Go before they’re delivered to your consumers. This is useful for transforming data into a format more suitable for your receiving end of the application.

That is, instead of your app's end point consuming json output in wal2json format (coming out of Sequin), you can write code that transforms the package to a custom format that your app can work with before it exits Sequin.

Contrast that with the trouble I had to go through when manually implementing a CDC/Webhook functionality that involved a complex pipeline of triggers, database events and Perl transformation code, under the Ingres dbms in order to transform and manipulate data collected by SQL queries into HL7 packets before they got automatically sent to a Web service at the end of the pipeline. That is GUI/App-->Enter Data-->SQL-->HL7-->Web Service, automatically when a value gets inserted. Read "Connecting To The Outside World with Perl and Database Events" if you want to find out how it was done.

In conclusion Sequin is a one-stop solution that takes a lot of pain out of replicating and synchronizing data, broadcasting events, distributing workloads and decoupling services
by turning Postgres into a continuous stream, and most importantly, a stream that lives alongside its data.

More Information

Sequin 

Related Articles

Connecting To The Outside World with Perl and Database Events

PostgreSQL 17 Is Released   

To be informed about new articles on I Programmer, sign up for our weekly newsletter, subscribe to the RSS feed and follow us on Twitter, Facebook or Linkedin.

Comments

Make a Comment or View Existing Comments Using Disqus

or email your comment to: comments@i-programmer.info