Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • I like data lakes and their SQL-like query languages. They feel like an advanced version of the "Everything is a file" philosophy.

    Under "Everything is a file", you can read or manipulate a wide variety of information via simple, open/read/write() APIs. Linux provides APIs to modify system settings via filesystem. Get the screen brightness setting? `cat /sys/class/backlight/device0/brightness`. Update? `echo 500 > /sys/class/backlight/device0/brightness`. No need for special APIs, just generic file operations and the kernel handles everything.

    FUSE (Filesystem in Userspace) provides even more flexibility by allowing user space programs to build their own drivers that handle any data operation via the filesystem. You can mount remote systems (via SSH) and google drive, and copying files is as easy as `cp /mnt/remote/data/origin /mnt/googledrive/data/`. Or using unique FUSE like pgfuse and redisfs, updating redis value by postgres DB data is just `cat /mnt/postgres/users/100/full_name > /mnt/redis/user_100_full_name`.

    But filesystems are only good for hierarchical data while a lot of real world data is relational. Many FUSE software tries hard to represent inherently non-hierarchical data in a filesystem. Data lake allows to use SQL, the elegant abstraction for relational data, across different underlying data sources. They can be physically distant and have different structures. A lot of real world applications are just CRUD on relational data. You can accomplish much more much easier if those data are just a big single database.

    by hamasho
  • How do you use your data lake? For me it is much more than just storing data, it is just as much for crunching numbers in unpredictable ways.

    And this is where postgres does not cut it.

    You need some more CPU and RAM than what you pay for in your postgres instance. I.e. a distributed engine where you don't have to worry about how big your database instance is today.

    by fifilura
  • With S3 Table Buckets, Cloudflare R2 Data Catalog and now this, Iceberg seems to be winning.
    by ayhanfuat
  • I’m not a data engineer but work in an adjacent role. Is there anyone here who could dumb the use case down? Maybe an example of a problem this solves. I am struggling to understand the value proposition here.
    by NeutralCrane
  • Man, we are living in the golden era of PostgreSQL.
    by gajus
  • When Snowflake bought Crunchy Data I was hoping they were going to offer a managed version of this

    It's great that I can run this locally in a Docker container, I'd love to be able to run a managed instance on AWS billed through our existing Snowflake account

    by anentropic
  • This is huge!

    When people ask me what’s missing in the Postgres market, I used to tell them “open source Snowflake.”

    Crunchy’s Postgres extension is by far the most ahead solution in the market.

    Huge congrats to Snowflake and the Crunchy team on open sourcing this.

    by ozgune
  • Why not just use Ducklake?[1] That reduces complexity[2] since only DuckDB and PostgreSQL with pg_duckdb are required.

    [1] https://ducklake.select/

    [2] DuckLake - The SQL-Powered Lakehouse Format for the Rest of Us by Prof. Hannes Mühleisen: https://www.youtube.com/watch?v=YQEUkFWa69o

    by boshomi

Explore Birbla archives