Birbla

Join the discussion

Write your take first — we'll ask for email only when you're ready to publish.

Sign in
  • Hacker News
  • There aren't any actual videos in the dataset though.
    by pr337h4m
  • The data is listed near the bottom, under "The 24 Endpoints" https://tiktok-api.seeksocial.io/#api
    by smallerize
  • The first immediate smell is that if you have 4.5B rows and 289GB in data, you have ~60 bytes per row.
    by 405error
  • This is being posted all over the place. on multiple subreddits and stuff. Why?
    by ckugblenu
  • Ads for an exploit
    by vachina
  • Well, most engineers are familiar with GIGO. Garbage in... Err, thats where my comment stops. Garbage in. Yup.
    by bartleeanderson
  • The AI keeps mentioning how a HTTP 200 can silently pollute your dataset. Why not just check contents of body? Usually APIs follow strict JSON contract for successful queries, alert or throw an error when that changes.
    by vachina
  • It's probably AI coded and hallucinated many things. That drumming up the importance of a minor thing is a real tell. Another hallucination - it hasn't found any video APIs (despite statements that it has and uploaded it). It has video metadata.
    by 405error
  • Very impressive figuring out all 4 checks. I'm not even sure this was reverse engineered, rather than leaked. I wish it mentioned anything about the methodology they used
    by negura
  • >Is it legal? It is against TikTok's terms of service. It is sold for research and educational use.

    Oh, ok. Otherwise, very detailed deconstruction to scrape their API. Lots of layers of registration and creating a request that looks like it is valid client.

    by magicmicah85
  • There's no way this dataset is going to survive on HF, right? It will be hit with so many DMCA takedowns.
    by smallerize
  • My thoughts also
    by deviation
  • It's metadata only, not video content. I expect it will likely survive - that's a pretty common pattern for machine learning datasets.

    Stable Diffusion was enabled by LAION, for example. That was metadata about images and URLs to those images, but not the actual image files.

    by simonw
  • 1787990085 | X and Meta's past data scraping lawsuit losses and how it would relate to nitter | https://www.reuters.com/legal/musks-x-corp-loses-lawsuit-aga... | https://news.ycombinator.com/item?id=49487864

    Perhaps Google can succeed where others like Meta and X have failed, or perhaps not

    https://storage.courtlistener.com/recap/gov.uscourts.cand.46...

    by 1vuio0pswjnm7
  • > Everything described here is a private Go repository. One-time payment, permanent access, complete source. > $699 one time · lifetime access

    Not open-source apparently.

    And I cant find the reddit post but I think I read that videos/assets are not actually pre-downloaded, they have to be requested through Tiktok API using the provided code. So if Tiktok patches, the code will need updates too.

    by moinism
  • I think the post is essentially a decent technical-explainer (value adding and interesting) in exchange for effectively a little product placement (selling either just the code or code already running on a server at additional cost)

    But I think this is the 289GB data (free): https://huggingface.co/datasets/kuben-developer/tiktok-video...

    by nomilk
  • Can any brave soul wade through the LLM prose to provide a human-readable summary?
    by Retr0id
  • Man, I could use $699. If this gets even a single sale, then maybe I need to try having less decency...

    (I guess that is sort of a roundabout summary...)

    by arcfour
  • I cannot verify whether it is technically correct, but it's about how to defeat Tiktok's bot filters to scrape it.
    by 405error
  • "I uploaded 4.5 billion of those videos to Hugging Face" = I scraped the metadata (title, view count, etc) for 4.5 billion TikTok videos using the same API as their Android app. I posted the data to HF as Parquet. The video content itself is not included. I'll sell you my Go scraping code.
    by simonw
  • > Three things to notice, because each one bites later:

    Very LLMish language!

    by nomilk
  • It's that mix of dense, impressive sounding jargon, but even scanning across it raises glaring problems. Like, if you have 4.5 billion videos on HF, and it's 289GB, it's about 60 bytes per video. Checking the column fields as well, there doesn't seem to be any video files*.
    by 405error