Show HN: Telemetry.sh – Simplifying Telemetry Measurement

44 points by thebuilderjr 11 months ago | 28 comments
  • thebuilderjr 11 months ago
    Hi HN,

    I’m excited to introduce telemetry.sh, a service designed to streamline telemetry measurement in your applications. Our API/SDKs accept any JSON object, making it easy to get started with just a few lines of code.

    Under the hood, telemetry.sh converts JSON into Parquet schemas, stores data in S3, caches data locally based on tenant locality, and updates metadata stores for fast querying.

    Once your data is in our system, you can leverage our AI-augmented UI to quickly write SQL queries in plain English. We also offer robust visualization tools, including line charts, bar charts, and scatter plots, to help you easily plot and understand your data.

    We’ve prioritized team collaboration by integrating teams and memberships as core features, making it simple to work with your co-workers on telemetry coverage and analysis.

    Additionally, our query API allows you to integrate your telemetry data into your own products or internal dashboards seamlessly.

    This project has been a labor of love over the past year, and I’m thrilled to share it with the HN community. I look forward to your feedback.

    Thanks,

    JR

    • artisin 11 months ago
      I went down the telemetry rabbit hole a while back because I couldn't find an analytics/event log that met my needs. Nothing fancy—just a basic API to log certain events with some pretty graphs to display the data. I ended up building my own solution; however, telemetry.sh seems like it could have met my needs, but information on the site is sparse. Is this an interface for DataFusion, something akin to InfluxDB? It looks like a nice project, but some additional documentation (like you've detailed here) and use case examples on the site could go a long way.
      • thebuilderjr 11 months ago
        We provide an abstraction layer on top of DataFusion, allowing users to focus solely on logging JSON and writing SQL.

        Let’s go through an example for clarity. Imagine you’re developing the first version of Uber. Without funds to invest in setting up ClickHouse, Kafka, and other infrastructure, you still need a service to log and query data to calculate the average price of rides by city.

        With our telemetry service, you can achieve this with just a few lines of code. When finalizing a ride, you'd log the data as follows:

        ```javascript

        telemetry.log("uber_rides", { city: "Paris", price: 42, timestamp: new Date().toISOString() });

        ```

        For your internal dashboards, you can query this data with a single line of code:

        ```javascript

        const results = await telemetry.query(` SELECT city, AVG(price) FROM uber_rides GROUP BY city `);

        ```

        Behind the scenes, we utilize DataFusion, S3, and other technologies, but these complexities are abstracted away. As a user, you can concentrate on what data to log and how to query it.

        • squishy47 11 months ago
          I my opinion this example should be on the website to support your code snippets. the top of the page should tell me what problem your solving for me rather than what features the product has. this post does it perfectly but without reading through this thread i'd still be struggling to understand why i should use the product.

          I hope that didn't come across mean. the product looks really cool. congrats on the launch :)

        • irisguy 11 months ago
          Will you open source your solution? Curious to see it
          • thebuilderjr 11 months ago
            Originally, open source wasn't in our plan, but I'm now recognizing its benefits and am seriously considering it. Observing companies like Dub and Typesense use OSS to foster community contributions, goodwill, and trust has evidently helped them grow more rapidly and gain enterprise clients who prefer transparent solutions. While there's still some work needed before we can open source our project, it's likely to happen in the next few months. I'll update here once it's live, so you can take a look!
      • anon91 10 months ago
        Super interesting! Congrats on the launch.

        The use of parquet files and S3 as the main data storage seems interesting, but can you elaborate on how telemetry is optimized for frequent data inserts, since S3 only allows replacing a full file and data cannot be appended directly to it?

        Does it mean you use some sort of partitions with many files under the hood or a local buffer (memory/file/db) to make batch updates every x seconds on S3 for better performance?

        Would be happy to give it a try!

        • autocole 11 months ago
          I love the simplicity, but I would be hesitant to try it since I don’t know what 10MB of logs looks like. I would be interested in trying this if it could be set to auto-evicting past 10MB so I could start to understand that boundary better.
          • thebuilderjr 11 months ago
            Thank you for the compliment! I also understand your concern. I'll be adding a FAQ section to the pricing page soon.

            For your reference, the average size of each row is about 66 bytes when compressed. This means that 10MB should accommodate approximately 150,000 rows if your data is of average size.

            If pricing is a significant issue due to your current financial situation, please feel free to email me (you can find my contact in my profile). I'd be happy to offer a discount in exchange for your honest feedback.

          • bcjordan 11 months ago
            This is a really nice dev-friendly API, could see it becoming a compelling go-to for eg indie hackers who have moved simple view analytics to reasonably priced tools like Plausible/SimpleAnalytics over the years.

            In some ways it feels like Loggly, worth taking a look at their query/dashboard system as it has some really nice tools for clicking around to slice and dice graphs using unstructured data.

            Will have to give it a try some time!

            • thebuilderjr 11 months ago
              Thank you! Please don't hesitate to reach out if you have any questions. My contact info is on my profile.
            • msnkarthik 11 months ago
              Awesome product. congrats. I'm curious, can you share more about the security measures in place to protect the telemetry data stored in S3?
              • thebuilderjr 11 months ago
                Thank you! All data is securely stored in R2, Cloudflare's S3-compatible storage solution. We use TLS for encryption during transit and AES256 for encryption at rest. For enterprise customers, we also support the option to use your own S3-compatible storage, providing additional security measures such as granular access control.
            • halfdan 11 months ago
              I'm not quite sure how to place this. For general app telemetry there's https://opentelemetry.io and for tracking user behavior similar to how it's explained in your docs it looks like https://segment.io would be the closest competitor.

              How is telemetry.sh different from the above-mentioned?

              • thebuilderjr 11 months ago
                The biggest advantage is simplicity. To log an object in your application with OpenTelemetry, you either need munge your data to adapt it to their spec or rewrite a lot of your application code using their span abstraction. In contrast, with Telemetry, you can ingest any object you already have in your code as JSON with a single line of code and start analyzing the data immediately. Moreover, our SQL API makes it easy to integrate this data into your product or internal dashboards, unlike tools like Mixpanel or Amplitude that offer similar ingest APIs but lack APIs for SQL querying.
                • irisguy 11 months ago
                  How's it compare to Vercel's solution?: https://vercel.com/docs/analytics/custom-events
                  • thebuilderjr 11 months ago
                    Vercel's custom events offer similar ingestion APIs, but their analysis capabilities are limited compared to Telemetry. With Vercel, you are confined to the predefined reports and graphs they provide. I firmly believe in the flexibility and extensibility of data, and there's no better way to achieve this than with a comprehensive SQL API, which Telemetry offers.
                • John23832 11 months ago
                  OpenTelemetry is anything but simple.
                  • artisin 11 months ago
                    If anything, that's an understatement.
                • QuiCasseRien 11 months ago
                  Screenshot !

                  i'm lazy and i don't even test or register/login without screenshot.

                  • thebuilderjr 11 months ago
                    I thought videos would be more effective since they show the product in action. Is there something specific you were looking for that wasn't covered in the videos?
                  • throwaway63820 11 months ago
                    big fan of the approach & simplicity!