More Memory Safety for Let's Encrypt: Deploying ntpd-rs

348 points by Dunedan 1 year ago | 163 comments
  • NelsonMinar 1 year ago
    I like the idea of NTPD in Rust. Is there anything to read about how well ntpd-rs performs? Would love a new column for chrony's comparison: https://chrony-project.org/comparison.html

    Particularly interested in the performance stats, how well the daemon keeps time in the face of various network problems. Chrony is very good at this. Some of the other NTP implementations (not on that chart) are so bad they shouldn't be used in production.

    • rnijveld 1 year ago
      In our internal testing we are very close to Chrony with our synchronization performance, some of our testing data and an explanation of our algorithm is published in our repository: https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a...

      Given the amount of testing we (and other parties) have done, and given the strong theoretical foundation of our algorithm I’m pretty confident we’d do well in many production environments. If you do find any performance issues though, we’d love to hear about them!

      • hcfman 1 year ago
        Ah okay. Nice. Love to know whether you have plans to also provide gps time synchronisation? I guess if the point was to deal with security issues surrounding network communications I’d guess the answer is no.

        But if the goals was also be a dominant player in time synchronisation then it might be a nice to have.

        BTW. Letsencrypt certificates are the best. I install them with pretty much every installation of my other software. Thanks guys.

    • ComputerGuru 1 year ago
      Unlike say, coreutils, ntp is something very far from being a solved problem and the memory safety of the solution is unfortunately going to play second fiddle to its efficacy.

      For example, we only use chrony because it’s so much better than whatever came with your system (especially on virtual machines). ntpd-rs would have to come at least within spitting distance of chrony’s time keeping abilities to even be up for consideration.

      (And I say this as a massive rust aficionado using it for both work and pleasure.)

      • hi-v-rocknroll 1 year ago
        You might be doing too much work at the wrong level of abstraction. VMs should use host clock synchronization. It requires some work and coordination, but it eliminates the need for ntp in VMs entirely.

        Hosts should then be synced using PTP or a proper NTP local stratum (just get a proper GNSS source for each DC if you have then funds).

        https://tsn.readthedocs.io/timesync.html

        Deploy chrony to bare metal servers wherever possible.

        • rnijveld 1 year ago
          Our project also includes a PTP implementation, statime (https://github.com/pendulum-project/statime/), that includes a Linux daemon. Our implementation should work as well or even better than what linuxptp does, but it's still early days. One thing to note though is that NTP can be made to be just as precise (if not more precise), given the right access to hardware (unfortunately most hardware that does timestamping only does so for PTP packets). The reason for this precision is simple: NTP can use multiple sources of time, whereas PTP by design only uses a single source. This gives NTP more information about the current time and thus allows it to more precisely estimate what the current time is. The thing with relying purely on GNSS is that those signals can be (and are in practice) disrupted relatively easily. This is why time synchronization over the internet makes sense, even for large data centers. And doing secure time synchronization over the internet is only practically possible using NTP/NTS at this time. But there is no one size fits all solution for time synchonization in general.
          • ComputerGuru 1 year ago
            We've had issues relying on ESXi host/guest time synchronization, depending on the guest OS, and found this to be a better solution.
            • xorcist 1 year ago
              This makes sense. The clock is just another piece of hardware to be virtualized and shared among the guests.

              But last time I said that with some pretense of authority, someone shoved me a whitepaper from VMware that said the opposite. Best practice was stated be to sync each guest individually with a completely virtual clock.

              I'm not sure I agree, but at least I try to be open to be possibility that there are situations I had not considered. If anyone else knows more about this, please share.

              • yjftsjthsd-h 1 year ago
                > a whitepaper from VMware that said the opposite

                Did it say why?

              • hcfman 1 year ago
                And by funds. That’s typically less than 100 euros with a Raspberry Pi
              • syncsynchalt 1 year ago
                The biggest danger in NTP isn't memory safety (though good on this project for tackling it), it's

                (a) the inherent risks in implementing a protocol based on trivially spoofable UDP that can be used to do amplification and reflection

                and

                (b) emergent resonant behavior from your implementation that will inadvertently DDOS critical infrastructure when all 100m installed copies of your daemon decide to send a packet to NIST in the same microsecond.

                I'm happy to see more ntpd implementations but always a little worried.

                • rnijveld 1 year ago
                  I agree that amplification and reflection definitely are worries, which is why we are working towards NTS becoming a default on the internet. NTS would prevent responses by a server from a spoofed packet and at the same time would make sure that NTP clients can finally start trusting their time instead of hoping that there are no malicious actors anywhere near them. You can read about it on our blog as well: https://tweedegolf.nl/en/blog/122/a-safe-internet-requires-s...

                  One thing to note about amplification: amplification has always been something that NTP developers have been especially sensitive to. I would say though that protocols like QUIC and DNS have far greater amplification risks. Meanwhile, our server implementation forces that responses can never be bigger than the requests that initiated them, meaning that no amplification is possible at all. Even if we would have allowed bigger responses, I cannot imagine NTP responses being much bigger than two or three times their related request. Meanwhile I've seen numbers for DNS all the way up to 180 times the request payload.

                  As for your worries: I think being a little cautious keeps you alert and can prevent mistakes, but I also feel that we've gone out of our way to not do anything crazy and hopefully we will be a net positive in the end. I hope you do give us a try and let us know if you find anything suspicious. If you have any feedback we'd love to hear it!

                  • dfc 1 year ago
                    > I cannot imagine NTP responses being much bigger than two or three times their related request.

                    I think you must be limiting your imagination to ntp requests related to setting the time. There are a lot of other commands in the protocol used for management and metrics. The `monlist` command was good for 200x amplification. https://blog.cloudflare.com/understanding-and-mitigating-ntp...

                    • syncsynchalt 1 year ago
                      Thank you for your considered response!

                      I hadn't heard about NTS and I'm rolling it out to my fleet of timeservers now.

                    • timmytokyo 1 year ago
                      I really wish more internet infrastructure would switch to using NTS. It addresses these kinds of issues.
                      • jaas 1 year ago
                        ntpd-rs support NTS, I agree it would be great if more people used it!
                        • 1over137 1 year ago
                          Never heard of it. Shockingly little on wikipedia for example.
                      • rnijveld 1 year ago
                        I would encourage you to take a look at some of our testing data and an explanation of our algorithm in our repository (https://github.com/pendulum-project/ntpd-rs/tree/main/docs/a...). I think we are very much in spitting distance of Chrony in terms of synchronization performance, sometimes even beating Chrony. But we’d love for more people to try our algorithm in their infrastructure and report back. The more data the better.
                        • agwa 1 year ago
                          What exactly does "time keeping abilities" mean? If I had to choose between 1) an NTP implementation with sub-millisecond accuracy that might allow a remote attacker to execute arbitrary code on my server and 2) an NTP implementation which may be ~100ms off but isn't going to get me pwned, I'm inclined to pick option 2. Is writing an NTP server that maintains ~100ms accuracy not a solved problem?
                        • cogman10 1 year ago
                          This seems like a weird place to be touting memory safety.

                          It's ntpd, it doesn't seem like a place for any sort of attack vector and it's been running on many VMs without exploding memory for a while now.

                          I'd think there are far more critical components to rewrite in a memory safe language than the clock synchronizer.

                          • jaas 1 year ago
                            I'm the person driving this.

                            NTP is worth moving to a memory safe language but of course it's not the single most critical thing in our entire stack to make memory safe. I don't think anyone is claiming that. It's simply the first component that got to production status, a good place to start.

                            NTP is a component worth moving to a memory safe language because it's a widely used critical service on a network boundary. A quick Google for NTP vulnerabilities will show you that there are plenty of memory safety vulnerabilities lurking in C NTP implementations:

                            https://www.cvedetails.com/vulnerability-list/vendor_id-2153...

                            Some of these are severe, some aren't. It's only a matter of time though until another severe one pops up.

                            I don't think any critical service on a network boundary should be written in C/C++, we know too much at this point to think that's a good idea. It will take a while to change that across the board though.

                            If I had to pick the most important thing in the context of Let's Encrypt to move to a memory safe language it would be DNS. We have been investing heavily in Hickory DNS but it's not ready for production at Let's Encrypt yet (our usage of DNS is a bit more complex than the average use case).

                            https://github.com/hickory-dns/hickory-dns

                            Work is proceeding at a rapid pace and I expect Hickory DNS to be deployed at Let's Encrypt in 2025.

                            • twothreeone 1 year ago
                              It continues to astonish me how little people care (i.e., it triggers the $%&@ out of me). I really appreciate the professionalism and cool rationale when faced with absolute ignorance of how shaky a foundation our "modern" software stack is built upon. This is a huge service to the community, kudos to you and many others slowly grinding out progress!
                              • vmfunction 1 year ago
                                Lol, shaky indeed. A business person once said, "can you imagine if machine engineer (like auto makers) behave like software engineering?".

                                Seems no digital system is truly secure. Moving foundational code to memory safe seems like a good first step.

                              • landmarker10 1 year ago
                                [flagged]
                                • jaas 1 year ago
                                  People bring up postfix all the time in this context because supposedly nobody has ever found a memory safety vulnerability in it. Presumably this is supposed to make the point that it is possible to write complex programs in C safely.

                                  The reason people know this about postfix and keep bringing this one specific example up is because it's so unusual! It's an example that almost stands alone, it's extraordinary.

                                  I don't think this is making the point about the safety of C that you think it is.

                                  There is a mountain of evidence suggesting that C is dangerous, particularly for network services, and one possible example to the contrary doesn't change that.

                                  • kaql 1 year ago
                                    [flagged]
                                  • dheera 1 year ago
                                    Why are C and C++ all of a sudden unsafe? Did I miss something?

                                    What is safe now? JavaScript? PyTorch?

                                    • ghshephard 1 year ago
                                      One of the major drivers (if not the driver) for the creation of Rust the fact that C is not a memory-safe language.

                                      This has been known for decades, but it wasn't until 2010 that a serious attempt at writing a new system-language that was memory safe was attempted and got traction - Rust.

                                      https://kruschecompany.com/rust-language-concise-overview/#:....

                                      • pjmlp 1 year ago
                                        They have been unsafe from their very early days, Multics got a higher security score than UNIX thanks to PL/I, and C.A.R Hoare has addressed C's existence on his Turing Award in 1980, while Fran Allen has also made similar remarks back in the day.
                                        • dequan 1 year ago
                                          All of a sudden? They've been unsafe for decades, it's just that you had less of a choice then.
                                          • alkonaut 1 year ago
                                            Because you can cast a pointer to a number and back again. Then you can stuff that value into index [-4] of your array. More or less.
                                            • runiq 1 year ago
                                              Tcl
                                              • t3rra 1 year ago
                                                [flagged]
                                            • oconnor663 1 year ago
                                              > it's been running on many VMs without exploding memory for a while now

                                              Most of the security bugs we hear about don't cause random crashes on otherwise healthy machines, because that tends to get them noticed and fixed. It's the ones that require complicated steps to trigger that are really scary. When I look at NTP, I see a service that:

                                              - runs as root

                                              - talks to the network

                                              - doesn't usually authenticate its traffic

                                              - uses a bespoke binary packet format

                                              - almost all network security depends on (for checking cert expiration)

                                              That looks to me like an excellent candidate for a memory-safe reimplementation.

                                              • cogman10 1 year ago
                                                > runs as root

                                                ntpd can (and should) run as a user

                                                > talks to the network

                                                Makes outbound requests to the network. For it to be compromised, the network itself or a downstream server needs to be compromised. That's very different from something like hosting an http server.

                                                > doesn't usually authenticate its traffic

                                                Yes it does. ntp uses TLS to communicate with it's well known locations.

                                                > uses a bespoke binary packet format

                                                Not great but also see above where it's talking to well known locations authenticated and running as a user.

                                                It's a service that to be compromised requires state level interference.

                                                • woodruffw 1 year ago
                                                  I don’t think NTP uses TLS by default. The closest equivalent I can find online is NTS, which was only standardized in 2020 (and for which I can’t find clear adoption numbers).

                                                  (But regardless: “the transport layer is secure” is not a good reason to leave memory unsafe code on the network boundary. Conventional wisdom in these settings is to do the “defense in depth” thing and assume that transport security can be circumvented.)

                                                  • kortilla 1 year ago
                                                    Ntp does not use TLS widely. It’s also a UDP protocol which makes it subject to spoofing attacks without a compromised network (no protection from kernel network stack verifying TCP sequence numbers).
                                                    • franga2000 1 year ago
                                                      > It's a service that to be compromised requires state level interference

                                                      For servers it's definitely harder, although definitely not only state-level, but another big issue could be client-side. NTP servers can be set by DHCP, so the admin of any network you connect to could exploit such a bug against you. And once you have code execution on a desktop OS, all bets are off, even if you're not under the primary UID.

                                                      It's not the most important threat vector, but it also doesn't seem as difficult as some of the other system services to rewrite, so I'd say it was a good first step for the memory-safe-everything project.

                                                      • adolph 1 year ago
                                                        >> doesn't usually authenticate its traffic

                                                        > Yes it does. ntp uses TLS to communicate with it's well known locations.

                                                        My knee-jerk reaction is that TLS is not authentication. After skimming through the relevant spec [0], it is interesting how NTP uses TLS.

                                                        [NTS-KE] uses TLS to establish keys, to provide the client with an initial supply of cookies, and to negotiate some additional protocol options. After this, the TLS channel is closed with no per-client state remaining on the server side. [0]

                                                        0. https://datatracker.ietf.org/doc/html/rfc8915

                                                        • wbl 1 year ago
                                                          The network is just a bunch of strangers in a meet me room.
                                                      • luma 1 year ago
                                                        It's present on loads of systems, it's a very common service to offer, it's a reasonably well-constrained use case, and the fact that nobody thinks about it might be a good reason to think about it. They can't boil the ocean but one service at a time is a reasonable approach.

                                                        I'll flip the question around, why not start at ntpd?

                                                        • cogman10 1 year ago
                                                          > I'll flip the question around, why not start at ntpd?

                                                          Easy, because there are loads of critical infrastructure written in C++ that is commonly executed on pretty much every VM and exposed in such a way that vulnerabilities are disasterous.

                                                          For example, JEMalloc is used by nearly every app compiled in *nix.

                                                          Perhaps systemd which is just about everywhere running everything.

                                                          Maybe sshd, heaven knows it's been the root of many attacks.

                                                          • kelnos 1 year ago
                                                            > For example, JEMalloc is used by nearly every app compiled in nix.*

                                                            JEmalloc is used by very very very few apps compiled for *nix. That's a conscious decision that an app developer needs to make (and few would bother without specialized needs) or a distro packager needs to shoehorn into their package builds (which most/all would not do).

                                                            • hedora 1 year ago
                                                              There’s a nice pure-rust ssh client/server already.

                                                              Systemd should just be scrapped. This week’s wtf “systemd-tmpfile —-purge” intentionally changed its behavior to “rm -rf /home”. Confirmed not-a-bug. There are dozens of other comparable screwups in that stack, even ignoring the long list of CVEs (including dns and ntp, I think). Rust can’t fix that.

                                                              I haven’t heard of any issues with jemalloc, though that seems reasonable (assuming calling rust from C doesn’t break compiler inlining, etc).

                                                          • lambdaone 1 year ago
                                                            NTP is a ubiquitous network service that runs directly exposed to the Internet, and that seems to me like a good thing to harden. Making NTP more secure does not stop anyone else from working on any other project.
                                                            • rnijveld 1 year ago
                                                              I do think that memory safety is important for any network service. The probability of something going horribly wrong when a network packet is parsed in a wrong way is just too high. NTP typically does have more access to the host OS than other daemons, with it needing to adjust the system clock.

                                                              Of course, there are many other services that could be made memory safe, and maybe there is some sort of right or smart order in which we should make our core network infrastructure memory safe. But everyone has their own priorities here, and I feel like this could end up being an endless debate of whatabout-ism. There is no right place to start, other than to just start.

                                                              Aside from memory safety though, I feel like our implementation has a strong focus on security in general. We try and make choices that make our implementation more robust than what was out there previously. Aside from that, I think the NTP space has had an under supply of implementations, with there only being a few major open source implementations (like ntpd, ntpsec and chrony). Meanwhile, NTP is one of those pieces of technology at the core of many of the things we do on the modern internet. Knowing the current time is one of these things you just need in order to trust many of the things we take for granted (without knowledge of the current time, your TLS connection could never be trusted). I think NTP definitely deserves this attention and could use a bunch more attention.

                                                              • astrobe_ 1 year ago
                                                                Agreed. It is a "good old" binary protocol, so the many gotchas of text protocols are not there.
                                                              • mre 1 year ago
                                                                I spoke with Folkert, one of the developers on this project, on the 'Rust in Production' podcast. Some of you might find it interesting: https://corrode.dev/podcast/s01e05-tweede-golf/
                                                                • akira2501 1 year ago
                                                                  Why does your ntpd have a json dependency?
                                                                  • danudey 1 year ago
                                                                    This is a good question to ask, especially in the age of everything pulling in every possible dependency just to get one library function or an `isNumeric()` convenience function.

                                                                    The answer is that there is observability functionality which provides its results as JSON output via a UNIX socket[0]. As far as I can see, there's no other JSON functionality anywhere else in the code, so this is just to allow for easily querying (and parsing) the daemon's internal state.

                                                                    (I'm not convinced that JSON is the way to go here, but that's the answer to the question)

                                                                    [0] https://docs.ntpd-rs.pendulum-project.org/development/code-s...

                                                                    • motrm 1 year ago
                                                                      If the pieces of state are all well known at build time - and trusted in terms of their content - it may be feasible to print out JSON 'manually' as it were, instead of needing to use a JSON library,

                                                                        print "{"
                                                                        print "\"some_state\": \"";
                                                                        print GlobalState.Something.to_text();
                                                                        print "\", ";
                                                                        print "\"count_of_frobs\": ";
                                                                        print GlobalState.FrobsCounter;
                                                                        print "}";
                                                                      
                                                                      Whether it's worth doing this just to rid yourself of a dependency... who knows.
                                                                      • Gigachad 1 year ago
                                                                        This looks like the exact kind of thing that results in unexpected exploits.
                                                                        • syncsynchalt 1 year ago
                                                                          Even better to just use TSV. Hand-rolling XML or JSON is always a smell to me, even if it's visibly safe.
                                                                          • fiedzia 1 year ago
                                                                            > If the pieces of state are all well known at build time - and trusted in terms of their content

                                                                            .. than use library, because you should not rely on the assumption that next developer adding one more piece to this code will magically remember to validate it with json spec.

                                                                            • mananaysiempre 1 year ago
                                                                              That’s somewhat better than assembling, say, HTML or SQL out of text fragments, but it’s still not fantastic. A JSON output DSL would be better still—it wouldn’t have to be particularly complicated. (Shame those usually only come paired with parsers, libxo excepted.)
                                                                            • 1 year ago
                                                                            • rnijveld 1 year ago
                                                                              I don’t think our dependency tree is perfect, but I think our dependencies are reasonable overall. We use JSON for transferring metrics data from our NTP daemon to our prometheus metrics daemon. We’ve made this split for security reasons, why have all the attack surface of a HTTP server in your NTP daemon? That didn’t make sense to us. Which is why we added a readonly unix socket to our NTP daemon that on connecting dumps a JSON blob and then closes the connection (i.e. doing as little as possible), which is then usable by our client tool and by our prometheus metrics daemon. That data transfer uses json, but could have used any data format. We’d be happy to accept pull requests to replace this data format with something else, but given budget and time constraints, I think what we came up with is pretty reasonable.
                                                                              • stavros 1 year ago
                                                                                If you're only dumping a string, couldn't you replace this dependency with some string concatenation?
                                                                                • rnijveld 1 year ago
                                                                                  Probably, but we still need to parse that string on the client side as well. If you’re willing to do the work I’m sure we would accept a pull request for it! There’s just so many things to do in so little time unfortunately. I think reducing our dependencies is a good thing, but our dependencies for JSON parsing/writing are used so commonly in Rust and the way we use it hopefully prevents any major security issues that I don’t think this should be a high priority for us right now compared to the many things we could be doing.
                                                                              • orf 1 year ago
                                                                                Would you rather it had a JSON dependency to parse a config file, or yet another poorly thought out, ad-hoc homegrown config file format?
                                                                                • cozzyd 1 year ago
                                                                                  JSON is a terrible configuration format since it doesn't support comments.
                                                                                  • orf 1 year ago
                                                                                    Ok. Do you want to now add anything relevant to the comment you’re replying to?
                                                                                  • akira2501 1 year ago
                                                                                    > yet another poorly thought out, ad-hoc homegrown config file format

                                                                                    OpenBSD style ntpd.conf:

                                                                                        servers 0.gentoo.pool.ntp.org
                                                                                        servers 1.gentoo.pool.ntp.org
                                                                                        servers 2.gentoo.pool.ntp.org
                                                                                        servers 3.gentoo.pool.ntp.org
                                                                                    
                                                                                        constraints from "https://www.google.com"
                                                                                    
                                                                                        listen on *
                                                                                    
                                                                                    I mean, there's always the possibility that they used a common, well known and pretty decent config file format. In this particular case, this shouldn't be the thing that differentiates your ntpd implementation anyways.
                                                                                    • IshKebab 1 year ago
                                                                                      That config file perfectly illustrates the point. There's no need for it to be custom, and require me to waste time learning its syntax when it could just be JSON or TOML. Honestly I would even take YAML over that and YAML is the worst.
                                                                                    • itishappy 1 year ago
                                                                                      It uses TOML for configuration.
                                                                                      • orf 1 year ago
                                                                                        Cool, thats why this is a hypothetical question
                                                                                      • amiga386 1 year ago
                                                                                        Poorly thought out, ad-hoc homegrown config file format, please. Every time.

                                                                                        1. Code doesn't change at the whims of others.

                                                                                        2. The entire parser for an INI-style config can be in about 20 lines of C

                                                                                        3. Attacker doesn't also get to exploit code you've never read in the third party dependency (and its dependencies! The JSON dependency now wants to pull in the ICU library... I guess you're linking to that, too)

                                                                                        4. Complexity of config file formats are usually format-independent, the feature-set of the format itself only adds complexity, rather than takes it away. To put it another way, is this any saner...

                                                                                            {"user":"ams","host":"ALL","runas":["/bin/ls","/bin/df -h /","/bin/date \"\"","/usr/bin/","sudoedit /etc/hosts","OTHER_COMMANDS"}
                                                                                        
                                                                                        ... than ...

                                                                                            # I may be crazy mad but at least I can have comments!
                                                                                            ams ALL=/bin/ls, /bin/df -h /, /bin/date "", /usr/bin/, sudoedit /etc/hosts, OTHER_COMMANDS
                                                                                        
                                                                                        
                                                                                        All the magic in the example is in what those values are and what they imply, the format doesn't improve if you naively transpose it to JSON.

                                                                                        An example of an NTP server's config:

                                                                                            # I can have comments too
                                                                                            [Time]
                                                                                            NTP=ntp.ubuntu.com
                                                                                            RootDistanceMaxSec=5
                                                                                            PollIntervalMinSec=32
                                                                                            PollIntervalMaxSec=2048
                                                                                        
                                                                                        
                                                                                        If you just want key-value pairs of strings/ints, nothing more complex is needed. Using JSON is overdoing it.
                                                                                        • vlakreeh 1 year ago
                                                                                          1. I can pin my json parser dependency and literally never update it again

                                                                                          2. And how many times have we seen 20 lines of C backfire with some sort of memory safety issue.

                                                                                          3. First off, i'd go out on a limb and say the number of attacks from a well-established (or even a naive one) rust json parsing library is dwarfed by the number of attacks from adhoc config parsers written in C with some overlooked memory safety issue.

                                                                                          4. Usually being the key word, tons of adhoc config formats have weird shit in them. With json (or yaml/toml) you know what you're getting into and you immediately know what you're able and unable to do.

                                                                                          • kelnos 1 year ago
                                                                                            I feel like using the incomprehensibly error-prone and inscrutable sudoers format as an example kinda argues against your point.

                                                                                            (I do agree that JSON is a terrible configuration file format, though.)

                                                                                            • whytevuhuni 1 year ago
                                                                                              I once saw an .ini for a log parser:

                                                                                                  [Alarm]
                                                                                                  Name=Nginx Errors
                                                                                                  Pattern="[error] <pid>#<tid>: <message>"
                                                                                              
                                                                                              The thing worked. Without any errors. And yet it took:

                                                                                                  Pattern="[error] <pid>
                                                                                              
                                                                                              ..and then considered the rest of the line a comment. It didn't even error on the fact that the quotes were not closed.

                                                                                              Hand-rolling config formats is hard.

                                                                                            • patmorgan23 1 year ago
                                                                                              Why isn't there a decent parser in the standard library? More than 50% of programs will probably touch json at this point.
                                                                                              • tialaramex 1 year ago
                                                                                                Rust's stdlib is (at least notionally) forever. Things in the standard library (including core and alloc) get deprecated but must be maintained forever, which means that "at this point" isn't enough.

                                                                                                In 2003 those programs would have used XML, in 1993 probably .INI files. Are you sure that despite all its shortcomings JSON is the end of history? I don't believe you.

                                                                                                If you want "at this point" you can, as software does today, just use a crate. Unlike the stdlib, if next week Fonzie files are huge and by 2026 "nobody" is using JSON because Fonzie is cool, the JSON config crate merely becomes less popular.

                                                                                          • hcfman 1 year ago
                                                                                            If you want to setup a chrony time server that maintains accuracy to within a microsecond and doesn’t do this with a network connection then you could try my sbts-aru project and just not use the audio recorder parts of it.

                                                                                            https://github.com/hcfman/sbts-aru

                                                                                            It installs with a single command on all Raspberry Pi versions and takes care of all the dependencies, configuration and startup order details to install and start working with one command.

                                                                                            It’s a sound localizing audio recorder platform and that’s why it also sets up accurate time.

                                                                                            It’s using GPS to get its time from.

                                                                                            • _joel 1 year ago
                                                                                              Reading this reminded me of ntpsec, anyone actually use that?
                                                                                            • xvilka 1 year ago
                                                                                              BGP probably should be the next.
                                                                                              • nubinetwork 1 year ago
                                                                                                The problem with ntp isn't the client, it's the servers having to deal with forged UDP packets. Will ntpd ever become TCP-only? Sadly I'm not holding my breath. I stopped running a public stratum 3 server ~10 years ago.
                                                                                                • Faaak 1 year ago
                                                                                                  On the contrary, I'm hosting a stratum 1 and 2 stratum 2s (at my previous company we offered 3 stratum 1s) on the ntp pool. It's useful, used, and still needed :-)
                                                                                                  • brohee 1 year ago
                                                                                                    When one can make a stratum 1 server for $100, there is very little reason for the continuous existence of public NTP servers. ISP can offer the service to their customers, and any company with a semblance of IT dept can have its own stratum 1.
                                                                                                    • ssl-3 1 year ago
                                                                                                      One can build a GPS-backed stratum 1 server for a lot less than $100 in hardware (and I have done so in the past). It was a fun little project for me, but it involved a confluence of skillsets that many [especially smaller] companies may not collectively possess. And even then, it needs to be documented so someone else can work on it, and that maintenance has to actually-happen, and it needs a view of the sky in order for it to chooch, and it also needs redundancy. This takes time. (And if this IT department doesn't work for free, then the cost is very quickly a lot more than $100.)

                                                                                                      And going full-assed with one or more actually-outside antennas can also be problematic, since it's a Bad Day when (eg) grounding is done improperly and lightning comes by to fuck their shit up.

                                                                                                      And ISPs can (and certainly should!) provide decent NTP servers. That's the logical place for shared servers, network-wise. But the one's choice of ISP is often limited by geography, and the most-viable ISP here stopped publishing what their NTP servers are -- if they still exist in a form that customers can use, they don't publish this information. (It wasn't always this way and I remember using them years ago when they were more forthcoming. They were shitty NTP servers with high jitter. Good enough for some things, I suppose, but not as good as the members of the public NTP pool tend to be.)

                                                                                                      I mean: Many ISPs can't even manage to handle DNS queries quickly. When public servers like 8.8.8.8 and 1.1.1.1 (or the semi-public ones like 4.2.2.1) are faster than the ISP's own servers, then that's a problem. And it's a stupid problem, and it should not happen. But it does happen.

                                                                                                      So thus, public NTP servers are useful for many -- including those who have a tinkered-together NTP server with a GPS antenna in a window somewhere, where public servers can be used as backup.

                                                                                                      It's good to have options, and it's nice that some organizations provide some options for the greater network.

                                                                                                  • akaletF 1 year ago
                                                                                                    [flagged]
                                                                                                    • skilled 1 year ago
                                                                                                      [flagged]
                                                                                                      • mianosm 1 year ago
                                                                                                        The Jonestown massacre was actually grape flavor-aid:

                                                                                                        https://www.vox.com/2015/5/23/8647095/kool-aid-jonestown-fla...

                                                                                                        They really do appear to be all in on avoiding memory leaks from C/CPP:

                                                                                                        > Over the next few years we plan to continue replacing C or C++ software with memory safe alternatives in the Let’s Encrypt infrastructure: OpenSSL and its derivatives with Rustls, our DNS software with Hickory, Nginx with River, and sudo with sudo-rs. Memory safety is just part of the overall security equation, but it’s an important part and we’re glad to be able to make these improvements.

                                                                                                        It seems like a really challenging endeavor, but I appreciate their desire to maintain uptime and a public service like they do.

                                                                                                        • tialaramex 1 year ago
                                                                                                          Correctness matters, in their particular game that's especially true although I'm doubtful of common insistence that it's better for this or that software to be fast than correct.

                                                                                                          Rust is really good for correctness. Take "Hello, World", the obvious toy program. Someone tried giving it various error states instead of (as would be usual) a normal happy terminal environment. In C or C++ the canonical "Hello, World" program terminates successfully despite any amount of errors, it just doesn't care about correctness.

                                                                                                          The default Rust Hello World, the one you get out of the box when you make a new project, or you'd show people on a "My First Rust Program" course, will complain about the errors when they happen. Because doing so is correct.

                                                                                                          It's the New Jersey style. The priority for these languages was simplicity of implementation. It's more important that you can cobble together a C compiler easily than that the results are useful or worthwhile. This contributed to C's survival, but we pay the price until we give it up.

                                                                                                          • akaletF 1 year ago
                                                                                                            If println panics if the write fails that is kind of cheating, isn't it. Yes, toy C programs do not check the return value of printf. So what.
                                                                                                            • dralley 1 year ago
                                                                                                              I would venture to say that most C programs don't check the return value of printf, including ones that ought not to be "toys"
                                                                                                              • toast0 1 year ago
                                                                                                                Setting up your environment so that you have to check for errors or the errors propagate doesn't seem to be cheating when the complaint is that other environments often omit checking for errors even when it matters.
                                                                                                            • _flux 1 year ago
                                                                                                              Is it completely unwarranted, though? It seems most of the issues listed here are indeed memory safety bugs that are more difficult to pull off in memory-safe languages such as Rust: https://www.cvedetails.com/vulnerability-list/vendor_id-2153...
                                                                                                              • itishappy 1 year ago
                                                                                                                > I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner.

                                                                                                                TFA's second sentence explains the facts of the matter along with the flavor of Kool-Aid they stock:

                                                                                                                > The CA software itself is written in memory safe Golang, but from our server operating systems to our network equipment, lack of memory safety routinely leads to vulnerabilities that need patching.

                                                                                                                • kelnos 1 year ago
                                                                                                                  > I struggle to understand why they would say that as the opening statement in such a matter-of-fact manner.

                                                                                                                  Because it is a fact. An obvious, obvious fact to anyone who has been working in this ecosystem for any amount of time.

                                                                                                                  > Drinking the Rust kool-aid by the sound of it.

                                                                                                                  Would you say the same thing if they'd instead decided to use golang, zig, nim, ocaml, etc.? If not, maybe consider that your emotional stance around Rust is coloring your judgment.

                                                                                                                  Our modern systems are built on a house of cards where security is concerned. I agree that the "RiiR" meme is tiresome and dumb, but I'm tired of seeing report after report of new vulnerabilities found in foundational libraries and programs. The majority of those vulnerabilities are of the type that Rust won't even let you compile. Languages like C and C++ have their place, but for most applications, there are safer alternatives that don't require harsh compromises or significant trade offs.

                                                                                                                  • pjmlp 1 year ago
                                                                                                                    Because since the Morris Worm in 1988, there are still plenty of networking facing services that keep being written in C and C++, and without the necessary sanitary precautions.

                                                                                                                    True, there are plenty of alternatives for many of those networking services, not necessarily Rust.

                                                                                                                    • danudey 1 year ago
                                                                                                                      I hate writing code in golang, because I have to pepper every single function call with `if err != nil`, but then I think about how much C code I've seen that doesn't do that and I wonder how much of it should.
                                                                                                                      • hot_gril 1 year ago
                                                                                                                        Golang's error handling is safer than C's, but it's more cumbersome than it needs to be. In high-level code, nearly every func you write can return an error, and 99% of the time you're just going to pass the error up. Webserver will catch all and send 4xx or 5xx, for example. Exceptions are a lot more convenient and encourage solid error handling.

                                                                                                                        Rust chose a good in-between (the "?" unwrapping syntax). In mid or low level code, anything can still fail, but I suspect the performance impact of supporting exceptions or similar for basic operations (integer overflow, div by 0, etc) wouldn't be worthwhile vs just crashing the program or doing something else. Interestingly, Rust doesn't crash in this case: https://doc.rust-lang.org/book/ch03-02-data-types.html

                                                                                                                        • SAI_Peregrinus 1 year ago
                                                                                                                          Lots!

                                                                                                                          Also lots of C code that doesn't properly check `errno` for things like `strtol` to check for errors that aren't returned directly from a function.

                                                                                                                      • hot_gril 1 year ago
                                                                                                                        What's incorrect about that statement?
                                                                                                                        • bakugo 1 year ago
                                                                                                                          [flagged]
                                                                                                                        • johnklos 1 year ago
                                                                                                                          [flagged]
                                                                                                                          • dequan 1 year ago
                                                                                                                            I agree that it would be great if the ecosystem was a bit slower to use every new version and it does seem like things are beginning to tend in that direction as many foundational crates have begun declaring MSRVs of !LATEST.

                                                                                                                            However I don't think the pace of updates really changes anything in terms of tool chain security. If Rust decided to go to a 36 week release cycle, each release would just have 6x as much stuff in it. If you can't keep up reviewing N changes in a 6 week release cycle, moving to a 6*X release cycle will not help you review N*X changes.

                                                                                                                            • rnijveld 1 year ago
                                                                                                                              I also agree that there is too much churn in the Rust ecosystem and that we should try and slow things down in the coming years. ntpd-rs also does this: our MSRV is 1.70 right now (that was released over a year ago) and we test our code on CI against this version (as well as the current stable release). And we go a little further. Using the `direct-minimal-versions` (nightly only right now unfortunately) flag we downgrade our dependencies to the minimal ones we've specified in our `Cargo.toml` and test against those dependencies as well, as well as the latest dependencies specified in `Cargo.lock` which we update regularly. This allows us to at least partially verify that we still work against old versions of our dependencies, allowing our upstream packagers to more easily match their packages against our own. Of course we all should update to newer versions whenever possible, but sometimes that is hard to do (especially for package maintainers in distributions such as Fedora and Debian, who have to struggle with so many packages at the same time) and we shouldn't create unnecessary work when its not needed. Hopefully this is our way of helping the ecosystem slow down a little and focus more on security and functionality and less on redoing the same thing all over again every year because of some shiny new feature.
                                                                                                                              • dequan 1 year ago
                                                                                                                                100% agreed. Thank you for your work on ntpd-rs!
                                                                                                                              • chaoskitty 1 year ago
                                                                                                                                Serious question: why does it need to change so much and so often? I know nothing about rust development, so I'm curious about why it's worlds different from the development of other toolchains.
                                                                                                                                • sans-seraph 1 year ago
                                                                                                                                  Let us be clear that the notion of "change" being referred to here is forward compatibility, not backward compatibility. The user is commenting on the fact that Rust library authors make use of new features as they become available, and as a result in order to compile Rust code you will often need a recent version of the compiler, or otherwise you will need to find an older version of the library in question.

                                                                                                                                  In addition Rust was born from Mozilla and imitates Firefox's rapid release schedule of one release per six weeks. This does not mean that Rust releases are substantial or that they are traumatizing, only that they are frequent. The contract with users is that Rust releases must be painless so as to not fatigue users and discourage them from upgrading. The success of this painless upgrade strategy is proved by the fact that library authors are so quick to upgrade, as mentioned.

                                                                                                                                  This is in contrast to other languages where historically a new version of their compiler might only be released as infrequently as once per three years. It seems that these languages have begun taking queues from Rust as even Java now releases once per six months.

                                                                                                                                  • kelnos 1 year ago
                                                                                                                                    Consider that there's still a good amount of work being done on gcc & clang's C++ frontend despite how old these languages are. Wouldn't it stand to reason that Rust, a comparatively very new language, would have new features and compiler improvements added at an even faster pace?

                                                                                                                                    I suspect if you were to look at other reasonably popular languages that are of the same era, you'll see a similar level of changes.

                                                                                                                                    • stouset 1 year ago
                                                                                                                                      > why does it need to change so much and so often

                                                                                                                                      I think a lot less is changing than you seem to think is.

                                                                                                                                  • 1 year ago
                                                                                                                                  • hoseja 1 year ago
                                                                                                                                    Free pair of knee-high socks with every cert.