Silkenweb Example: Hackernews Clone

Thanksgiving 2023 security incident

643 points by nomaxx117 1 year ago | 322 comments

BytesAndGears 1 year ago
Writeups and actions like this from cloudflare are exactly why I trust them with my data and my business.
Yes, they aren’t perfect. They do some things that I disagree with.
But overall they prove themselves worthy of my trust, specifically because of the engineering mindset that the company shares, and how serious they take things like this.
Thank you for the blog post!
- nimbius 1 year ago
  Then, the advertisement worked.
  - Insist that you have better integrity than your competitors
  - share a few operational investigations after your latest security event
  what cloudflare doesnt do is provide their SOC risk analysis as a PCI/DSS payment card processor. Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.
  They mention a third-party audit was conducted, but thats not because they care about you. Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information, it needs to pass a yearly on-premise audit to ensure PCI compliance. if they didnt, major credit houses would stop processing their payments.
  - tptacek 1 year ago
    None of this would have had anything to do with PCI (nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches). At much smaller company sizes than this, insurance requires you to retain forensics/incident response firms. There's a variety of ways they could do that cheaply. They brought in the IR firm with the best reputation in the industry (now that Google owns Mandiant), at what I'm assuming are nosebleed rates, because they want to be perceived as taking this as seriously as they seem to be.
    It's a very good writeup, as these things go. Cloudflare is huge. An ancillary system of theirs got popped as sequelae to the Okta breach. They reimaged every machine in their fleet and burned all their secrets. People are going to find ways to snipe at them, because that's fun to do, but none of those people are likely to handle an incident like this better.
    I am not a Cloudflare customer (technically, I am a competitor). But my estimation of them went up (I won't say by how much) after reading this writeup.
    - mardifoufs 1 year ago
      Yeah at best PCI is somewhat hard to get at first, but after that it's basically only good, or less shady, corporations that bother keeping up compliance or make sure that they follow the guidelines at every step. Shady/troubled operators don't, and to an extent don't have to really be afraid of losing said certification unless they just go fully rogue.
    - yardstick 1 year ago
      > nobody gives a shit about PCI; the worst shops in the world, the proprietors of the largest breaches, have had no trouble getting PCI certified and keeping certification after their breaches
      IMO it’s more a risk reward trade off. I know some companies are paying relative peanuts in non-compliance fines rather than spend money on some semblance of security which they may still not be compliant with and have to pay the fines anyway…
  - eastdakota 1 year ago
    I was the one who made the call to bring in CrowdStrike. It had zero to do with PCI/DSS or any other compliance obligation. It was to 1) bring in a team with deep experience with a broad set of breaches; and 2) to make sure our team didn’t miss anything. The CrowdStrike team were first class and it was good to confirm they didn’t find anything significant our team hadn’t already. And, for the sake of clarity, no system breached touched customer credit card or traffic information.
    - elitistphoenix 1 year ago
      Was the self hosted environment running a AV like the Crowdstrike agent? Or was it running different AV and that's why you chose to use Crowdstrike as someone different?
      I guess no need to specific names. I'm just using that as examples.
    - 1 year ago
  - JakeTheAndroid 1 year ago
    I am not sure where you're getting your information on requirements for PCI service providers. There isn't anything inside of PCI DSS that requires some sort of SOC report to be generated and distributed to customers. And Cloudflare does make their PCI AoC available to customers.
    They clearly defined the scope of impact, and demonstrated that none of this impacts systems in scope for PCI. There was no breach to change management inside of BitBucket, and none of the edge servers processing cardholder data were impacted. They will have plenty of artifacts to demonstrate that by bringing in an external firm. So I am really not clear why you're bringing up PCI at all here. They made it clear no cardholder data was impacted so your perspective on the required "on-site" audits is moot.
    Cloudflare operates two entirely different scopes for PCI; The first being as a Merchant where you the customer pays for the services. This is a very small scope of systems. The second is as a Service Provider that processes cards over the network. The network works such that it is not feasible to exfiltrate card data from the network. There are many reasons as to why this is, but they demonstrate year over year that this is not something that is reasonably possible. You can review their PCI AoC and get the details (albeit limited) to understand this better. Or you could get their SOC 2 Type 2 Report which will cover many aspects of the edge networks control environment with much better testing details. After reading that you can then come back to the blog and see that clearly no PCI scoped systems were impacted in a way that would require any on-prem audit to occur.
    And they are not a card network. They are a PCI Service Provider because cards transit over their network. They are not at risk of being unable to process payments or transactions for their Merchant scope even if there are issues with their Service Provider scope. Because, again, these are two separate PCI audits that are done, testing two different sets of systems and controls.
    And, as an aside, Cloudflare effectively always has on-prem PCI audits occur. Because the PCI QSA's need to physically visit Cloudflare datacenters to demonstrate not only the software side of compliance, but the datacenters deployed globally.
  - autoexec 1 year ago
    > Cloudflare doesnt explain why they ignored/failed to identify the elevated accounts or how those accounts became compromised to begin with. They just explain remediation without accountability.
    They did, and they admitted that it was their fault. I have to give them credit for that much.
    > They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023...The one service token and three accounts were not rotated because mistakenly it was believed they were unused. This was incorrect and was how the threat actor first got into our systems and gained persistence to our Atlassian products. Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
    - tru3_power 1 year ago
      The fact that they got their internal source/all bug reports is so bad. Literally every known and unknown vuln in their source is now up for grabs.
  - mynameisvlad 1 year ago
    I’m sorry did we read the same write-up?
    Like I get cynicism, but they very clearly explained the lead-up to the accounts being compromised and the mistakes that caused that. They took full accountability of it. Which is frankly more than most companies dealing with security incidents. This entire write-up is more than most companies obligations or responses.
    - michaelt 1 year ago
      > This entire write-up is more than most companies obligations or responses.
      The thing is: What standard of security would you expect of someone who was decrypting 1/3rd of your internet traffic?
      I would say "better than most companies" is too low a bar. Hell, I'm not sure any organisation could be secure enough to be trusted with that.
  - bimguy 1 year ago
    Nimbius, you sound like you work for a Cloudflare competitor.
    No competitors were mentioned in Cloudflares article, they explained what kind of information was breached, nothing to do with payment/card info... so I doubt you even read past the first few paragraphs/conclusion.
  - ziddoap 1 year ago
    >Its because PCI/DSS mandates when an organization of any level experiences a data breach or cyber-attack that compromises payment card information
    No payment card information was compromised.
- Xeyz0r 1 year ago
  Nobody is perfect, but Cloudflare indeed inspires confidence. Especially thanks to cases where they don't hesitate to talk about the issue and how they resolved it. It's precisely these descriptions of such situations that demonstrate their ability to overcome any challenges.
- overstay8930 1 year ago
  We're one of their larger enterprise customers and stuff like this makes it easy to get renewals approved easily, keeping engineers in the loop makes it such an easy sell.
- zelon88 1 year ago
  You actually believe that an intruder gained access to their KB/Tickets and didn't manage to get valuable information? That's not what Jira is for. If you know what Jira is for, and you're willing to run it on-prem, then you know the purpose of doing all that work is because you have something valuable to store in there.
  I don't believe they didn't lose anything. That's not how this works, and most Jira/Confluence I've seen is loaded with secrets.
- encom 1 year ago
  Better hope you stay on their good side, and don't say anything their CEO doesn't approve of.
- el-dude-arino 1 year ago
  [flagged]
  - _heimdall 1 year ago
    What are they now, if not an engineering company?
    - el-dude-arino 1 year ago
      [flagged]
kccqzy 1 year ago
> Analyzing the wiki pages they accessed, bug database issues, and source code repositories, it appears they were looking for information about the architecture, security, and management of our global network; no doubt with an eye on gaining a deeper foothold.
For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company and then have the person send back "information about the architecture, security, and management" of the target company.
Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.
- neilv 1 year ago
  > Fun (but possibly apocryphal) fact: more than a decade ago in a social gathering of SREs at Google, several admitted to being on the payroll of some national intelligence bureaus.
  They had government engagements with Google's consent, and all those various engagements could be disclosed to each other?
  If not, what kind of drugs were flowing at this social gathering, to cause such an orgy of bad OPSEC?
  - slowbdotro 1 year ago
    Knowing google employees, it's coke. Lots of coke
    - neilv 1 year ago
      At last, an explanation for their fratbro interviews.
- _kb 1 year ago
  Payroll? You guys are getting paid?
  Australians get the 'opportunity' to be part of that sort of that sort of espionage as a base level condition of citizenship [0].
  As an upside, I guess it helps with encouraging good practices around zero trust processes and systems dev.
  [0]: https://en.wikipedia.org/wiki/Mass_surveillance_in_Australia...
- owlstuffing 1 year ago
  > For a nation state actor, the easiest way to accomplish that is to send one of their loyal citizens to become an employee of the target company
  Precisely. Particularly in the case of US businesses. Why bother picking a lock when you have both the key and permission?
- toyg 1 year ago
  Not if such citizens are sanctioned. Code Red. Hint hint.
  - eep_social 1 year ago
    > we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”.
    Code red is a standard term in emergency response that means smoke/fire. In general, in order to “redirect” that much effort one must do some paperwork to prove the urgency and immediacy of the threat.
    The MO screams China to me but I wouldn’t read anything into the name “code red” which would have been selected before they identified the specific threat actor anyway.
    - eastdakota 1 year ago
      The name has nothing to do with where we believe the attacker came from. We borrowed it from Google. At Google they have a procedure where, in an emergency, they can declare a Code Yellow or Code Red — depending on the severity. When it happens, it becomes the top engineering priority and whoever is leading it can pull any engineer off to work on the emergency. Those may not be the exact details of Google's system but it's the gist that we ran with. We'd had an outage of some of our services earlier in the Fall that prompted us to first borrow Google's idea. Since our logo is orange, we created "Code Orange" to mitigate the mistakes we'd made that led to that outage. Then this happened and we realized we needed something that was a higher level of emergency than Code Orange, so we created Code Red. At some point we'll write up how we thought of the rules and exit criteria around these, but I think they'll become a part of how we deal with emergencies that come up going forward.
    - 4gotunameagain 1 year ago
      > The MO screams China to me
      How exactly ?
      Nothing out of the ordinary/regular infiltration, investigation and attempt to move laterally is exposed.
  - elashri 1 year ago
    I think this probably was a name after the famous Code Red worm [1], not a reference to China.
    [1] https://en.wikipedia.org/wiki/Code_Red_(computer_worm)
    - duskwuff 1 year ago
      Or after the flavor of Mountain Dew which was that worm's namesake. Not all names have to make sense. :)
    - curiousgal 1 year ago
      It's the tech scene on the Internet, everything is a reference to the CCP! /s
marcinzm 1 year ago
> we were (for the second time) the victim of a compromise of Okta’s systems
I'm curious if they're rethinking being on Okta.
- twisteriffic 1 year ago
  This wasn't really an additional failure at Okta. This was credentials lost during the original Okta compromise that CloudFlare failed to rotate out.
  Okta deserves criticism for their failure, but this feels like CloudFlare punching down to shift blame for a miss on their part.
  - cowsandmilk 1 year ago
    This wasn’t a new compromise, but there were still two Okta compromises that impacted CloudFlare
    January 2022: https://blog.cloudflare.com/cloudflare-investigation-of-the-...
    October 2023: https://blog.cloudflare.com/how-cloudflare-mitigated-yet-ano...
  - sophacles 1 year ago
    How is Cloudflare ($384M in revenue, q3 2023) punching down at Okta ($584M in revenue, q3 2023) by stating exactly what happened.
    If anything Okta is a bigger company (by revenue, by employee count) and they were founded a year earlier.
  - bigbluedots 1 year ago
    > They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023
    It's fair to "punch down" imo as that's how the credentials were originally compromised. I'd agree with you if CF were trying to minimize their own mistake but that doesn't seem to be what is happening here
    - BeefWellington 1 year ago
      If a breach is disclosed and some time later your systems are compromised because you didn't bother to take appropriate action in response to that, it's not "fair" to punch down, or even reasonable to do so.
  - jamiesonbecker 1 year ago
    Agreed, but Okta's still a $14B company.
- BytesAndGears 1 year ago
  My company will only give us new laptops that are preinstalled with Okta’s management system.
  I am grandfathered in to an old MacBook that has absolutely no management software on it, from the “Early Days” when there was no IT and we just got brand new untouched laptops.
  They offered me an upgrade to an M1/M2 pro, but I refused, saying that I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.
  Since that would hugely disrupt my work, I can’t upgrade. Maybe I can use incidents like this to justify my beliefs to the IT department…
  - samcat116 1 year ago
    > new laptops that are preinstalled with Okta’s management system
    Okta doesn't make device management software, thats made by companies like Jamf. Okta can integrate with them but Okta isn't what manages your laptop at all.
    > I wasn’t willing to use Okta’s login system if I have my own personal passwords or keys anywhere on my work computer.
    Do not do this, its not a personal device.
    - michaelt 1 year ago
      > Do not do this, its not a personal device.
      You think nobody's logged into their personal spotify on their work computer? All those guys wearing headphones in the office have brought in CDs to play in their laptop CD drives?
      And that business traveller away from their partner and kids for a week+ isn't going to video call them? Or watch some netflix in their hotel room in the evening?
      That's so unrealistic, you could write IT security policy for a Fortune 100 company :)
    - derivagral 1 year ago
      > Do not do this, its not a personal device.
      Agreed, but I knew many devs in my career who mix personal stuff into work hardware. Maybe its just spotify/pandora, maybe some HR thing they needed their personal gmail to make it easier.
      This included "senior" and other levels, it isn't just ppl out of college.
  - yjftsjthsd-h 1 year ago
    > if I have my own personal passwords or keys anywhere on my work computer.
    Well... don't do that? Why would you ever have personal anything on a work computer?
    - Symbiote 1 year ago
      A Github account, for one possible example.
    - 2devnull 1 year ago
      HR forms require personal information and do not allow anyone to access from anything but a corporate device.
  - ocdtrekkie 1 year ago
    I wouldn't use Okta at work, but as a network administrator, I also wouldn't allow your improperly managed laptop to talk to business resources (and I'd demand anyone overruling me sign a written statement demanding it to exempt me for responsibility for it). Wild you work somewhere that is letting you get away with that.
  - verve_rat 1 year ago
    Why do you need personal passwords on your laptop to do your work? I'm not understanding this.
    - BytesAndGears 1 year ago
      Fair question, but I use a lot of things that are varying degrees of helpful for my work:
      * personal ChatGPT and copilot subscriptions, since company doesn’t pay for these
      * Trello account for keeping track of my todo list (following up with people, running deploys)
      * Obsidian for keeping notes, as a personal knowledge-base (things like technologies and reminders)
      * Apple account for music, copy/paste, sharing photos from my travel with coworkers, synching docs related to my work visa and taxes
      * Personal slack login for communicating with my partner in our private server
      * personal GitHub account credentials for synching my private dotfiles repo with my neovim config. basically can’t work without my dotfiles, but I could theoretically email these to myself or something, to prevent this one.
      And sure, I could be stubborn and not use any of this, but I’d be way less productive and kinda miserable.
    - deathanatos 1 year ago
      The parent's view does seem a bit extreme, but there is always some overlap. Whatever HR system you have is going to be in a weird area of personal/employee overlap, as it'll need to have a password that your personal life has access to. (As tax documents, pay stubs, benefits stuff, etc. all impact the "personal" side of one's life. E.g., I need to store — in my personal archives — the years W-2.)
      Also, people just do things for convenience. (Although I tend to pipe these passwords over an SSH connection, so that they're not resident on the work laptop. Though there is a good argument to be had about me permitting my work laptop SSH access to my personal laptop. From a technical standpoint, my employer could hack/compromise my personal laptop. From a legal and trust standpoint, I presume they won't.)
  - prg20 1 year ago
    IT departments do not care for the musings of self-proclaimed neckbeards. The arrogance of such users to comfort to security policy is a well known risk and usually grounds for swift disciplinary action.
  - giancarlostoro 1 year ago
    At a former employer, the company wanted everyone's machine to have full hard drive encryption, thankfully we were on Linux and the vendor they wanted to use was not compatible. Even so, I always encrypt my installs on laptops, not so much workstations (I didn't install it, and reinstalling would be a mess) since at that point you're physically in the office building and we have more serious problems.
  - 1 year ago
- Icathian 1 year ago
  The challenge being, who else could possibly handle Cloudflare's requirements? I imagine the next step is to build their own, and that's obviously not an easy pill to swallow.
  - amluto 1 year ago
    Why not? Cloudflare already operates a system that can help customers to require SSO for access to their services — why not try to capture more of that vertical by becoming an IdP?
  - whalesalad 1 year ago
    They already run their own zero trust infrastructure for customers, kinda surprised they are not dogfooding it. https://www.cloudflare.com/plans/zero-trust-services/
    - tomschlick 1 year ago
      They are, but they don't have management for user accounts, 2fa, etc. You setup a connection to something like Okta, Google Apps, O365, SAML, etc to be your persistent user db and cloudflare just enforces it.
      I wouldn't be surprised if they are working on first party IAM user support though.
    - jgrahamc 1 year ago
      We use our Zero Trust stuff extensively. In fact, we built it for ourselves initially.
    - margalabargala 1 year ago
      There are good reasons not to dogfood critical services like that; it can make recovering from unexpected issues much harder if you introduce mutual dependencies.
      For example, if Slack devops team were to exclusively communicate over Slack, then a Slack outage would be much harder to resolve because the team trying to fix it would be unable to communicate.
    - mikey_p 1 year ago
      Did you read the article?
      They are using zero trust and explained that it's why the scope of the security incident was extremely limited.
- 1 year ago
OJFord 1 year ago
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.
Eh? So why weren't they revoked entirely? I'm sure something's just unsaid there, or lost in communication or something, but as written that doesn't really make sense to me?
- crdrost 1 year ago
  I would assume that "believed" is not meant to be interpreted in an active personal sense but in a passive configuration sense.
  That is, I'd expect there was a flag in a database somewhere saying that those service accounts were "abandoned" or "cleaned up" or some other non-active status, but that this assertion was incorrect. Then they probably rotated all the passwords for active accounts, but skipped the inactive ones.
  Speaking purely about PKI and certificate revocation, because that's the only similar context that I really know about, there is generally a difference between allowing certificates to expire, vs allowing them to be marked as "no longer used", vs fully revoking them: a certificate authority needs to do absolutely nothing in the first case, can choose to either do nothing or revoke in the second case, and must actively maintain and broadcast that revocation list for the third case. When someone says "hey I accidentally clobbered that private key can I please have a new cert for this new key," you generally don't add the old cert to the revocation list because why would you.
- htrp 1 year ago
  blameless post mortem most likely
  Great call out too
  > Note that this was in no way an error on the part of AWS, Moveworks or Smartsheet. These were merely credentials which we failed to rotate.
  - OJFord 1 year ago
    It can still be blameless though? The 'because' makes it sound like that's a correct reason to leave it; that the only error was thinking they were unused. (i.e. that it's fine to leave them if unused, only a problem if they're used)
    i.e. instead of 'because they were mistakenly thought to be unused' you can say 'because they were mistakenly thought to be ok to leave as unused' (or something less awkward depending on exactly what the scenario was) and there's no more blame there? And if you really want to emphasise blamelessness you can say how your processes and training failed to sufficiently encourage least privilege, etc.
- stepupmakeup 1 year ago
  Rotating could have been manual and the person in charge wanted to save time. Stress could be a factor too.
- phyzome 1 year ago
  Betting they have a new item in their compromise runbook. :-)
  - OJFord 1 year ago
    No I don't think so, I do think something's just difficult to say because of what they can't say, or they just neglected to say/didn't word it well, or something. i.e. a bug in the writing, not the post mortem itself.
    Because if you take it exactly as it's written it's just too weird, I'm not a security expert with something to teach Cloudflare about err maybe don't leave secrets lying around that aren't actually needed for anything, that's not news to many people, and they surely have many actual security people for whom that would not even be a fizzbuzz interview question reviewing any kind of secret storage or revocation policy/procedure. And also the mentioned third-party audit.
sevg 1 year ago
> Even though we believed, and later confirmed, the attacker had limited access, we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials), physically segment test and staging systems, performed forensic triages on 4,893 systems, reimaged and rebooted every machine in our global network including all the systems the threat actor accessed and all Atlassian products (Jira, Confluence, and Bitbucket).
> The threat actor also attempted to access a console server in our new, and not yet in production, data center in São Paulo. All attempts to gain access were unsuccessful. To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
They didn't have to go this far. It would have been really easy not to. But they did and I think that's worthy of kudos.
- barkingcat 1 year ago
  I think they did have to do that far though.
  Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit. Imagine getting in at the centre of a new Meet-Me room (https://en.wikipedia.org/wiki/Meet-me_room) and having persistent access to key switches there.
  Cloudflare datacentres tend to be at the hub of insane amounts of data traffic. The fact that the attacker knew how valuable a "pre-production" data centre is means that cloudflare probably realized themselves that it would be a 100% game over if someone managed to get a foot hold there before the regular security systems are set up. It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.
  Also remember, at the beginning of data centre builds, all switches/equipment have default / blank root passwords (admin/admin), and all switch/equipment firmware are old and full of exploits (you either go into each one and update the firmware one by one or hook them up to automation for fleet wide patching) Imagine that this exploit is taking place before automation services had a chance to patch all the firmware ... that's a "return all devices to make sure the manufacturer ships us something new" event.
  - vasco 1 year ago
    What I think they meant is customers would keep paying them. And they are right, one just has to look at Okta, Solarwinds and other providers that have been owned, not done half of this and somehow are still in business. Everyone whistles to the side and pretends they shouldn't switch vendors, rotate all creds, cycle hardware, because it saves lots of work and this stuff falls under "reasonable oopsie" to the general public, when in fact there should be rules about what to do in the event of a breach that should be much stricter. So they do some partial actions to "show work" in case of lawsuits and keep going. The old engineers leave, new ones come in, and now you have systems who are potentially owned for years to come.
    It takes some honesty and good values by someone in the decision-making to go ahead with such a comprehensive plan. This is sad because it should be tablestakes, as you say correctly, but having seen many other cases, I think although they did "the expected", it's definitely above and beyond what peers have done.
  - tgsovlerkhgsel 1 year ago
    > It would be a company ending event if someone managed to install themselves inside a data centre while it was being built/brought up.
    It wouldn't. Most people like to assume the impact of breaches to be what it should be, not what it actually is.
    Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.
    - mschuster91 1 year ago
      > Look at the 1-year stock chart of Okta and, without looking up the actual date, tell me when the breach happened/was disclosed.
      The problem with this is that while security minded people know what Okta is and why to stay the fuck away from handing over your crown jewels to a SaaS company is warranted, C-level execs don't care. They only care about their golf course or backroom deal friends and about releasing PR statements full of buzzwords like "zero trust", "AI based monitoring" and whatever.
      The stock markets don't care either, they only look at the financial data, and as long as there still are enough gullible fools signing up, they don't care and stonk goes up.
  - nolok 1 year ago
    > It would be a company ending event
    Given they got out of cloudbleed without any real damage let alone lasting damage, I disagree.
    (I don't disagree with your point about how bad of a problem this would be, I'm just insisting that security failure is not taken seriously at all by anyone)
    - chx 1 year ago
      Presuming taviso is not exaggerating and why would he CF's reply to cloudbleed was ... not quite nice.
      https://twitter.com/taviso/status/1566077115992133634
      > True story: After cloudbleed, cloudflare literally lobbied the FTC to investigate me and question the legality of openly discussing security research. How come they're not lobbying their DC friends to investigate the legality KF?
      For those not familiar with the history , this tweet started the cloudbleed disclosure to cloudflare:
      https://twitter.com/taviso/status/832744397800214528
      > Could someone from cloudflare security urgently contact me.
      This followed: https://blog.cloudflare.com/incident-report-on-memory-leak-c...
    - Hrundi 1 year ago
      I don't remember any companies that ended thanks to cloudbleed, but I'd be happy to be proven wrong
  - meowface 1 year ago
    >Getting in at the "ground floor" of a new datacentre build is pretty much the ultimate exploit.
    I can just imagine the attackers licking their lips when they first breached the data center.
    Good reminder to use "Full (Strict)" SSL in Cloudflare. Then even if they do get compromised, your reverse-proxied traffic still won't be readable. (Of course other things you might use Cloudflare for could be vectors, though.)
    - ownagefool 1 year ago
      Cloudflare is essentially a massive mitm proxy. If you manage to pwn a key, you have access to traffic.
      I'm sure they're better than this than me, but ipxe & tftp are plain text, so it wouldn't be shocking if something in the bootstrap process was plaintext.
      At the very least you need to tell the server what to trust.
  - sneak 1 year ago
    > Imagine getting in at the centre of a new Meet-Me room and having persistent access to key switches there.
    This wouldn't get you much. We already assume the network is insecure. This is why TLS is a thing (and mTLS for those who are serious).
    - sophacles 1 year ago
      I suspect "we" is a much smaller group than you imagine. I've gotten pcaps from customers as recently as this year that include unencrypted financial transaction data. These were captured on a router, not an end host, so the traffic was going across the client's network raw.
    - noizejoy 1 year ago
      > We already assume the network is insecure.
      Maybe naively, I wish this assumption became universal.
  - swyx 1 year ago
    do manufacturers share some of the cost of this kind of security related return or is this a straight up "pay twice for the same thing" financial hit?
    - eastdakota 1 year ago
      We have very good relations with our network vendors (in this case, Cisco, Juniper, and Arista). The CEOs of all of them 1) immediately got on a call with me late on a weekend; 2) happily RMAed the boxes at no cost; and 3) lent us their most senior forensics engineers to help with our investigation. Hat tip to all of them for first class customer service.
- readyplayernull 1 year ago
  > The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
  Aha, the old replace-your-trusted-hardware trick.
  - zitterbewegung 1 year ago
    Manufacturers have had security vulnerabilities for hardware to the point that the firmware on device couldn’t be trusted to be replaced so they said to get new hardware so it’s not a bad strategy.
  - AzzyHN 1 year ago
    In a corporate environment, standard procedure when an employee's computer gets infected is to re-image it. Even if it was a stupid virus that was immediately caught, the potential risk of undetected malware running amuck is just too high.
    Now imagine, instead of Steve from HR's laptop, it's one of Cloudflare's servers.
- schainks 1 year ago
  Having seen the small number of DEFCON talks that I've seen, I would have absolutely gone that far.
- ldoughty 1 year ago
  The nuclear response to compromise should be the standard business practice. It should be exceptional to deviate from it.
  If you assume that they only accessed what you can prove they accessed, you've left a hole for them to live in. It should require a quorum of people to say you DON'T need to do this.
  Of course, this is ideal world. I'm glad my group is afforded the time to implement features with no direct monetary or user benefit.
- tptacek 1 year ago
  This is why old secops/corpsec security hands are so religious about tabletop exercises, and what's so great about BadThingsDaily† on Twitter. Being prepared to do this kind of credential rotation takes discipline and preparation and, to be frank, most teams don't make that investment, including a lot of really smart, well-resourced ones.
  If Cloudflare is in a position where their security team can make a call to rotate every secret and reimage every machine, and then that happens in some reasonable amount of time, that's pretty impressive.
  † https://twitter.com/badthingsdaily?lang=en
  - akira2501 1 year ago
    It'd be more impressive if they actually got all the credentials.
    It's good that you think you can absorb a complicated security task, it's useless if you have no way to test or verify this action.
    - swyx 1 year ago
      yes but this is a nice #2. not many fortune 500s would 1) even know they were breached and 2) if they were breached, have the breach be so contained.
- syncsynchalt 1 year ago
  Honestly I wish we'd had an excuse/reason to do an org-wide prod creds refresh like this at some places I've been.
  You find some scary things when you go looking for how exactly some written-by-greybeard script is authenticating against your started-in-1990s datastore.
- orenlindsey 1 year ago
  Cloudflare is showing how to correctly respond to attacks. Other companies should take note.
- 1 year ago
sebmellen 1 year ago
The most surprising part of this is that Cloudflare uses BitBucket.
- Cthulhu_ 1 year ago
  How so? It integrates well with the other Atlassian products they use.
- toyg 1 year ago
  Integrates with Jira and the rest of Atlassian's stuff, and it's just another git server at the end of the day.
- infecto 1 year ago
  Maybe but maybe not. I don't like Bitbucket but there are a number of large companies where they worry about using services owned by competitors in one of their verticals.
  - kccqzy 1 year ago
    Bitbucket doesn't have to be a service. It can be an old-fashioned downloaded software that you install on your own machines. Not everything is SaaS.
    - infecto 1 year ago
      Not sure what you mean? If you are alluding to the OP that said it was surprising...I don't think he found it suprising they they use Bitbucket over Mercurial. I think its safe to assume he meant bitbucket over a Github.
      In the git universe there is a pretty short list of services, locally or hosted that you would probably use as an entity as large as cloud flare.
- aitchnyu 1 year ago
  Wonder how powerful is Scriptrunner for Jira. They got the security certifications but I cant tell how sandboxed it is.
  - itomato 1 year ago
    As well as can be expected in a company that gives Smartsheet access to Jira with an Admin Service Account.
    https://github.com/BishopFox/sliver
    "Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin."
    https://blog.cloudflare.com/thanksgiving-2023-security-incid...
- gempir 1 year ago
  A lot of very big companies use Bitbucket, it's just a lot more cost effective than Gitlab/Github.
- 1 year ago
mmaunder 1 year ago
Thing about a data breach is once the data is out there - source code in this case - it’s out there for good and you have absolutely no control over who gets it. You can do as much post incident hardening as you want, and talk about it as much as you want, but the thing you’re trying to protect against, and blogging about how good you’re getting at preventing, has already happened. Can’t unscramble those eggs.
- BandButcher 1 year ago
  agreed, to me this is a big deal for CF. especially coupled with confluence documentation which most likely includes future plans and designs, org charts, meeting minutes... you could also find other easter eggs in any legacy code, almost all companies have undocumented backdoors
  obviously a customer data breach would be worse but this is really no bueno
- burnished 1 year ago
  Whats your point?
  - mmaunder 1 year ago
    That this is messaged and received as a net win. It’s not.
    - malwrar 1 year ago
      Are they just supposed to be invincible? Next best thing is an incident response with this level of quality and transparency. Thats definitely a win in my book, I want to know the provider of a core part of my infra is able to competently and maturely respond to a security incident and this post strongly communicates that.
- arp242 1 year ago
  The source code next year is not the same as source code this year.
  The customer data next year is not the same as the customer data this year.
muzso 1 year ago
> The threat actor searched the wiki for things like remote access, secret, client-secret, openconnect, cloudflared, and token. They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).
In Atlassian's Confluence even the built-in Apache Lucene search engine can leak sensitive information and this kind of access (to the info by the attacker) can be very hard to track/identify. They don't have to open a Confluence page if the sensitive information is already shown on the search results page.
fierro 1 year ago
>The one service token and three accounts were not rotated because mistakenly it was believed they were unused.
This odd to me - unused credentials should probably be deleted, not rotated.
- pbhjpbhj 1 year ago
  This smells weird, surely? I'd be looking at who chose not to rotate those particular credentials.
  1: "what are these accounts?"
  2: "oh they're unused, they don't even appear in the logs"
  1: "we should rotate them"
  2: "no, let's keep those rando accounts with the old credentials, the ones we think might be compromised ... y' know, for reasons"
  ?
  - pphysch 1 year ago
    More likely: "no one has any idea what these old credentials do, so let's not touch them and potentially break everything"
    - sodality2 1 year ago
      Sounds like the perfect time to revoke the credentials and find out what uses them, so we can find why they weren't registered as credentials in use. Personally I'd rather do that, have a team ready, and break production for x minutes in order to properly register auth keys.
      I'd definitely consider a "silent" credential - a credential not registered centrally - to be a huge red flag. Either it could get stolen, or break and no one knows how to regenerate it. And it's pretty easy as devs to quickly generate an auth key that ends up being used permanently, without any documentation.
    - fierro 1 year ago
      this is more plausible to me
- mparnisari 1 year ago
  Agreed. This whole post reads as "I'm the victim" but they don't admit on the one mistake that snowballed
  - 1 year ago
londons_explore 1 year ago
So after the Okta incident they rotated the leaked credentials...
But I think they should have put honeypots on them, and then waited to see what attackers did. Honeypots discourage the attackers from continuing for fear of being discovered too.
wepple 1 year ago
They mention Zero Trust, yet you can gain access to applications with just a single bearer token?
Am I missing something here?
There’s no machine cert used? AuthN tokens aren’t cryptographically bound?
This doesn’t meet my definition of ZT, it seems more like “we don’t have a VPN”
- prg20 1 year ago
  You're not. The article makes no sense. They claim robust security controls but apparely lacked a proper accounting of service accounts with external access, especially with admin access to freakin' Jira.
  - Bluecobra 1 year ago
    Thank you! The more I think about his it makes no sense to me. If a service account needs external access why can't they also whitelist connectivity to specific public IP addresses?
- asmor 1 year ago
  these were service accounts used by third parties to provide jira integrations, not a user account
  - Bluecobra 1 year ago
    If they are using Active Directory, wouldn’t a service account be no different than a regular employee account? Both a Jira service account and the CEO of Cloudflare are still Domain Users in AD. Granted, a service account should be way more locked down and have the least amount of access possible.
    - asmor 1 year ago
      https://developers.cloudflare.com/cloudflare-one/identity/se...
- Bluecobra 1 year ago
  Yeah it seems odd to me that their internal wiki, code repo, and Jira is exposed directly to the internet and arbitrary IPs could connect to it. Atlassian had a rash of vulnerabilities recently, who knows how many undiscovered ones still exist.
  If they had a VPN in place secured with machine certs, that would be yet another layer for an attacker to defeat.
this_steve_j 1 year ago
This is an excellent report, and congratulations are due to the security teams at CS for a quick detection, response and investigation.
It also highlights the need for a faster move in the entire industry away from long-lived service account credentials (access tokens) and toward federated workload identity systems like OpenId connect in the software supply chain.
These tokens too often provide elevated privileges in devops tools while bypassing MFA, and in many cases are rotated yearly. Github [1], Gitlab, and AZDO now support OIDC, so update your service connections now!
Note: I’m not familiar with this incident and don’t know whether that is precisely what happened here or if OIdC would have prevented the attack.
Devsecops and Zero Trust are often-abused buzzwords, but the principles are mature and can significantly reduce blast radius.
[1] https://docs.github.com/en/actions/deployment/security-harde...
jrockway 1 year ago
Which "nation state" do we think this was?
- meowface 1 year ago
  For these kinds of attacks it's nearly always China, Russia, US, or sometimes Iran. 95% chance it's either China or Russia, here.
  - 2OEH8eoCRo0 1 year ago
    When has it been the US?
    - toyg 1 year ago
      Stuxnet?
    - AzzyHN 1 year ago
      We do a lot of hacking
  - toyg 1 year ago
    Their response program being called "Code Red" is likely a hint.
- jedahan 1 year ago
  The writeup contains indicators, including IP addresses, and the location of those addresses. In this case, the IP address associated with the threat actor is currently located in Bucharest, Romania.
  - tomschlick 1 year ago
    No nation state is going to use IPs from their own country if they don't want to be caught. They will use multiple layers of rented VPS's with fake identities to pay for those resources.
    - jrockway 1 year ago
      Yeah. I've dealt with definitely-not-nation-states before, and their pattern was to sign up for free/cheap CI services (CircleCI, Github Actions, that sort of thing) and launch their attacks from there. The VPS thing also sounds very very plausible to me, I figured there was a long tail, but until I was looking up every network that was attacking us, I really had no idea how deep the long tail goes. I now feel like half the world's side hustle is to rent a server that they never update and host a couple of small business websites there.
  - CubsFan1060 1 year ago
    M247 is commonly used by VPN providers (https://www.reddit.com/r/PrivateInternetAccess/comments/8xwn...)
- lijok 1 year ago
  Which nation state has good enough employment protection laws that they can take weekends off while doing recon on a top value target?
  - icepat 1 year ago
    Yes, they must have been a member of the Norwegian Foreningen Svartehattehackere. They are a very strong union.
  - toyg 1 year ago
    Might be a coincidence. A certain nation-state is currently engaged in all-out war; the intruder might have been summoned to another, more urgent task.
  - papertokyo 1 year ago
    I assume the break is to have less chance of their activities be discovered and/or connected.
- godzillabrennus 1 year ago
  China.
- wubbert 1 year ago
  Israel.
orenlindsey 1 year ago
Cloudflare being compromised would be enormous. Something between 5 and 25% of all sites use CF in some fashion. An attacker could literally hold the internet hostage.
zelon88 1 year ago
What I don't understand is how they got access to Jira yet you still insist there was no compromise.
The very nature of Jira and Confluence (both terrible products, btw) is to collect documentation. I'm assuming it was an internal Jira/Confluence for engineering teams, but still. There have got to be addresses, passwords, service account info, all kinds of info. If it was a tech support server then it's impossible to assert that you didn't lose customer data.
So we have this double standard where you pay for this product that is designed to house your deepest secrets and most cherished organizational information, that's so important to you that you run on premises servers to keep it safe, but it's not important enough to constitute a real "beach".
You're lying. Either the server contained junk of no value in which case it wouldn't have existed in the first place, or you actually did lose something of value that you won't identify to us. Nobody sets up on-prem Jira just to leave it empty and never put secrets in it.
londons_explore 1 year ago
Am I the only one who just sees a totally blank page?
Viewing the HTML shows it's got an empty body tag, and a single script in the <head> with a URL of https://static.cloudflareinsights.com/beacon.min.js/v84a3a40...
- chankstein38 1 year ago
  No, that's also what I see. I'm not sure why you're getting downvoted.
  EDIT: re-opened the link a few minutes later and now I see the post
- overstay8930 1 year ago
  Happened to me on my iPhone too
j-rom 1 year ago
> To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers. The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
The thoroughness is pretty amazing
wowmuchhack 1 year ago
Such a beautiful report and beautiful ownage.
Whenever some shitty Australian telco gets owned, people are angry and call them incompetent and idiots; it's nice to see Cloudflare gets owned in style with much more class and expertise.
Like the rest of the HN crowd, this incident has only increased my trust in Cloudflare.
jshier 1 year ago
Fascinating and thorough analysis! I guess if you think an account is unused, just delete it!
- phyzome 1 year ago
  Probably safer to rotate the credentials and then schedule it for deletion later. Then if you discover it wasn't unused after all, you have an easier recovery... :-)
1 year ago
htrp 1 year ago
>They did this by using one access token and three service account credentials that had been taken, and that we failed to rotate, after the Okta compromise of October 2023. All threat actor access and connections were terminated on November 24 and CrowdStrike has confirmed that the last evidence of threat activity was on November 24 at 10:44.
Okta hitting everywhere
lopkeny12ko 1 year ago
> The manufacturers’ forensic teams examined all of our systems to ensure that no access or persistence was gained. Nothing was found, but we replaced the hardware anyway.
This seems incredibly wasteful.
Replacing an entire datacenter is effectively tossing tens of millions of dollars of compute hardware.
- perlgeek 1 year ago
  The sentence before...
  > To ensure these systems are 100% secure, equipment in the Brazil data center was returned to the manufacturers.
  It doesn't say all equipment, and that would have been very helpful. But if it's just two or three access devices sitting on the border, it's not so bad.
  Also, the manufacturer likely just sold the hardware to a different customer, sounds like it was pretty new and unused anyway. Just flash the firmware and you're good.
- rjzzleep 1 year ago
  It is, but for most of these components there is no other choice since there is no way to guarantee that nothing was changed. lvrick would say that's what why want to attest everything.
  Anyway, I really hope that the hardware isn't just tossed into the recycling, but provided to schools and other places that could put them to good use.
prg20 1 year ago
> Then, from November 27, we redirected the efforts of a large part of the Cloudflare technical staff (inside and outside the security team) to work on a single project dubbed “Code Red”.
Why didn't they start this effort BEFORE there was an incident?
> we undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials
Bearer credentials should already be rotated on a regular basis. Why did they wait until an incident to do this?
> To ensure these systems are 100% secure
Nothing is 100% secure. Not being to see and acknowledge that is a huge red flag.
> Nothing was found, but we replaced the hardware anyway.
Well that is just plain stupid and wasteful.
> We also looked for software packages that hadn’t been updated
Why weren't you looking for that prior to the incident?
> we were (for the second time) the victim of a compromise of Okta’s systems which resulted in a threat actor gaining access to a set of credentials.
And yet they continue using Okta. The jokes just write themselves.
> The one service token and three accounts were not rotated because mistakenly it was believed they were unused.
Wait, wait, wait. You KNEW the accounts with remote access to your systems were UNUSED and yet they continue to be active? Hahahahaha.
> The wiki searches and pages accessed suggest the threat actor was very interested in all aspects of access to our systems: password resets, remote access, configuration, our use of Salt, but they did not target customer data or customer configurations.
Totally makes sense, I'm sure the attacker was just a connoisseur of credentials and definitely did not want them to target customer data.
Aeolun 1 year ago
Reading this 2 months after the fact feels a bit late, but I guess it’s better for your stock price if these revelations happen with remediation already in hand?
Since they didn’t really have reason to believe my data was accessed, maybe that’s ok. I know from firsthand experience how hard rotating all your credentials across the whole org is.
- Cthulhu_ 1 year ago
  The final security report was only released yesterday, and the amount of work they did to make sure all of their systems were secure after the incident was A Lot; two months is pretty quick for a project of that scale IMO.
  - Aeolun 1 year ago
    Yes, but if after two months they’d found out that customer data had been compromised, that would be a little late for me to do anything about it.
    - eastdakota 1 year ago
      Had customer data been impacted we would have disclosed it immediately.
    - nemothekid 1 year ago
      What do you expect them to do? It sounds like you are complaining that they weren't able to instantly ascertain if customer data had been compromised.
- lavezzi 1 year ago
  Since December 18, new SEC rules require companies to report “material” cybersecurity incidents on a Form 8-K within four business days of their materiality determination.
  The rule does not set any specific timeline between the incident and the materiality determination, but the materiality determination should be made without 'unreasonable delay'.
joshbetz 1 year ago
What makes them think it's a nation state?
mmvasq 1 year ago
Really good write up
nullpointer00 1 year ago
I am trying to learn and understand the attack. Can someone please help me brainstorm some possibilities? (please note, I do not intend to question Cloudflare's business practices or security program; my intention is to learn and understand). --
Blog: The threat actor (TA) accessed Okta’s customer support system and viewed files uploaded by Cloudflare (CF) as support cases.
Why was the session token part of the support files uploaded to Okta support? Does Okta require it for troubleshooting?
TA hijacked a session token of a CF employee from a support ticket.
Blog: Using the token extracted from Okta, the TA accessed Cloudflare’s Okta and compromised two separate Cloudflare employee accounts within the Okta platform.
How did this happen? Was the stolen token privileged? Also, why only 2 employee accounts? Were these different employees, or was one the same one whose token was compromised? What does Okta employee account compromise mean - did TA reset the password and MFA, and how or was there no MFA?
Blog: TA used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code.
Did the stolen employee credentials only have access to Atlassian?
Blog: TA gained access to a set of credentials
Does this mean multiple credentials got uploaded to the Okta support system?
Blog: The Okta compromise was in October, but the threat actor only began targeting CF systems using stolen credentials from the Okta compromise in mid-November.
Does this mean the compromised token was long-lived?
Blog: We failed to rotate one service token and three service accounts (out of thousands) of credentials that were leaked during the Okta compromise.
I didn’t get this. Does this mean that over time, CF employees had uploaded support info for 1000s of apps managed by Okta? Which credentials did CF rotate initially after the Okta compromise?
Leaked Credentials: 1. Moveworks service token that granted remote access into our Atlassian system.
Is this service token a bearer token? And without expiry. Is this like an API key?
TA accessed Atlassian Jira and Confluence using the Moveworks service token to authenticate through the gateway.
2. A service account used by the SaaS-based Smartsheet application that had administrative access to our Atlassian Jira instance,
So here, the Smartsheet Saas was given access to the on-prem Atlassian Jira instance? What kind of trust is it? Is this as well managed through Okta? And how does support case filing include a service account? Here, does the Service account mean again some kind of hardcoded API key without expiry
TA used the Smartsheet service account to gain access to the Atlassian suite. They used Smartsheet credentials to create an Atlassian account that looked like a normal Cloudflare user. They added this user to a number of groups within Atlassian so that they’d have persistent access to the Atlassian environment.
Since the Smartsheet service account had administrative access to Atlassian Jira, the TA was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for the Jira plugin. This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access, the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL. The access was denied, and they could not access any global networks.
3. A Bitbucket service account, which was used to access our source code management system
4. AWS environment that had no access to the global network and no customer or sensitive data.
Were these AWS access keys? Also, it looks like these keys did provide access to the AWS account. That means the access key didn’t require MFA.
The only production system the TA could access using the stolen credentials was our Atlassian environment.
Mitigations:
Blog: We decided a huge effort was needed to further harden our security protocols to prevent the threat actor from being able to get that foothold had we overlooked something from our log files.
What does hardening security protocol mean here? Is it techniques for D&R or something else
Blog: We undertook a comprehensive effort to rotate every production credential (more than 5,000 individual credentials)
I believe this means forced resets on employees (Okta users), right?
jbverschoor 1 year ago
It's almost valentine
belltaco 1 year ago
Great write up.
> Over the next day, the threat actor viewed 120 code repositories (out of a total of 11,904 repositories
> They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 14,099 pages).
Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.
> Since the Smartsheet service account had administrative access to Atlassian Jira, the threat actor was able to install the Sliver Adversary Emulation Framework, which is a widely used tool and framework that red teams and attackers use to enable “C2” (command and control), connectivity gaining persistent and stealthy access to a computer on which it is installed. Sliver was installed using the ScriptRunner for Jira plugin.
> This allowed them continuous access to the Atlassian server, and they used this to attempt lateral movement. With this access the Threat Actor attempted to gain access to a non-production console server in our São Paulo, Brazil data center due to a non-enforced ACL.
Ouch. Full access to a server OS is always scary.
- spenczar5 1 year ago
  12k git repos can happen if the team uses github enterprise with forking internally.
  It can also happen in franken-build systems which encourage decoupling by making separate repos: one repo that defines a service’s API, containing just proto (for example). A second repo that contains generated client code, a third with generated server code, a fourth for implementation, a fifth which supplies integration test harnesses, etc…
  Sound insane? It is! But its also how an awful lot of stuff worked at AWS, just as an example.
  - sesm 1 year ago
    I can relate to this, it seems that code hosting providers push their users into having more repos with their CI limitations. I’ve noticed that with GitHub Actions, I assume Atlassian does the same.
- OJFord 1 year ago
  They probably have thousands of devrel/app engineer type example/demo/test repos alone - it doesn't say they're active.
  2M tickets - in my 4.5y at present company we've probably averaged about 10 engineers and totalled 4.5k tickets. Cloudflare has been around longer, has many more engineers, might use it for HR, IT, etc. too, might have processes like every ticket on close opens a new one for the reporter to test, etc. It sounds the right sort of order of magnitude to me.
- pkkim 1 year ago
  At least 25% of that is a single Jira project consisting of formulaic tickets to capture routine changes in production, and a large portion of that is created by automated systems. There may be other such projects too.
  Source: former Cloudflare employee
- ummonk 1 year ago
  The number of repositories sounds really high. The number of tickets doesn't.
- keltraine 1 year ago
  Blog updated:
  They accessed 36 Jira tickets (out of a total of 2,059,357 tickets) and 202 wiki pages (out of a total of 194,100 pages)
- Aeolun 1 year ago
  > Is it just me or 12K git repos and 2 million JIRA tickets sound like a crazy lot. 15K wiki pages is not that high though.
  I think my org has on the order of 3 repositories per dev? They seem to have 3200 employees, with what I assume to be a slightly higher rate of devs, so you’d expect around 6-7 thousand?
  2M Jira tickets is probably easily achieved if you create tickets using any automated process.
  - trollied 1 year ago
    They might create a JIRA ticket for each customer support interaction. Would make sense.
JoyRivers 1 year ago
[dead]
alam2000 1 year ago
[dead]
throwaway67743 1 year ago
[flagged]
- tomschlick 1 year ago
  Switching an entire orgs authentication to a new provider (or something built in house) isn't something you can (or should) do quickly.
TABrazil 1 year ago
[flagged]
- eastdakota 1 year ago
  I think that’s unlikely. São Paulo was a new, not fully provisioned facility that didn’t have all our security hardening in place. That’s why the threat actor likely targeted it. That it was in Brazil is most likely incidental.
culopatin 1 year ago
Im growing more and more annoyed at Cloudflare and their stupid “are you a human” crap.