I really can’t find myself to ever recommend atlassian products though.

If cost is the only measure: I understand. But time lost in various areas of the software package (performance alone! Before we get into weird UX paradigms and esoteric query languages, shoddy search systems etc;) surely has an impact on cost. Having your employees spending a lot of time navigating janky software has a cost too.

We did the same thing. We were fairly early and heavy users of the pagerduty API. We'd been using it for over a year and had integrated it fairly deeply into our SRE stack. When the renewal came up (we had a 3 year contract), they wanted to massively up the price due to our API usage. IIRC, it was more than 5x more. Our CTO had us port all of the tooling to opsgenie and we've never looked back.

I'm generally not a fan of Atlassian anything, but Opsgenie was really good before Atlassian purchased them and they're still really good.

I made that leap too (moved from a company that used Pagerduty to a company that used Opsgenie) and it is the same feature set but not the same quality. Pagerduty in my experience is rock solid - I used it on various high-page-frequency rotations for something like seven straight years and literally never saw a dropped alert/notification once.

On the flip side in the 8ish months I used Opsgenie I saw a litany of issues, like the mobile app silently logging people out (and thus not delivering push notifications) and the app failing to send SMS notifications. We had pages get dropped by the primary because they never got a notification.

I want exactly one feature from my pager, delivering 100% of my pages. It's not a situation where I'm ok with 99% success rate, and that seemed to be the tradeoff and what you are paying for with Pagerduty.

the opsgenie api is really bad though if you want to manage it as code/declaratively

I agree with the weird pushiness on other products that add specious business value, but I do think there is room for feature growth in the main product.

For example, the auto-merge functionality is somewhat useful, but it sometimes gets it wrong. Both merging and splitting alerts is extremely clunky. I'd also love to be able to have policies like "this alert is low-priority outside of business hours". The ability to re-open an accidentally closed alert. The ability to edit the alerting rules (the Global Ruleset) without needing to be a super-admin, as the Power That Be are reluctant to hand that out, also, the ability to just read the Global Ruleset with my peon privileges.

… the sort of spit & polish that doesn't happen, IMO, anymore, because everything is an MVP feature, before the agile scrum PM moves on to the next MVP feature. "Polish" just accumulates in the JIRA graveyard.

If I know someone else is on call and he's competent, I can sleep better.

I've said it before, I'll say it again. I don't care if it's the Joint Chiefs and the whole United Nations calling me up to tell me an asteroid will destroy the planet unless I sign on [0]. If it's after hours, it'll have to wait until tomorrow, possibly Monday.

[0] Spoiler alert: it's not.

Fixed: https://grafana.com/docs/grafana-cloud/oncall/

Dude, I hope you also read when I say that I love what you do and your reply just confirms I'm putting my money in the right hands.

I just wouldn't mind to be the last to upgrade to a newer version :)

I think your decisions were reasonable, as is the opinion of the person you're responding to.

To be fair, even in its current form, it should be possible to operate this system with sqlite (i.e. no db server) and in-process celery workers (i.e. no rabbit MQ) if configured correctly, assuming they're not using MySQL-specific features in the app.

Using a message bus, a persistent data store behind a SQL interface, and a caching layer are all good design choices. I think the OP's concern is less with your particular implementations, and more with the principle of preventing operators from bringing their own preferred implementation of those interfaces to the table.

They mentioned that it makes sense because you were a standalone product, so stack portability was less of a concern. But as FOSS, you're opening yourself up to different standards on portability.

It requires some work on the maintainer to make the application tolerant to different fulfillments of the same interfaces. But it's good work. It usually results in cleaner separation of concerns between application logic and caching/message bus/persistence logic, for one. It also allows your app to serve a wider audience: for example, those who are locked-in to using Postgres/Kafka/Memcached.

Nothing wrong with that. I managed 7+ Sensu "clusters" at a previous job, and it's stack was a ruby server, Redis and RabbitMQ. But I completely ditched RabbitMQ and used Redis for the queue and data. Simpler, more performant and more reliable (even if the feature was marked experimental). Our alerts were really spammy, and we had ~8k servers (each running a bunch of containers) per cluster, so these things were busy. Each cluster was 3x small nodes (6gb memory, 2CPU) Memory usage was miniscule, typically <300mb. Any box could be restarted without any impact because Redis just operated in (failover) mode and Sensu was horizontally scalable.

I get why you would add a relational DB to the mix. Personally, I'd like a Rabbit-free option.

Hearing your message bus assumption sounds like one of the most ridiculous claims I heard.

Sorry but why is rabbitmq really necessary?

You don't need Rabbit, Celery, or Redis. You should be able to replace MySQL with SQLite. Then it would be radically easier to deploy.

That picture shows the complete monitoring stack with lots of optional components (pushgw, service discovery, Grafana).

When comparing to OnCall you need OnCall AND still the rest of that Prometheus picture.

Compare with this picture where everything in the leftmost Alert Detection box is what you see in the Prometheus architecture. https://grafana.com/docs/oncall/latest/getting-started/

OnCall also does nothing unless you have something external firing alerts for you. They both fill similar niches in a larger monitoring system; this does not excuse OnCall having a drastically more complex internal architecture.

For something that is supposed to add some more features to the basic email/HTTP message alert like grafana generates, I do wonder what extra features require an additional 2 databases, a message queue and a separate task queue.

> several years of dealing with horrific Flask / Django projects.

I think you misspelled "beautiful"

Installation in a regular system without Kubernetes? Right now I can install Grafana, Prometheus and Alertmanager in a regular Linux system using distribution packages, and just worry about those programs themselves. If I want to install OnCall, I need not only OnCall plus four other non-trivial dependencies that will still need configuration, management and troubleshooting. All for something that is going to deal with far less load than any of Grafana/Prometheus/Alertmanager. I honestly do not understand it.

Not OP, but one may interpret your response as "I don't understand why you prefer a single binary over this architecture that requires 6 different services and prefers k8s".

IMHO, OP just stated that one could solve this with less dependencies and have the same (if not a better) result.

For a simple low-scale app you can often do without Redis and Celery/RMQ if you just push everything into Postgres.

Far less scalable, but it is dramatically simpler to deploy. Often gets you surprisingly far though. Would be interesting to know how many monitored integrations could be supported by that flow.

Any of the following:

Python(Django)+Redis+[SQLite]

Python(Django)+Postgres

[Compiled Go binary]+[SQLite]

SQLite barely even counts as an architectural dependency TBH :)

Not sure why you include java in that, as you mostly get a standalone file. No such thing as a jre in modern java deployment.

As for python, at least getting a dockerfile helps a lot. Otherwise it's a huge mess to get running, yes.

Python is still a hassle anyways, since the lack of true multithreading means that you often need multiple deployments, which the Celery usage here for instance shows.

Hell no, I want stuff like OnCall packaged into Linux distribution. I need something stable and reliable and that receive security fixes.

Maintaining tenths of binaries pulled from random github projects over the years is a nightmare.

(Not to mention all the issues around supply chain management, licensing issues, homecalling and so on)

> Docker Compose takes about 5 minutes and you don't have to think about it again

Except when you need to pin and repin versions to comply with a security policy, which may be why you aren't even running containers in the first place

Don't get me wrong, I love Django and think its a great framework for writing internal tools like this. Redis gets a pass too since Django has native support for it in 4.0+. It's really the (IMHO unnecessary) combo of MySQL+RabbitMQ+Celery that turns me off.

Redis itself has had solid support for building reliable distributed task streaming for nearly 4 years (Redis ConsumerGroups introduced in 2018).

Grafana used to be so simple, I don't know if I'm a fan of this direction towards many services.

Having to run alertmanager and configure it in addition to Grafana was bad enough, now you need to run and configure another service if you want some extra functionalities for those alerts? Are they going to keep maintaining acknowledgements and scheduled silence in AlertManager now that OnCall exists? Are we going to have "legacy notifications" in AlertManager when not running OnCall, the same way there are "legacy alerts" in Grafana when updating from Grafana 7 (pre-AlertManager)?

AlertManager does not do the evaluations, it does not connect to any metrics database; those are done by Prometheus/etc and forwarded to AlertManager, which handles deduplication and routing among other things.

Honestly I haven't tried that much, but didn't find anything in the docs so I assumed it wasn't a prioritized area.

In our particular scenario, we'd probably want to run Loki + Grafana locally, and then hosted Prometheus + hosted Grafana for metrics.

But would be great if we could just tell the two about each other, and under which domains they exist. That way, Prometheus-grafana could construct URLs that linked straight into Loki-grafana (that we host) for e.g. the same interval, or the same label filter (GET params).

But it would only work if I (the end-user) had access to both. That way, we don't have to expose Loki to the internet. But linking would still work.

There are quite a lot of services that does this with Github and commits. You can link from e.g. Bugsnag to Github by only telling Bugsnag your org and repo names. But Bugsnag won't have read access to Github (they also have another integration method which does require access, but that's not the one I'm talking about here).

Those types of "linking into a known URL pattern of another service" integrations are easy to setup and very easy to secure.

GPL and its variants are a no go where I work.