Alertmanager routing tree doesn't respect the active time interval (?)

Hi devs,
I'm trying to use the recently added time_intervals and active_time_intervals features to enable an alert to go to our pager only during working hours. I'm defining my time interval and corresponding route as follows:
    time_intervals:
      - name: workinghours
        time_intervals:
          - times:
              - start_time: 08:00
                end_time: 18:00
            weekdays:
              - monday
              - tuesday
              - wednesday
              - thursday
              - friday
...
    route:
      routes:
        - receiver: oncall-pager
          matchers:
            - severity="workinghours"
          active_time_intervals:
            - workinghours

Furthermore, I have a prometheus rule with label severity="workinghours" defined. With this configuration, I would expect my prometheus rule to only be active during working hours, i.e., Monday to Friday from 8am until 6pm. However, for some reason my prometheus rule also fires and moreover gets routed to our pager outside of the hours 8am until 6pm. That is, I get paged even when I shouldn't get paged according to the above-defined time interval workinghours.
Did I do something wrong here in the configuration? Or might this be a problem with the K8s prometheus-operator (or the kube-prometheus-stack helm chart) through which I'm using this new alertmanager feature?
At the moment, I'm using the following versions:
alertmanager: v0.24.0
prometheus-operator: v0.62.0
kube-prometheus-stack helm chart: 44.4.1
Originally posted by @bollmann in https://github.com/prometheus/alertmanager/issues/2779#issuecomment-1427501988

Have you checked the final configuration generated by the operator? I've tried a similar example on my local machine and it works. I'd enable --log.level=debug to see if anything pops up from the logs.

I'm experiencing a similar issue using the following config
global:
  resolve_timeout: 5m

route:
  receiver: default
  group_by:
  - alertname

  routes:

  - receiver: DevOutOfHours
    matchers:
    - namespace=~".+dev|.+uat"
    active_time_intervals:
    - outofhours
    - weekends
    mute_time_intervals:
    - officehours

  - receiver: TeamA
    matchers:
    - label_team="TeamA"

  - receiver: TeamB
    matchers:
    - label_team="TeamB"

receivers:
- name: DevOutOfHours
- name: default
- name: TeamA
- name: TeamB

time_intervals:
- name: officehours
  time_intervals:
  - weekdays: ['monday:friday']
  - times:
    - start_time: "08:00"
      end_time: "20:00"
    location: Europe/London
- name: outofhours
  time_intervals:
  - weekdays: ['monday:friday']
  - times:
    - start_time: "00:00"
      end_time: "08:00"
    - start_time: "20:00"
      end_time: "23:59"
    location: Europe/London

- name: weekends
  time_intervals:
  - weekdays: [saturday, sunday]
    location: Europe/London

I've tried copying and pasting into https://prometheus.io/webtools/alerting/routing-tree-editor/ and using { namespace="app-dev", label_team="TeamA"} as the test labal set, but my alerts keep routing to DevOutOfHours instead of TeamA during office hours.
I only want the dev and uat namespace alerts to be notified during working hours, not out of hours and weekends

It looks to me like the matching is working fine. Matching is done top-down, so the DevOutOfHours route is matching first. Perhaps give the configuration doc a re-read to make sure you are familiar with the routing and configuration option. It seems like you may be confused about what active_time_intervals and mute_time_intervals do (I believe you have them inverted).
To me, it seems like you would want to set up sub-routes for the teams routes. My general structure for each team's routing looks like this: I have the default route that matches only team name, then I have sub-routes that match more specific alerts, etc. In your case, this would look like:
- receiver: TeamA
    matchers:
    - label_team="TeamA"
    routes:
      - receiver: TeamA
      matchers:
      - namespace=~".+dev|.+uat"
    mute_time_intervals:
      - outofhours
      - weekends
    active_time_intervals:
      - officehours

This matches any label_team="TeamA", then if namespace=~".+dev|.+uat" alertmanager will match the sub-route and fire under those times specified. And under this model, you can completely remove the DevOutOfHours route.
Let me know if that all makes sense, and solves your problem!

thanks for the reply, I'll give it a go.
I was using the example found here https://prometheus.io/docs/alerting/latest/configuration/#example
# All alerts with the service=inhouse-service label match this sub-route
    # the route will be active only during offhours and holidays time intervals.
  - receiver: 'on-call-pager'
    matchers:
      - service="inhouse-service"
    active_time_intervals:
      - offhours
      - holidays

I would expect my DevOutOfHours route only to be active out of hours and move on to the next route during office hours. Maybe I've understood it wrong.

just saw this bit Additionally, the root node cannot have any active times.
let me try it with different subroutes

not having much luck. Shouldn't the below just route to the default receiver? It seems like the time_intervals aren't having any impact. Running it using { namespace="app-dev", label_team="TeamA"} routes to TeamA when i assume it should only do during the weekends. I've tried adding an additional route within the receiver, but it didn't like it.

global:
  resolve_timeout: 5m

route:
  receiver: default
  group_by:
  - alertname

  routes:

  - receiver: TeamA
    matchers:
    - label_team="TeamA"
    active_time_intervals:
      - weekends

  - receiver: TeamB
    matchers:
    - label_team="TeamB"
    active_time_intervals:
      - weekends

receivers:
- name: default
- name: TeamA
- name: TeamB

time_intervals:
- name: officehours
  time_intervals:
  - weekdays: ['monday:friday']
  - times:
    - start_time: "08:00"
      end_time: "20:00"
    location: Europe/London
- name: outofhours
  time_intervals:
  - weekdays: ['monday:friday']
  - times:
    - start_time: "00:00"
      end_time: "08:00"
    - start_time: "20:00"
      end_time: "23:59"
    location: Europe/London

- name: weekends
  time_intervals:
  - weekdays: [saturday, sunday]
    location: Europe/London

It's possible there is a formatting issue with the time_intervals. Yours do look different than my own. I believe the routes look correct.
I would check using Amtool and the online routing tree editor to confirm. I find this tool extremely useful: https://www.prometheus.io/webtools/alerting/routing-tree-editor/
Here's an example of one of my own time intervals:
time_intervals:
- name: interval_1
  time_intervals:
  - times:
    - start_time: '02:45'
      end_time: '23:45'
    weekdays: ['monday:friday']
    location: 'America/Chicago'

I think the subtle difference between the config of @cbryant42  and @dtwilliamsWork is a '-'.
Take for example (A):
time_intervals:
- name: officehours
  time_intervals:
  - weekdays: ['monday:friday']
  - times:
    - start_time: "08:00"
      end_time: "20:00"
    location: Europe/London

And (B):
time_intervals:
- name: officehours
  time_intervals:
  - weekdays: ['monday:friday']
    times:
    - start_time: "08:00"
      end_time: "20:00"
    location: Europe/London

In (A) there are two time interval definitions.

The weekdays Monday through Friday.
The time ranges from 8 o'clock till 20 o'clock.

In (B) there is one time interval defined, the weekdays Monday through Friday and the time range between 8 o'clock and 20 o'clock.
I think the alert manager uses multiple time intervals as a logical OR rather than an AND.
I had the same problem and noticed this subtle difference. I still need to verify that my hunch is correct though ;)

I struggled with this as well, something that was unclear to me, but the doc states something important for active_time_intervals (and similar for mute_time_intervals):
The route will send notifications only when active, but otherwise acts normally (including ending the route-matching process if the `continue` option is not set).

I was thinking that using active_time_intervals would ignore entirely the route outside the interval, but it only mutes the notifications, the route is still active at all time and will show up in the alertmanager UI when the alerts triggers.
Only the notifications will be paused outside the time interval (or during the interval with mute_time_intervals)
Hope that clarifies something for some people like me.


Removing dependencies that are seemingly not needed to compile

Mainly making this CR to test the integration suite to see if this compiles without all of the headers listed. Not sure why its still able to compile for me.

Interesting builds failed on windows but passed on the others


Add VCS-based sidebar label coloring

Essentially a copy-paste from Toxin, thanks @p3lim! One divination is that I chose to decolor the VCS badges, instead of outright hiding them, since they help distinguish created/modified from staged. Another is the italic for ignored files, which I wouldn't mind dropping, but it's somewhat useful.
Didn't know what colors to choose, so just looked at VS Code's Base16 Themes.
This is opt-in, spacegray_sidebar_vcs_label setting enables the whole thing, spacegray_sidebar_vcs_expanded also applies the coloring to expanded folders. ~The latter one is the same setting name as in Toxin, for potential interop.~
Added variables introduced in 3179, because why not. Opted not to indent the rules as to not ruin the diff for the main feature and to not pollute the blame trail. base prefix because reasons.
Also piggybacked in a fix for #188, since the original PR has died a slow and painful death.
Fixes #214
Fixes #188

Thanks for the contribution! Will test this locally and review the changes!

@marcofugaro Took a closer look at the other themes, besides Eighties, and yeah, those needed tuning, especially Light. Also got rid of the selected effect, doesn't look good, and isn't really necessary.

Rebased to fix a conflict in readme.

@marcofugaro

@kkga @marcofugaro Hi! I'm willing to adopt this theme. Could fork, but I don't intend to change much, just modernize.

@alecmev hey! Thanks for your interest. Do you want to move this repo to community packages org?

@kkga Yeah! I haven't coordinated with them yet though. I think you need to be a part of an organization to be able to transfer to it. You can transfer to me, and I can take of the rest, what do you prefer?

@alecmev any idea what this means?

Ah, I suspect that is because of an already existing fork. I didn't think of that.
You may also transfer the repo to me and I will take care of it.

Ah, I suspect that is because of an already existing fork. I didn't think of that.

Indeed! I renamed the fork to spacegray-fork ahead of it, but that didn't help.

You may also transfer the repo to me and I will take care of transfering it to the org.

I already submitted a "detach a fork" request to GitHub, but this is faster, thanks for offering! @kkga Can you transfer to @FichteFoll instead, please?

@kkga The fork has been detached, just in case, so you can transfer to me too :wink:

Indeed, all done on my part.

Great, thanks all!

Thank you! :+1:

Is there any progress?

Didn't realize there was demand, will try to clean up and release soon. This isn't ready as-is yet, unfortunately, as I've noticed some inconsistencies between dark/light in personal use.

Thanks for sticking to it. I have released 1.4.0 with these changes.
I'm also adding you as a collaborator since you obviously still care about this theme and have shown interest in improving it (so I don't have to 😉 ).

Thanks for taking care of this!


Advance group by

Hello! I have a hard sql query. This is part of sql
group by (cast(height / $delegates as integer) + (case when height % $delegates > 0 then 1 else 0 end)) having height <= $height
I made something like that:
group: {
expression: "(cast(height / $delegates as integer) + (case when height % $delegates > 0 then 1 else 0 end))",
having: {
height: {$lte: height}
}
}
Can you add this feature, I can´t make this.

Thanks for your suggestion. I will implement this functionality soon.

Also to add to this, mostly the group by queries are grouping of fields or field calculations itself, so will be a good idea to include a group key and boolean value in the field itself?
Adding to this there can also be a group key on its own, something like:
const sql = jsonSQL.build({
    type: 'select',
    table: 'table',
    fields: [
        {
            func: {
                name: 'year',
                args: ['column1']
            },
            grouped: true
        },
    ],
    group: ['column2']
});

@artzhookov did you get chance to implement the aggregations ?