[Discussion] Enhanced abstraction of "file" concept in libsinsp filter/display classes for effective monitoring

Motivation

fd.name: if the fd.type represents a file or directory, the fd.name field contains the full path. If the path is not already an absolute path, a custom traversal parser can be used to derive the absolute path based on the cwd and the given arg.
fd.nameraw was introduced to expose the arg as given to detect path traversal attacks like ../../../etc/passwd
fd.name.symlink_resolved or similar name (also applicable for executable path) is discussed in https://github.com/falcosecurity/libs/issues/1111

Syscalls in Linux often reference files or file names using custom arguments. This can make rule writing and data analysis cumbersome, as it requires remembering the relevant file paths args for a given syscall. Many users are also unfamiliar with the evt.arg.* fields and their meaning. Enhancing our documentation can address these issues by providing clear explanations and references to the concept of files in Linux. This will improve understanding and facilitate effective rule analysis.
At the same time, enhancing the user experience by overloading the fd.name field would be beneficial.
Feature
We can introduce more "overloading" instead of creating new field classes, as suggested in https://github.com/falcosecurity/libs/pull/1060. This approach promotes code reusability, simplifies system maintenance, and makes ETLs simpler, along with writing downstream rules.
Besides definitely embracing the reality that the concept of files and their attributes in Linux can be intricate ... 😅 same applies to the syscall args naming conventions in the Linux kernel ...
Reading below, it seems that keeping the fd. field class is the ideal choice for encapsulating file-related attributes.
In the Linux kernel, a file is an abstract representation of a resource that can be accessed or manipulated. It can represent various types of data, including regular files, directories, symbolic links, device files, and more.

A file descriptor is a unique identifier or reference to an open file in a process.

On the other hand, an inode (index node) is a data structure in a file system that stores metadata about a file.

A file descriptor number (or simply fd number) is a unique identifier assigned by the operating system to an open file in a process. It is an integer value that represents the file within that process.

An inode number is a unique identifier assigned to a file's metadata within a file system, uniquely identifying the file within that file system.

...

Current State + proposal of "overloading" fd.name even more:

syscalls
syscall arg
fcomments and suggestions for ilter field class.name abstraction in libsinsp

[open, openat, openat2, unlinkat, umount, umount2, access]
evt.arg.name
already mapped to fd.name, a mix of file or directory fd.type -> also fd.nameraw is available, however double check for each syscall

[chroot, mkdir, rmdir, unlink, mkdirat, open_by_handle_at]
evt.arg.path
overload each to fd.name

[linkat, renameat, renameat2]
evt.arg.oldpath
fd.name.source and overload to fd.name

[linkat, renameat, renameat2]
evt.arg.newpath
fd.name.target

[linkat, renameat, renameat2]
evt.arg.olddir
fd.name.source

[linkat, renameat, renameat2]
evt.arg.newdir
fd.name.target

[symlink, symlinkat]
evt.arg.linkpath
fd.name.target

[symlink, symlinkat]
evt.arg.target
fd.name.source and overload to fd.name (this is the file we create a symlink over)

[quotactl]
evt.arg.quotafilepath
overload to fd.name

[fchownat]
evt.arg.pathname
overload to fd.name

[chmod, fchmodat, fchmod]
evt.arg.filename and evt.arg.fd as fchmod has no filename
fd.filename, also check on fd.name and fd.nameraw behavior here to stay consistent

[connect, accept* etc]
various
network related fd.name is overloaded with reference indicating src->dst ip tuples

[socketpair]
evt.arg.source
fd.name should be ?

[socketpair]
evt.arg.peer
fd.name should be ?

I think I'd avoid using fd.* to represent the targets of all the syscalls that can act on file paths. Currently, fd.* fields are only valid for syscalls that create a fd, and I've used that distinction when trying to explain to users why there isn't a "fd" for syscalls like renameat2, mkdir, etc. Usually the users are asking why they can't "look up" the mkdir, etc for the process afterwards, and noting the distinction between fds and paths was useful.
I understand the appeal of just building out below the fd. namespace, but I think keeping the fd. names limited to file descriptors is a good one.

Here's a recent discussion on slack where we pointed out  out that fd.* fields only make sense for syscalls on a fd: https://kubernetes.slack.com/archives/CMWH3EH32/p1685803022045059
(This isn't exactly the same, as the user was talking about spawned processes, but it's pretty close).

@incertum What's your opinion? Are there blockers for #1060 on your end?

what would help from my perspective is to have one field that is always populated with the "file path" on disk I would want to write my detection against, so that I don't have to remember which one the one is I have to care about for each unique system call, ...

That's exactly the goal here--to have a set of consistent fields so they can be used for a variety of file based syscalls. file.path is for any syscall that acts on a single path (open, mkdir, unlink, etc), and file.source/file.target is for any syscall that works on two paths (rename, symlink, etc). And file.path is populated for open/openat/openat2, etc: https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R187. (In case the direct link gets changed from a rebase, look at sinsp_filter_check_file::create_fd_checks().

If file.path is the new field for having everything also fd* in case of fd.type being a file why is it then not ok to go the reverse direction and use existing fd field class instead.  In both scenarios we are blurring the boundaries of "Linux concepts". In addition file field class shall then also actually represent dir? Would we also add a file.type field which would indicate if it is a directory or a file (further replicating the fd* setup)?
Asking again because once we introduce a new field class, The Falco Project supports it, including sending clear messaging to adopters.
If we start with adding a file class fields, will we tear fd field class further apart in the future for other cases?
What about fd.nameraw? Is there going to be an unsanitized file.pathraw version?

file.path is for any syscall that acts on a single path (open, mkdir, unlink, etc), and file.source/file.target is for any syscall that works on two paths (rename, symlink, etc)

Appreciate your additional input, but I have some reservations regarding the benefits and scope in this particular context. Referencing the table I created above, see https://github.com/falcosecurity/libs/issues/1134#issue-1738790659

cases like [linkat, renameat, renameat2] naturally map from olddir, newdir or oldpath, newpath to source and target [I don't see olddir, newdir addressed in the PR as of today and your above statement "mkdir" suggests the intention to throughout support path also for dirs?]
but cases like [symlink, symlinkat] are tricky as evt.arg.linkpath should be target but there exists evt.arg.target which should be the source rather or the reverse? [I don't see them addressed in the PR as of today]
those ones are tricky too [chmod, fchmodat, fchmod] or [fchownat], as it naturally mixes file and fd anyways, evt.arg.filename and evt.arg.fd as fchmod has no filename, for fchmod we would also populate fd.name to stay consistent? [I don't see them addressed in the PR as of today]
will file.path also be populated with proc.exepath?

What about overloading file.path or the field we agree on with the source or target that is typically of interest for writing rules? That in my opinion would improve usability, but if everyone says it's a terrible idea, no feelings hurt.
Thank for working on this Mark!

If file.path is the new field for having everything also fd* in case of fd.type being a file why is it then not ok to go the reverse direction and use existing fd field class instead.
To me, file.path is a superset of fd.name--it will work for any syscall that creates fds (that is open/openat/openat_2/openat2), but also work for syscalls that act on filesystem paths. We won't be removing the fd.name field, it's just that the same information will be available in file.path. file.path would also represent a directory, in cases where the syscall acts on a directory. mkdir is an example. It takes a path argument, and file.path will contain the path argument. (But it doesn't create a fd of course).

In both scenarios we are blurring the boundaries of "Linux concepts".
I think using file.path actually helps keep the concepts different. To me, the name file.path can be used for any syscall that takes a path argument, and fd.* can continue to be used for syscalls that work with fds.

If we start with adding a file class fields, will we tear fd field class further apart in the future for other cases?
I was only planning on adding the 3 fields file.path, file.source, and file.target for now. I just wanted to cover the use case of having a single predictable set of fields that can be used for a variety of file based syscalls.

What about fd.nameraw? Is there going to be an unsanitized file.pathraw version?
For all of file.path/file.source/file.target, the path will be fully resolved, adding the thread cwd when necessary. I guess we could have a file.pathraw if we really want it for consistency, but I found that the raw paths aren't very useful for creating falco rules because you can't really do matching based on them, as the thread cwd could comprise any prefix of the path.

About syscall support:

linkat: I believe file.source/file.target are supported for linkat here: https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R299 and https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R317. The values need to be retrieved from the enter event so they don't have a simple field alias in sinsp_filter_check_file::create_fd_checks().
renameat/renameat2: I have aliases for source/target here: https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R203 and https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R207
mkdir: I support file.path here: https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R164 and a few lines below for the other mkdir variants.
symlink/symlinkat: should be supported here: https://github.com/falcosecurity/libs/pull/1060/files#diff-70b8039f1cd1e5664df22d0b231919b7b042dfdf223134fb8356a4b7f4d0da02R223 and a few lines below for symlinkat.
chmod/fchmodat/fchmod/fchownat: I didn't include any of these simply because my use case didn't require them. I was prepared to add support for them later if needed. If you'd like support for them now I can add them.

Will file.path also be populated with proc.exepath?
I wouldn't include proc.exepath as that's about the path used for an exec and not any of these file operations.

What about overloading file.path or the field we agree on with the source or target that is typically of interest for writing rules?
I'm not sure I understand this comment. For the use case I was working on, depending on the syscall, I needed to look at the source in some cases and in the target in others. I don't think we would want to combine source and target together into a single field because you don't always know whether a rule author is interested in the source or the target.

If it helps I can schedule an in-person meeting or we can discuss it during one of the upcoming falco meetings as well.
Thank you for the detailed feedback! I want to make a change that is well-thought out that everyone can agree on as well.

Why not choose something like path.* field class then?

I don't mind changing file.* to path.* if that's the consensus but to me file.* makes more sense as a prefix as it refers to filesystem paths. I could think of other paths that wouldn't be applicable here. proc.exepath is one example. Maybe fspath for filesystem path?
I don't mind adding a field to distinguish between files and directories. But I don't think having fd as a type would make sense as a path for an open would rightly belong in both the file and fd categories. Also there will be many fds (network connections, shared memory regions) that would not be mapped to a filesystem path. If you want to distinguish between directories and files we could add a boolean file.isdir.
I can also add .nameraw, although I think it will be difficult to build rules based on it as depending on the syscall, in some cases you will have fully resolved paths and for other syscalls you will not.
I will add support for all the chmod/chown variants and quotactl (for now continuing to use file.*).
You're right about how symlink maps source/target compared to link, they are currently inconsistent. Here's the current mappings:

link/linkat (events LINK, LINKAT, LINK_2, LINKAT_2): man pages use terms oldpath/newpath. oldpath is currently mapped to file.source, newpath is currently mapped to file.target.
symlink/symlinkat (events SYMLINK, SYMLINKAT): man pages use terms linkpath/target. linkpath is currently mapped to file.source, target is currently mapped to file.target.

Here's the sysdig output for ln /tmp/origfile /tmp/hardlink (e.g. create a hard link /tmp/hardlink that points to /tmp/origfile) and ln -s /tmp/origfile /tmp/softlink (e.g. create a soft link /tmp/softlink that points to /tmp/origfile):
# sudo sysdig "evt.type in (link, linkat, symlink, symlinkat)"
72021 10:02:31.235151471 5 ln<PHONE_NUMBER>14) > linkat
72022 10:02:31.235183585 5 ln<PHONE_NUMBER>14) < linkat res=0 olddir=-100(AT_FDCWD) oldpath=/tmp/origfile newdir=-100(AT_FDCWD) newpath=/tmp/hardlink flags=0
104584 10:02:37.127153735 5 ln<PHONE_NUMBER>81) > symlinkat
104585 10:02:37.127193844 5 ln<PHONE_NUMBER>81) < symlinkat res=0 target=/tmp/origfile linkdirfd=-100(AT_FDCWD) linkpath=/tmp/softlink

So to keep things consistent and also vaguely aligned with the man pages, I'll change the link/linkat variants to map oldpath to .target and newpath to .source.
And I will definitely add documentation for how these new fields will work. It will be a standalone page and referred to from https://falco.org/docs/reference/rules/supported-fields/.
I'll comment again (here I guess) once I've made the above changes.

There's a problem with supporting file.isdir (or in general differentiating files and directories) for rename/renameat/renameat2. These can be used for either files or directories and there isn't anything in the event that can be used to distinguish between files/directories. You can't just do a stat() when coming up with a value for the field either, as that wouldn't work for capture files where the original filesystem is not available.
Given this, I think it's better to not include a file.isdir field (or determining the type of the path) at all, unless there are really compelling reasons for it.

fspath.* is something I could live with, and not intending to keep poking the bear, but initially it was argued that for the open* syscalls it is important to distinguish that we are dealing with fds and that is why a new field class was needed specifically for syscalls around file manipulations when they are not fds, but now it's not anymore important and we even include directories (here I have my peace saying it once more).
Adding .nameraw would be appreciated for consistency and that's what it is we want to know the raw arg for data modeling, and to reiterate for some adopters it is valuable.
Re the symlinks/hardlinks thanks for checking and yes staying closer to the manpages is probably better, although personally I am not sure why they choose these conventions in the symlink case ...
Thanks for planning a detailed docs page!
Understood re the edge cases around dir/file distinction (thanks also for checking), arguably the syscall tells you if it's a dir I suppose ...
In summary, we could live with the following? @falcosecurity/libs-maintainers any objections or other suggestions? fspath.* or simply fs.* should be ok would say, and I would appreciate broader consensus. Thanks!
fspath.name
fspath.nameraw
fspath.source
fspath.target

@mstemm one more request since we are aligning more and have clarity on including dir, we shall also include umount* syscalls etc?

Sounds good to me! I'll start making the changes. If we're going to support .nameraw I think we also want .sourceraw/.targetraw so I will support those as well.
And yes I can support umount. Seems like I should support mount as well then. source will be the path to the device and target will be where the device is mounted. This matches the man page.

I finished all the changes above. fspath.name isn't working for fchmod/fchown as they work on fds and not paths. I should be able to fix that, but I'm going to make the change in a separate PR which I'm working on now.

Great just left some comments in the PR, I think we can finish the review in the PR and unless someone has objections we can consider the discussion here resolved.
For above plz add code comments indicating it's a todo to be fixed, ok from my end to do it in a new PR of course.

Thanks to you both for the great discussion. I think both fspath.* and fs.* would make a good UX and satisfy the use cases involved. My vote would go for fs.* because it's less verbose and in the same style line as fs.*, but I'm not gonna fight for it. I'm gonna review Mark's PR.

@jasondellaluce would we then go with fs.path instead of fs.name? What about renaming the other ones? Just so it can be changed correctly in one clean commit. Thank you!

@jasondellaluce would we then go with fs.path instead of fs.name? What about renaming the other ones? Just so it can be changed correctly in one clean commit. Thank you!

I like fs.path! But then it will be dependent on whether everyone is onboard with using fs.*.


Explore the benefits of fuzzing for testing

Incorporating a fuzzing framework into the testing workflow might help reveal parser bugs or all kinds of unwanted behavior, not only for PGPainless, but also BouncyCastle.

A popular fuzzing framework appears to be Jazzer, so this might be a good candidate to look into.

Apparently Jazzer has been integrated into oss-fuzz?
Not sure which interface is easier to use?

Some thoughts from a discussion with the Sequoia Team:

You probably want to fuzz the packet parser
For this you want to eliminate the crypto layer
Therefore you want to have a null-backend to swap the crypto layer with
The null-backends crypto operations are just the identity function
Now you can round-trip "encrypted" and "signed" packets without the crypto getting into the way of the fuzzer.


The password for node 'Lava Test' is invalid.

Hello good afternoon, I get the error I wrote:
Node Lava Test is not able to connect! - Reason: The password for node 'Lava Test' is invalid.
Can you help me?

and how active my dashboard (web dashboard) i create linux (ubuntu) server.

Hello good afternoon, I get the error I wrote:
Node Lava Test is not able to connect! - Reason: The password for node 'Lava Test' is invalid.
Can you help me?

The error provided are quite straightforward. It has been indicated that the password entered is invalid. Therefore, please ensure that you’re entering the correct password.

:)

and how active dashboard (web dashboard) i in a create linux (ubuntu) server.

I recommend starting by watching this video: https://youtu.be/5Y5GPO95uxE?si=jCGTxU2v0oM6KWMd
The setup process is identical, with one key difference: you need to modify the host IP from ‘<IP_ADDRESS>’ to ‘<IP_ADDRESS>’ in web app.py. This change will enable network-wide access to your website. If still extra help feel free to join our Discord support server for more support.


Problems setting GeoPoint

I was having problems with setting locations
geoFirestore.set('some_key', { coordinates: new firebase.firestore.GeoPoint(37.79, -122.41)})
giving: "Invalid GeoFire location '[object Object]': location must be an array"
using this instead solved it:
geoFirestore.set(userId, [42.497855, 27.470660])
Using:
<EMAIL_ADDRESS><EMAIL_ADDRESS>

You're looking at the docs for v2 😅 not 1.2.0. Try npm i geofirestore@next to try the version 2 beta (it's pretty much ready to launch, I just need to expand typing support for firebase-admin)


Fix build error for special characters

#63

Use ElementAccessExpression when accessing properties.
When doing destructuring assignment, wrap the key in a StringLiteral.
Normalizes characters that cannot be used in identifier.

Thank you for the PR! It looks like you've changed the line endings of the file though, could you please use the line endings defined in the .prettierrc.json file?

@rmagur1203 Are you still planning to make the edits required on this PR?

In prettierrc, the end of line is set to crlf, but in reality it seems to be set to lf.
Changed it to lf and modified it to merge into the dev branch.


InMemoryQueue using System.Threading.Channels

InMemoryQueue implemented using System.Threading.Channels API.

The tests probably failed because the concurrency limit isn't set high enough to reach the batch threshold.


Remove unnecessary casts within FileManager

As the title suggests there were a couple unnecessary casts typically between int types in FileManager.swift. These are fixed in this pr.

@swift-ci please test


Disable job batching by JobRescheduler

Hello. I am having the same behavior as in issue #248.
The phone is in idle mode. The previously scheduled job is triggered just on time and schedules the next job to run in an hour. But 5 seconds later this new job is being rescheduled to right now and run. Running Android 8.0.0 with library version 1.2.0-RC1. The job is exact.
As mentioned in issue 248, it happens because of batching jobs together to minimize wake ups.
Is it possible to make JobRescheduler not batch the jobs (to make jobs run at the scheduled time or later)?
Here is a log:
09-10 00:18:00.637 759-785/? I/ActivityManager: Start proc 24126:com.dshatz.delayedsender/u0a375 for broadcast com.dshatz.delayedsender/com.evernote.android.job.v14.PlatformAlarmReceiver
09-10 00:18:01.775 24126-24126/? V/FA: Cancelling job. JobID: -278187904
09-10 00:18:02.102 24126-24151/? D/JobRescheduleService: Reschedule service started
09-10 00:18:02.146 24126-24149/? D/PlatformAlarmServiceExact: Run job, request{id=5, tag=job_send_message$f6a42051-a57b-45fb-853a-a5ebfd929824, transient=false}, waited 01:00:00, delay 00:59:58
09-10 00:18:02.211 24126-24149/? I/JobExecutor: Executing request{id=5, tag=job_send_message$f6a42051-a57b-45fb-853a-a5ebfd929824, transient=false}, context PlatformAlarmServiceExact
09-10 00:18:02.707 24126-24153/? I/JobExecutor: Finished job{id=5, finished=true, result=SUCCESS, canceled=false, periodic=false, class=SendMessageJob, tag=job_send_message$f6a42051-a57b-45fb-853a-a5ebfd929824}
09-10 00:18:02.711 24126-24149/? D/PlatformAlarmServiceExact: Finished job, request{id=5, tag=job_send_message$f6a42051-a57b-45fb-853a-a5ebfd929824, transient=false} SUCCESS
09-10 00:18:04.005 24179-24216/? W/BugleDataModel: tried to cancel job 100 that can't be found. already finished?
09-10 00:18:04.874 24126-24126/? D/JobProxy14: Scheduled alarm, request{id=6, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d, transient=false}, delay 00:59:56, exact true, reschedule count 0
09-10 00:18:06.663 24179-24216/? W/BugleDataModel: tried to cancel job 100 that can't be found. already finished?
09-10 00:18:07.196 24126-24151/? I/JobManager: Found pending job request{id=6, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d, transient=false}, canceling
09-10 00:18:09.650 24126-24151/? D/JobRescheduleService: Reschedule 1 jobs of 1 jobs
09-10 00:18:09.663 24126-24379/? D/PlatformAlarmService: Run job, request{id=6, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d, transient=false}, waited 00:00:00, delay 00:00:00
09-10 00:18:09.688 24126-24379/? I/JobExecutor: Executing request{id=6, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d, transient=false}, context PlatformAlarmService
09-10 00:18:09.755 24126-24153/? I/JobExecutor: Finished job{id=6, finished=true, result=SUCCESS, canceled=false, periodic=false, class=SendMessageJob, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d}
09-10 00:18:09.758 24126-24379/? D/PlatformAlarmService: Finished job, request{id=6, tag=job_send_message$f9397928-1f78-49d8-92a0-dc0caa752c3d, transient=false} SUCCESS

No, this is not a problem of batching alarms. If you see the same behavior like in #248, then you don't need to create a separate issue. I'm closing this as duplicate.

No, this is not a problem of batching alarms.

Then why is the job rescheduled to before the desired time? The official documentation of AlarmManager`s set method you referenced in issue 248 says that:

Beginning in API 19, the trigger time passed to this method is treated as inexact: the alarm will not be delivered before this time, but may be deferred and delivered some time later.

Please give an explanation, or a link to where it can be found. Thanks.

It was a bug in this library, see c8f7a2fadbf397e54c17e0beb8af767d0e6f8c18

Thanks for the fix! And sorry for the duplicate :)


automerge not merging when I change the logo

[x] bug report
[ ] feature request
[ ] question about the decisions made in the repository
[ ] question about how to use this project
@leoluk
I am trying to get this logo update to go through its saying:
Not eligible for automerge (click details for more information)
found removed line: - "logoURI": "https://pbs.twimg.com/profile_images/1260288995323502598/9aAFR0pr_400x400.jpg",

Summary
here is the commit
https://github.com/solana-labs/token-list/pull/5411/commits/0202d7fda527097973536e33540d16adbd059146

Not eligible for automerge (click details for more information)
its saying I removed the logo the logo was swapped and I added one to the repo since using hard coded links doesnt work on some wallets

Not eligible for automerge (click details for more information)
its saying I removed the logo the logo was swapped and I added one to the repo since using hard coded links doesnt work on some wallets

anything on this guys ? its really annoying when you get a message saying logo removed and well it was not removed it was replaced I tried 25 times now to update a logo we might re-consider and move back to ERC-20 at this rate we ll never get this thing out please help

Sorry for your trouble, please link a PR!

I'm having the same issue. Is there a fix?

i believe the PR was https://github.com/solana-labs/token-list/pull/5033
bad PR that deleted another token in the process. closed


 This is useful in large projects when wanting to dig into how a vendor package works, since the sidebar jumps to the vendor package directory, so accessing other files in the package is quick and simple.
This is no longer the case- the sidebar does not focus on the current file, so instead I have to right click on the file's tab and select Reveal in Sidebar.
Intentional or not, I can't find any settings which change this behaviour back to how it was before.
Thanks for the awesome software :)
VS Code version: Code 1.31.0 (7c66f58312b48ed8ca4e387ebd9ffe9605332caa, 2019-02-06T08:51:24.856Z)
OS version: Linux x64 4.15.0-45-generic

System Info

Item
Value

CPUs
Intel(R) Core(TM) i7-8550U CPU @ 1.80GHz (8 x 2058)

GPU Status
2d_canvas: enabledchecker_imaging: disabled_offflash_3d: enabledflash_stage3d: enabledflash_stage3d_baseline: enabledgpu_compositing: enabledmultiple_raster_threads: enabled_onnative_gpu_memory_buffers: disabled_softwarerasterization: disabled_softwaresurface_synchronization: enabled_onvideo_decode: unavailable_offwebgl: enabledwebgl2: enabled

Load (avg)
1, 1, 1

Memory (System)
15.55GB (0.57GB free)

Process Argv
--unity-launch

Screen Reader
no

VM
0%

Extensions (26)

Extension
Author (truncated)
Version

vscode-intelephense-client
bme
0.8.8

htmltagwrap
bra
0.0.7

regex
chr
0.2.0

vscode-gitignore-syntax
dun
0.1.2

apacheconf-snippets
eim
1.2.0

php-debug
fel
1.12.6

php-intellisense
fel
2.3.10

auto-close-tag
for
0.5.6

gc-excelviewer
Gra
2.1.28

composer
ika
0.7.1

smarty
imp
0.3.0

twig-language-2
mbl
0.5.2

dotenv
mik
1.0.1

vscode-apache
mrm
1.2.0

vscode-duplicate
mrm
1.2.1

sublime-keybindings
ms-
4.0.0

debugger-for-chrome
msj
4.11.1

php-docblocker
nei
1.7.0

vetur
oct
0.15.0

vscode-docker
Pet
0.5.2

phpstorm-snippets
phi
1.1.1

phpdoc-comment-vscode-plugin
rex
1.1.0

vscode-icons
rob
8.1.0

code-settings-sync
Sha
3.2.4

open-in-browser
tec
2.0.0

html-css-class-completion
Zig
1.18.0

(1 theme extensions excluded)

This works for me.
Do you have explorer.autoReveal setting set?
Can you repro with vscode insiders? https://code.visualstudio.com/insiders/

Yes, explorer.autoReveal was the first thing i checked.
Haven't tried with the insider build.
Since my initial post, i've realised that the issue is intermittent. I'll try and work out the pattern and will let you know

Please let me know the exact reproducable steps since I can not repro it.

I cant seem to reproduce it at all now !
Only thing i've changed since posting is reboot my machine, so must have been an issue on my end.
Sorry to waste your time


terminus not found by fontconfig in 18.03

Issue description
With pkgs.terminus_font, a bitmap font, in fonts.fonts, as of 18.03 fc-match Terminus does not find the font, fc-cache -v |grep fonts/terminus returns existing cache is valid: 0 fonts, 0 dirs, and fontconfig-based programs formerly configured to use it switch to a fallback font.
In 17.09, the font is found correctly, and fc-cache -v |grep fonts/terminus returns existing cache is valid: 18 fonts, 0 dirs. Strangely, xlsfonts is able to find the font in both 17.09 and 18.03. fc-scan seems to successfully process the font files.
The contents of the font directory appear to be the same on both systems:
fonts.dir        ter-x12n.pcf.gz  ter-x16b.pcf.gz  ter-x18n.pcf.gz  ter-x22b.pcf.gz  ter-x24n.pcf.gz  ter-x32b.pcf.gz
fonts.scale      ter-x14b.pcf.gz  ter-x16n.pcf.gz  ter-x20b.pcf.gz  ter-x22n.pcf.gz  ter-x28b.pcf.gz  ter-x32n.pcf.gz
ter-x12b.pcf.gz  ter-x14n.pcf.gz  ter-x18b.pcf.gz  ter-x20n.pcf.gz  ter-x24b.pcf.gz  ter-x28n.pcf.gz

Steps to reproduce
Set fonts.fonts = [ pkgs.terminus_font ]; and attempt to use the font, e.g. with fc-match Terminus.
Technical details

system: "x86_64-linux"
host os: Linux 4.14.27, NixOS, 18.03pre131327.0e7c9b32817 (Impala)
multi-user?: yes
sandbox: no
version: nix-env (Nix) 2.0
channels(root): "nixos-18.03pre131327.0e7c9b32817"
channels(user): ""
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos/nixpkgs

I confirm that this issue reproduces for me also.

Interestingly, fc-match Terminus returns nothing, but sudo fc-match Terminus gives the correct font files. Same issue with pkgs.profont
Issue both in 17.09 and now in 18.03.

This still reproduces for me on latest 18.03. I also confirm that fc-match gives the correct result as root for some reason.

There's a fix!
Nuking ~/.cache/fontconfig fixed the issue for me. I'm guessing a GC following the update to 18.03 invalidated the path to [an intermediate reference to] terminus and fontconfig silently fell back on another font instead of reporting the error.
To prevent this issue from recurring in the future, fontconfig could be fixed to drop a cache that contains invalid data, or perhaps we could somehow force all user-specific caches to be dropped.

Thank you @Ralith, that works for me as well!

@Ralith The issue actually reproduces for me after some time -- something in the cache/fontconfig folder is being recreated.

That seems very strange; where could it be getting invalid/obsolete data from? Still working fine here; I take it you're certain you didn't perform any nix operations that might have affected fonts or fontconfig in the intervening time?

fc-cache -f might also be worth trying.

I just had this issue recur, nuked the cache to fix it, and then had it recur again in the space of a few hours during which I ran no nix operations whatsoever. It looks like this isn't (just) an issue with updates: something is mangling the cache.

Which desktop environment are you using? Maybe there's some connection with that.

I don't use a DE, just i3.

The plot thickens…
(same here)
As a hack, maybe try removing the contents of ~/.cache/fontconfig and then setting it read-only?

I have figured out that the fix works fine for me unless I restart chromium, which then leads to the problem again. I am not sure, how it can be related, but it perfectly reproduces.

I haven't been restarting chromium at all, but I do make much heavier use of it on the machine where the issue was recurring than on the machine where it doesn't seem to recur...

I use Firefox, but used chromium for a quick check yesterday and that brought back the issue.

Encountered this issue yesterday, with same cause and fix.


Roaring64Bitmap.forInRange correctness issues

SUMMARY

Added test cases for the issues described in #577
Fixed a signed byte comparison that should have been unsigned in the two shuttle implementations of the trie.

This fixes #577
Automated Checks

[x] I have run ./gradlew test and made sure that my PR does not break any unit test.
[x] I have run ./gradlew checkstyleMain or the equivalent and corrected the formatting warnings reported.

@lemire Can you take a look at the fix? It's pretty straight forward...whoever decided that JVM languages don't get unsigned numerical primitives just caused a lot of developer pain :D
Also CI is repeatedly failing to fetch dependencies, apparently because the server is overloaded. I wonder if we could cache these in Github actions, so we don't have to flood the jfrog server as much?

On my desk

@lemire jmh dependencies still don't seem to be available. Not sure how long we want to wait? There is also a newer version of the plugin available (https://github.com/melix/jmh-gradle-plugin), but it requires a gradle update apparently. I know nothing about gradle, so this is kinda beyond my expertise.

I can take a look later today

I will let @richardstartin take a look and if we can't make progress or can't look, I will investigate.
Do not be concerned, we will resolve this issue.

@larsk-db thanks for spending your time fixing this. I fixed the dependency problem in #579 so if you rebase this will build and can be merged.

@larsk-db : I think we just need you to sync with our main branch to verify that everything is ready.


chore(deps): update dependency @vue/test-utils to v2.4.6

This PR contains the following updates:

Package
Change
Age
Adoption
Passing
Confidence

@vue/test-utils
2.4.2 -> 2.4.6

Release Notes

vuejs/test-utils (@vue/test-utils)
v2.4.6
Compare Source
What's Changed

Fix/circular references in props cause maximum call stack size exceeded by @Evobaso-J in https://github.com/vuejs/test-utils/pull/2371
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2374
docs: setup the translation helper by @Jinjiang in https://github.com/vuejs/test-utils/pull/2373
chore: translate translation sync message in french by @cexbrayat in https://github.com/vuejs/test-utils/pull/2377
docs: synchronize the french docs by @cexbrayat in https://github.com/vuejs/test-utils/pull/2378
chore(deps): update dependency vite to v5.2.2 by @renovate in https://github.com/vuejs/test-utils/pull/2376
chore(deps): pin dependency vitepress-translation-helper to 0.1.3 by @renovate in https://github.com/vuejs/test-utils/pull/2379
chore(deps): update dependency typescript to v5.4.3 by @renovate in https://github.com/vuejs/test-utils/pull/2380
chore(deps): update dependency vitepress-translation-helper to v0.2.0 by @renovate in https://github.com/vuejs/test-utils/pull/2381
chore: update vitepress-translation-helper by @Jinjiang in https://github.com/vuejs/test-utils/pull/2382
chore(deps): update dependency vitepress to v1.0.0 by @renovate in https://github.com/vuejs/test-utils/pull/2383
chore(deps): update dependency vitepress-translation-helper to v0.2.1 by @renovate in https://github.com/vuejs/test-utils/pull/2384
fix: update attachTo type in MountingOptions interface by @taku-y-9308 in https://github.com/vuejs/test-utils/pull/2375
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2388
docs(api): fix typo in attachTo anchor tag within isVisible by @matusekma in https://github.com/vuejs/test-utils/pull/2351
change vm to always provide global property by @taku-y-9308 in https://github.com/vuejs/test-utils/pull/2386
chore(deps): update dependency rollup to v4.13.1 by @renovate in https://github.com/vuejs/test-utils/pull/2389
chore(deps): update dependency reflect-metadata to v0.2.2 by @renovate in https://github.com/vuejs/test-utils/pull/2391
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2393
chore(deps): update dependency vite to v5.2.8 by @renovate in https://github.com/vuejs/test-utils/pull/2396
docs: fix missing equal sign by @w2xi in https://github.com/vuejs/test-utils/pull/2398
fix: renderStubDefaultSlot with scoped slots by @cexbrayat in https://github.com/vuejs/test-utils/pull/2397
docs(api): fix missing chars by @w2xi in https://github.com/vuejs/test-utils/pull/2399
docs: use innerHTML in teleport cleanup by @brc-dd in https://github.com/vuejs/test-utils/pull/2403
feat: Added dynamic return for element getter by @nandi95 in https://github.com/vuejs/test-utils/pull/2406
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2407
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2408
doc(api): fix missing char by @w2xi in https://github.com/vuejs/test-utils/pull/2410
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2412
chore: use node v18 on netlify by @cexbrayat in https://github.com/vuejs/test-utils/pull/2416
fix(stubs): avoid warning on normalized props with Vue v3.4.22 by @cexbrayat in https://github.com/vuejs/test-utils/pull/2413
chore: use the packageManager field from package.json in github action by @cexbrayat in https://github.com/vuejs/test-utils/pull/2417
chore(deps): update pnpm to v9 by @renovate in https://github.com/vuejs/test-utils/pull/2415
chore(deps): update all non-major dependencies to v3.4.23 by @renovate in https://github.com/vuejs/test-utils/pull/2418
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2419
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2420
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2421
Update index.md to fix typo and clarify get vs find behavior by @KatWorkGit in https://github.com/vuejs/test-utils/pull/2422
fix: set global provides before running vue plugins by @danielroe in https://github.com/vuejs/test-utils/pull/2423
ci: add build on node v22 by @cexbrayat in https://github.com/vuejs/test-utils/pull/2424
chore(deps): update all non-major dependencies by @renovate in https://github.com/vuejs/test-utils/pull/2426
chore(deps): update dependency unplugin-vue-components to v0.27.0 by @renovate in https://github.com/vuejs/test-utils/pull/2427
chore(deps): update dependency @types/node to v20.12.8 by @renovate in https://github.com/vuejs/test-utils/pull/2429
Fix/issue 2319 throw first error thrown during mount by @taku-y-9308 in https://github.com/vuejs/test-utils/pull/2428

New Contributors

@Jinjiang made their first contribution in https://github.com/vuejs/test-utils/pull/2373
@taku-y-9308 made their first contribution in https://github.com/vuejs/test-utils/pull/2375
@matusekma made their first contribution in https://github.com/vuejs/test-utils/pull/2351
@w2xi made their first contribution in https://github.com/vuejs/test-utils/pull/2398
@brc-dd made their first contribution in https://github.com/vuejs/test-utils/pull/2403
@KatWorkGit made their first contribution in https://github.com/vuejs/test-utils/pull/2422

Full Changelog: https://github.com/vuejs/test-utils/compare/v2.4.5...v2.4.6
v2.4.5
Compare Source
What's Changed

feat: add teleport stub possibility, https://github.com/vuejs/test-ut… by @blizzardKv in https://github.com/vuejs/test-utils/pull/2336
fix(findComponent): Use correct vm for stubbed component with selector by @freakzlike in https://github.com/vuejs/test-utils/pull/2327
fix(teleport.md): fix missed "global" key, https://github.com/vuejs/t… by @blizzardKv in https://github.com/vuejs/test-utils/pull/2337
fix(vue-router.md): update highlights when mocking real router by @dmoyadev in https://github.com/vuejs/test-utils/pull/2340
fix(deps): update dependency vue-component-type-helpers to v2 by @renovate in https://github.com/vuejs/test-utils/pull/2358
fix: Allow access to nested computed values (fix #2196) by @Evobaso-J in https://github.com/vuejs/test-utils/pull/2356
docs: add mention of enableAutoDestroy replacement by @strokirk in https://github.com/vuejs/test-utils/pull/2344

New Contributors

@blizzardKv made their first contribution in https://github.com/vuejs/test-utils/pull/2336
@dmoyadev made their first contribution in https://github.com/vuejs/test-utils/pull/2340
@strokirk made their first contribution in https://github.com/vuejs/test-utils/pull/2344
@Evobaso-J made their first contribution in https://github.com/vuejs/test-utils/pull/2356
@qmonmert made their first contribution in https://github.com/vuejs/test-utils/pull/2364

Full Changelog: https://github.com/vuejs/test-utils/compare/v2.4.4...v2.4.5
v2.4.4
Compare Source
What's Changed

fix: ignore prototype methods when using setData on objects by @Haberkamp in https://github.com/vuejs/test-utils/pull/2265
fix: always load cjs bundle in node environment by @danielroe in https://github.com/vuejs/test-utils/pull/2269
fix: experimentalVmThreads is now pool=vmThreads by @cexbrayat in https://github.com/vuejs/test-utils/pull/2275
feat: respect devtools definition by @webfansplz in https://github.com/vuejs/test-utils/pull/2311

New Contributors

@ArtemTropanets made their first contribution in https://github.com/vuejs/test-utils/pull/2267
@Haberkamp made their first contribution in https://github.com/vuejs/test-utils/pull/2265
@danielroe made their first contribution in https://github.com/vuejs/test-utils/pull/2269
@webfansplz made their first contribution in https://github.com/vuejs/test-utils/pull/2311

Full Changelog: https://github.com/vuejs/test-utils/compare/v2.4.3...v2.4.4
v2.4.3
Compare Source
What's Changed

types(setProps): setProps to Partial props by @pikax in https://github.com/vuejs/test-utils/pull/2241

Full Changelog: https://github.com/vuejs/test-utils/compare/v2.4.2...v2.4.3

Configuration
📅 Schedule: Branch creation - At any time (no schedule defined), Automerge - "after 8am and before 4pm on tuesday" in timezone Etc/UTC.
🚦 Automerge: Disabled by config. Please merge this manually once you are satisfied.
♻ Rebasing: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.
🔕 Ignore: Close this PR and you won't be reminded about this update again.

This PR has been generated by Renovate Bot.

Renovate Ignore Notification
Because you closed this PR without merging, Renovate will ignore this update (2.4.6). You will get a PR once a newer version is released. To ignore this dependency forever, add it to the ignoreDeps array of your Renovate config.
If you accidentally closed this PR, or if you changed your mind: rename this PR to get a fresh replacement PR.


Must use the Raspbian lite image for the 0.7.0-dev branch on a Pi Zero + Explorer HAT (GUI version doesn't work)

Tl;dr: I installed this on a Pi Zero W (2018-04-18-raspbian-stretch) with the Explorer HAT. The loop log thows extensive errors with Minimed comms when I first tried to fire it up. I progressed a little closer to where I can get mmtune.py to work, but it's pretty busted and everything still throws errors. Nowhere near running a full loop.
The saga starts just after a fresh OS load:
sudo bash
curl -s https://raw.githubusercontent.com/openaps/oref0/dev/bin/openaps-install.sh > /tmp/openaps-install.sh && bash /tmp/openaps-install.sh

Hit Ctrl+C before running oref0-setup
cd ~/src/oref0 && git fetch && git checkout 0.7.0-dev && git pull && npm run global-install
No apparent errors.
oref0-setup --dir=/root/myopenaps --serial=xxxxxx --cgm=G5-upload --ns-host=https://xxxxx --api-secret=xxxxx --tty=/dev/spidev0.0 --enable=' autosens meal autotune ' --radio_locale='US' --btmac=xx:xx:xx:xx:xx:xx --max_iob=6.0 --current_basal_safety_multiplier=8 --max_daily_safety_multiplier=8
No apparent errors yet here either. After a reboot though, the following log lines come flowing out as though nothing is even being run between log lines:
Starting oref0-pump-loop at Sun 17 Jun 15:24:08 EDT 2018 with 8 second wait_for_silence:
Waiting up to 4 minutes for new BG: First loop: not waiting
Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 15:24:22 EDT 2018
Preflight fail. Retry 1 of preflight
Preflight fail. Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 15:24:25 EDT 2018
Retry 2 of preflight
Preflight fail. Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 15:24:27 EDT 2018
Retry 3 of preflight
Preflight fail. Couldn't preflight
oref0-pump-loop failed. pump_loop_completed more than 15m old; waiting for 39 s silence before mmtuning
Could not parse profile.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/settings/profile.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/settings/profile.json' }
Could not parse edison-battery.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/monitor/edison-battery.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/monitor/edison-battery.json' }
Could not parse suggested.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/enact/suggested.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/enact/suggested.json' }
Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 15:24:44 EDT 2018
Listening for 39 s silence before mmtuning: Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 15:24:44 EDT 2018
mmtune: Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
No wait required.
If pump and rig are close enough, this error usually self-resolves. Stand by for the next loop.
Unsuccessful oref0-pump-loop at Sun 17 Jun 15:24:54 EDT 2018

Running mmtune.py yeilds an error about how the spi_serial module isn't installed. So we jump through some hoops to install that and mraa with some hints from this page (even though it's Edison-based help): https://learn.sparkfun.com/tutorials/installing-libmraa-on-ubilinux-for-edison
sudo apt-get install git build-essential swig3.0 python-dev nodejs-dev cmake libjson-c-dev
git clone https://github.com/intel-iot-devkit/mraa.git
mkdir mraa/build && cd $_
cmake .. -DBUILDSWIGNODE=OFF
make
make install
ldconfig && ldconfig -p | grep mraa

Add export PYTHONPATH=$PYTHONPATH:$(dirname $(find /usr/local -name mraa.py)) to .bash_profile, then install spi_serial:
git clone https://github.com/EnhancedRadioDevices/spi_serial.git
cd spi_serial
python setup.py install

Now the Python module is there and works. Hooray!
python /usr/local/bin/mmtune.py --serial xxxxxx --port /dev/spidev0.0 --subg_rfspy --radio_locale US
{
    "scanDetails": [
        [
            "916.300",
            0,
            -99
        ],
        [
            "916.324",
            0,
            -99
        ],
        [
            "916.348",
            0,
            -99
        ],
        [
            "916.372",
            0,
            -99
        ],
        [
            "916.396",
            0,
            -99
        ],
        [
            "916.420",
            0,
            -99
        ],
        [
            "916.444",
            0,
            -99
        ],
        [
            "916.468",
            1,
            -94
        ],
        [
            "916.492",
            0,
            -99
        ],
        [
            "916.516",
            0,
            -99
        ],
        [
            "916.540",
            0,
            -99
        ],
        [
            "916.564",
            0,
            -99
        ],
        [
            "916.588",
            3,
            -88
        ],
        [
            "916.612",
            5,
            -70
        ],
        [
            "916.636",
            1,
            -93
        ],
        [
            "916.660",
            5,
            -67
        ],
        [
            "916.684",
            5,
            -67
        ],
        [
            "916.708",
            5,
            -68
        ],
        [
            "916.732",
            4,
            -78
        ],
        [
            "916.756",
            0,
            -99
        ],
        [
            "916.780",
            0,
            -99
        ],
        [
            "916.804",
            0,
            -99
        ],
        [
            "916.828",
            1,
            -94
        ],
        [
            "916.852",
            0,
            -99
        ],
        [
            "916.876",
            0,
            -99
        ]
    ],
    "setFreq": 916.684,
    "usedDefault": false
}

But we still have a problem with the rest of the loop (note that mmtune spits out the grep help message):
Starting oref0-pump-loop at Sun 17 Jun 16:01:06 EDT 2018 with 11 second wait_for_silence:
Waiting up to 4 minutes for new BG: First loop: not waiting
Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 16:01:07 EDT 2018
Preflight fail. Retry 1 of preflight
Preflight fail. Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 16:01:08 EDT 2018
Retry 2 of preflight
Preflight fail. Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 16:01:08 EDT 2018
Retry 3 of preflight
Preflight fail. Couldn't preflight
oref0-pump-loop failed. pump_loop_completed more than 15m old; waiting for 27 s silence before mmtuning
Could not parse profile.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/settings/profile.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/settings/profile.json' }
Could not parse edison-battery.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/monitor/edison-battery.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/monitor/edison-battery.json' }
Could not parse suggested.json:  { [Error: ENOENT: no such file or directory, open '/root/myopenaps/enact/suggested.json']
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: '/root/myopenaps/enact/suggested.json' }
Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 16:01:17 EDT 2018
Listening for 27 s silence before mmtuning: Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 16:01:18 EDT 2018
mmtune: Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
No wait required.
If pump and rig are close enough, this error usually self-resolves. Stand by for the next loop.
Unsuccessful oref0-pump-loop at Sun 17 Jun 16:01:27 EDT 2018

The python-based pump comms have mostly been replaced in 0.7.0, but I think the installer didn't do what it was supposed to. Did you run the oref0-setup.sh in interactive mode?

Yeah, there are a couple new setup parameters that 0.7.0 introduces, and some code that auto-detects the explorer HAT in the interactive installer.

OK, this is the recommended "run again with the same options" command line from interactive setup (close enough):
oref0-setup --dir=/root/myopenaps --serial=386591 --cgm=g5-upload --ns-host=http://nightscout.lagor.io --api-secret=THISISTHESECRET --tty=/dev/spidev0.0 --max_iob=6 --enable=' autotune ' --radio_locale='US' --btmac='CC:F3:A5:DA:AD:7B'
I noticed that the automated setup that I originally ran did detect the Explorer HAT and made mention of it, but I would expect the automated setup to perform the same way as the interactive setup with the same options selected.
I should add that when it didn't work initially I went to my go-to method of debugging comms issues, mmtune.py. I knew that parts of the comms were switching, but since the script was still there I expected it to work. Can we have a commit that removes all the obsolete scripts we don't expect to work or don't want people to use so we can see what breaks before this goes to dev?

mmeowlink isn't entirely obsolete (it's still needed for Enlite CGM), so we can't remove it yet. The SPI serial and MRAA dependencies do get installed when someone selects mdt as their cgm.
Can you try a fresh reinstall of Raspbian, followed by the branch switch & interactive setup?

New clean OS install, new branch switch, new interactive setup:
Continuing oref0-pump-loop at Sun 17 Jun 20:15:21 EDT 2018
Listening for 5 s silence before mmtuning: Listening: .No interfering pump comms detected from other rigs (this is a good thing!)
Continuing oref0-pump-loop at Sun 17 Jun 20:15:21 EDT 2018
mmtune: Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
Usage: grep [OPTION]... PATTERN [FILE]...
Try 'grep --help' for more information.
No wait required.
If pump and rig are close enough, this error usually self-resolves. Stand by for the next loop.
Unsuccessful oref0-pump-loop at Sun 17 Jun 20:15:48 EDT 2018

Let me know what you want me to do to work this out.

You can manually run mmtune the way that oref0-pump-loop now runs it by doing:
cd ~/myopenaps/
export MEDTRONIC_PUMP_ID=`grep serial pump.ini | tr -cd 0-9`
killall -g oref0-pump-loop
Go-mmtune

Those commands going by quickly is symptomatic of spi not being enabled (they will show up in the logs between the initial setup and the reboot). If that's still happening after reboot, check to see if dtparam=spi=on is set in /boot/config.txt?

@scottleibrand Thanks for the new mmtune method@
@cluckj That setting is present, the first thing I do after booting the OS to the GUI is connecting to wifi and using the raspi-config interface to set autologin, boot-to-console, and enable all the interfaces, then reboot.
I just rebooted with cron disabled and tried the new mmtune command:
2018/06/18 06:34:26 cannot connect to CC111x radio on /dev/spidev0.0
2018/06/18 06:34:26 write /sys/class/gpio/export: device or resource busy

What else can I try?

Sounds like you’re not following the docs if you’re using the Raspberry Pi GUI. Not sure if that matters, but you might try again with the lite (non-GUI) image.

@scottleibrand I'll give you a very confused round of applause for pointing out the lite image being the culprit - I don't know what isn't there in the full OS image that is there in lite, but doing everything above the same except for the OS version seems to have gotten me moving forward! Mind if I keep this issue open until I get a full loop going? I'd like to add xdrip-js with its different version of node to see if everything keeps working correctly.

Good to know.  If you or anyone else cares to try to figure out why the full GUI image won't work, we might be able to make it work.  Until then, we should probably do a docs PR to note that only the Lite image works for now.
I'm fine keeping the issue open, but please retitle it to reflect what we found.

Done! Thank you!

Quick PR in openaps-menu: https://github.com/openaps/openaps-menu/pull/15

Full loop works now, even after you upgrade the version of node with the directions from Logger (https://github.com/xdrip-js/Logger):
mkdir node && cd node
wget https://nodejs.org/dist/v8.10.0/node-v8.10.0-linux-armv6l.tar.xz
tar -xf node-v8.10.0-linux-armv6l.tar.xz
cd *6l
sudo cp -R * /usr/local/
# add /usr/local/bin to PATH env variable in .bash_profile
node -v
# should show 8.10.0 at this point

openaps-menu complains about being compiled by a different version of node (and breaks the loop) if you do the upgrade after oref0-setup. Is there a chance we can get 0.7.0 to install this version of node in time for your release window so Logger is just a git clone away from working?

Is this a fresh install or an upgrade? I think we install node 8 if there’s no node yet.

It was a fresh OS, then the OpenAPS setup script and oref0-setup, and then Logger was added after that. Logger didn't work until I did the upgrade steps listed above.

Which node version did you have before upgrading?

I have v4.8.2 on my latest 4/18 Raspbian install from a couple days ago.
(btw, you can run sudo npm install in ~/src/openaps-menu/ and it'll rebuild it with the new version of node.)

So should we do https://github.com/openaps/oref0/commit/9006a6d831095beddeb0774cdc074ce37573ec98 ?

That's the one! Once you do the update for Logger node -v reports v8.10.0. And yes, that's the command I ran when I saw the version mismatch errors in the loop log.

@scottleibrand Do you want to put https://github.com/openaps/oref0/commit/9006a6d831095beddeb0774cdc074ce37573ec98 into 0.7.0-dev and I'll try a clean install?

Want to just try the entire misc-stuff branch? We’re running it without issue and have a PR open for it “the post-refactor one”.

Sorry, switch to that branch and try it? Because if you merged it here I would update and try it tonight too.

I'm not the most git-oriented person, so I just caught that misc-stuff is rooted on 0.7.0-dev so it doesn't actually have to be merged here in order to test 0.7.0-dev plus those changes. I'll try to test that out later today.

Still no major issues, but after a couples days of watching the loop log next to Nightscout I've noticed some things where what the loop has for data isn't always consistent with what's being displayed in Nightscout. The time delay isn't very long of course, but when I look at Nightscout (and it's not in the middle of a running loop) I expect it to reflect the state of the loop at the end of its last execution:

The OpenAPS pill reports a last runtime of around 4 min even at the end of a loop (and around 9 min around when the loop starts again). It never seems to get below that. When "completed oref0-pump-loop" gets logged the time in the pill is at that 4 minute mark.

After a loop-initiated bolus the new boluses don't show up on Nightscout for at least one iteration of the loop. I think this is a regression - in a previous discussion a while ago (on Gitter?) the loop-initiated boluses with new IOB calculation were sent up to Nightscout nearly immediately.

I think this is related to the above item, but I've noticed that a complete loop will finish with "bolusing" being the status when it gives boluses (and as mentioned the bolus doesn't show up until the start of the next loop). Could we add a wait loop where if the status from the pump is bolusing it waits 5 - 10 seconds, polls again, and continues to poll until until the status is error or normal where it then continues? Or is this intended behavior? If I remember what you said from our previous discussion this might also involve another read of pump events to see the completed bolus and send it up to Nightscout.

There are power saving optimizations that avoid re-doing pumphistory pulls and IOB recalculations unless you’re on wall power. You may get more prompt uploads if you plug in the rig.

How are you determining whether it's plugged into power? Since I was home-bound today and yesterday I was plugged in the entire time with the battery pill consistently reporting > 95%. I don't know that I think power savings is more important than reporting accuracy.

I’d take a look at oref0-pump-loop.sh and compare it to your pump-loop.log output.

Thanks for the pointer, I think I found it! To correct the above I'd recommend the following changes:

In main (https://github.com/openaps/oref0/blob/f754421f8e0189e9877c2b15a97a2789e2eb3854/bin/oref0-pump-loop.sh#L79) remove the onbattery call. If this does what I think it does it would bring the Nightscout display current regardless of charging status and I think that's important.
In onbattery (https://github.com/openaps/oref0/blob/f754421f8e0189e9877c2b15a97a2789e2eb3854/bin/oref0-pump-loop.sh#L819) drop the threshold to 95% at least, 90% maybe to be safe?

The 90% threshold seems like a no-real-downside change: I’d be happy to merge a PR there. Not sure we want to turn off power savings entirely, though, given how power hungry the Pi is compared to the Edison.

Just filed https://github.com/openaps/oref0/pull/1050. Funny that an ARM-based board is more power hungry than an x86-based board! That's definitely not something I would have assumed, is that still true to comparing an underclocked Pi Zero to an Edison with standard settings?
I'm not proposing removing power savings altogether, although I haven't looked at where else onbattery calls could be sprinkled to achieve more. Would it help to alter refresh_profile to only call get_settings (which does a bunch of pump interaction) when given a certain time period? I can try and look for some other places to use onbattery but I really think getting accurate data to the user in near real-time is important. The inconsistency of having pump-loop.log differ from Nightscout presents doubt.

I think all the issues within this issue have been resolved. Please open a new issue if there is a related issue on the latest 0.7.0-dev release candidate (in October 2019).


modifier evaluation

#317

Codecov Report

Merging #318 into master will decrease coverage by 0.47%.
The diff coverage is 58.33%.

@@            Coverage Diff            @@
##           master    #318      +/-   ##
=========================================
- Coverage   53.87%   53.4%   -0.48%
=========================================
  Files          21      21
  Lines         464     470       +6
  Branches      102     105       +3
=========================================
+ Hits          250     251       +1
- Misses        186     189       +3
- Partials       28      30       +2

Impacted Files
Coverage Δ

...kages/reactabular-table/src/evaluate-formatters.js
62.5% <50%> (-37.5%)
:arrow_down:

...kages/reactabular-table/src/evaluate-transforms.js
80% <66.66%> (-20%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dbe5808...60f4b38. Read the comment docs.

Can you add tests for the new functionality? Good to go after that.


cinnamon.cinnamon-session: 6.2.0 -> 6.2.1

Automatic update generated by nixpkgs-update tools. This update was made based on information from https://repology.org/project/cinnamon-session/versions.
meta.description for cinnamon.cinnamon-session is: Cinnamon session manager
meta.homepage for cinnamon.cinnamon-session is: https://github.com/linuxmint/cinnamon-session
Updates performed

Version update

To inspect upstream changes

Compare changes on GitHub

Impact
Checks done

built on NixOS
The tests defined in passthru.tests, if any, passed
found 6.2.1 with grep in /nix/store/5w4r9njkl0gvlh794bsnxxbaycmws0h8-cinnamon-session-6.2.1
found 6.2.1 in filename of file in /nix/store/5w4r9njkl0gvlh794bsnxxbaycmws0h8-cinnamon-session-6.2.1

Rebuild report (if merged into master) (click to expand)

5 total rebuild path(s)

5 package rebuild(s)

First fifty rebuilds by attrpath

cinnamon.cinnamon-common
cinnamon.cinnamon-gsettings-overrides
cinnamon.cinnamon-screensaver
cinnamon.cinnamon-session

Instructions to test this update (click to expand)

Either download from Cachix:
nix-store -r /nix/store/5w4r9njkl0gvlh794bsnxxbaycmws0h8-cinnamon-session-6.2.1 \
  --option binary-caches 'https://cache.nixos.org/ https://nix-community.cachix.org/' \
  --option trusted-public-keys '
  nix-community.cachix.org-1:mB9FSh9qf2dCimDSUo8Zy7bkq5CX+/rkCWyvRCYg3Fs=
  cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY=
  '

(The Cachix cache is only trusted for this store-path realization.)
For the Cachix download to work, your user must be in the trusted-users list or you can use sudo since root is effectively trusted.
Or, build yourself:
nix-build -A cinnamon.cinnamon-session https://github.com/r-ryantm/nixpkgs/archive/835ffef9da154f89960edf28b591da8cdd6cf59e.tar.gz

Or:
nix build github:r-ryantm/nixpkgs/835ffef9da154f89960edf28b591da8cdd6cf59e#cinnamon.cinnamon-session

After you've downloaded or built it, look at the files and if there are any, run the binaries:
ls -la /nix/store/5w4r9njkl0gvlh794bsnxxbaycmws0h8-cinnamon-session-6.2.1
ls -la /nix/store/5w4r9njkl0gvlh794bsnxxbaycmws0h8-cinnamon-session-6.2.1/bin

Pre-merge build results
We have automatically built all packages that will get rebuilt due to
this change.
This gives evidence on whether the upgrade will break dependent packages.
Note sometimes packages show up as failed to build independent of the
change, simply because they are already broken on the target branch.
Result of nixpkgs-review run on x86_64-linux 1

4 packages built:

cinnamon.cinnamon-common
cinnamon.cinnamon-gsettings-overrides
cinnamon.cinnamon-screensaver
cinnamon.cinnamon-session

Maintainer pings
cc @bobby285271 @mkg20001 for testing.

@ofborg test cinnamon


semantic: var hoisting when temporary variable is injected nested block in transformer

for (;;) {
  var _ref; // symbol should be hoisted
}

https://github.com/oxc-project/oxc/blob/d8ac38f9f1e498ffed8754c871bcb9fa1680f4e9/crates/oxc_transformer/src/jsx/refresh.rs#L583-L592


Get JS kernel variable in C#/F#

This may be a silly question and the requested feature exists but I can only find an example where C# variable is read from JS.
How to read JS variable in C# kernel ?

Right now this isn't directly supported, though I can imagine a few hacky workarounds.
We're planning on adding commands and events for retrieving remote variable states, which should make this straightforward programmatically, and then we'll extend #!share to include cross-process sharing.

I can imagine some pretty crazy workarounds...
I think one thing that could work with the tools already in the box is to send a command from the C# kernel to the JavaScript kernel that sends a command back to the C# kernel. I might give this a try to see if it actually works...
Longer-term, there will be a straightforward command for this, as well as support via #!share the same it currently works between C#, F#, and PowerShell.


There is a delay in response to requests.

Maybe the problem is caused by the middleware.

When reporting an issue you should try to provide a way to reproduce the issue, an explanation of what you expect to happen and what actually happens.
I suspect this is from the docker hostname check which is known to slow things down, but it's impossible to know for sure.
The SQL panel can also slows things down when you have many or large queries.
If you can start pinpointing the issue or provide a reproducible environment, please reopen the issue.


[CT-3525] [Bug] Option --no-send-anonymous-usage-stats creates different behavior compared to DBT_SEND_ANONYMOUS_USAGE_STATS=false

Is this a new bug in dbt-core?

[X] I believe this is a new bug in dbt-core
[X] I have searched the existing issues, and I could not find an existing issue for this bug

Current Behavior
Running time dbt ls -s something --no-send-anonymous-usage-stats generates these logs:
19:46:13.215433 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'start', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff98916350>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff96f84e50>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff96f87be0>]}

============================== 19:46:13.217242 | 57017d07-14d8-4178-884f-d2cdeda0a998 ==============================
19:46:13.217242 [info ] [MainThread]: Running with dbt=1.7.4
19:46:13.217977 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'log_cache_events': 'False', 'write_json': 'True', 'partial_parse': 'True', 'cache_selected_only': 'False', 'warn_error': 'None', 'fail_fast': 'False', 'version_check': 'True', 'log_path': '/usr/local/dbt_snowflake/logs', 'profiles_dir': '/usr/local/airflow/.dbt', 'debug': 'False', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'log_format': 'default', 'invocation_command': 'dbt ls -s something --no-send-anonymous-usage-stats', 'target_path': 'None', 'introspect': 'True', 'send_anonymous_usage_stats': 'True'}
19:46:13.409792 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'project_id', 'label': '57017d07-14d8-4178-884f-d2cdeda0a998', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff9911d210>]}
19:46:13.456307 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'adapter_info', 'label': '57017d07-14d8-4178-884f-d2cdeda0a998', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff95f31900>]}
19:46:13.457063 [info ] [MainThread]: Registered adapter: snowflake=1.7.1
19:46:13.518584 [debug] [MainThread]: checksum: 248e8aba381ba2d577dce7ab50d2010c2aac45d2b27e6e6214cdd96d2d41fabc, vars: {}, profile: , target: , version: 1.7.4
19:46:14.017287 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
19:46:14.017913 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
19:46:14.075769 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'load_project', 'label': '57017d07-14d8-4178-884f-d2cdeda0a998', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff94b2c0d0>]}
19:46:14.194683 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'resource_counts', 'label': '57017d07-14d8-4178-884f-d2cdeda0a998', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff94dcea70>]}
19:46:14.195421 [info ] [MainThread]: Found 337 models, 11 snapshots, 444 tests, 10 seeds, 2 operations, 145 sources, 0 exposures, 0 metrics, 683 macros, 0 groups, 0 semantic models
19:46:14.196130 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'runnable_timing', 'label': '57017d07-14d8-4178-884f-d2cdeda0a998', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff94dcea10>]}
19:46:14.199641 [warn ] [MainThread]: The selection criterion 'something' does not match any nodes
19:46:14.200179 [warn ] [MainThread]: No nodes selected!
19:46:14.201185 [debug] [MainThread]: Resource report: {"command_name": "list", "command_success": true, "command_wall_clock_time": 1.0115864, "process_user_time": 1.816159, "process_kernel_time": 0.270297, "process_mem_max_rss": "137156", "process_in_blocks": "0", "process_out_blocks": "0"}
19:46:14.201706 [debug] [MainThread]: Command `dbt ls` succeeded at 19:46:14.201608 after 1.01 seconds
19:46:14.202355 [debug] [MainThread]: Sending event: {'category': 'dbt', 'action': 'invocation', 'label': 'end', 'context': [<snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff98916350>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff95f31900>, <snowplow_tracker.self_describing_json.SelfDescribingJson object at 0xffff94dcc550>]}
19:46:14.203033 [debug] [MainThread]: Flushing usage events

and the time output is:
real    0m3.561s
user    0m2.040s
sys     0m0.281s

Whereas running time DBT_SEND_ANONYMOUS_USAGE_STATS=false dbt ls -s something generates these logs:

============================== 19:48:04.896626 | 66292a5a-377e-4a8a-a374-91bea051e347 ==============================
19:48:04.896626 [info ] [MainThread]: Running with dbt=1.7.4
19:48:04.898463 [debug] [MainThread]: running dbt with arguments {'printer_width': '80', 'indirect_selection': 'eager', 'write_json': 'True', 'log_cache_events': 'False', 'partial_parse': 'True', 'cache_selected_only': 'False', 'profiles_dir': '/usr/local/airflow/.dbt', 'fail_fast': 'False', 'version_check': 'True', 'log_path': '/usr/local/dbt_snowflake/logs', 'debug': 'False', 'warn_error': 'None', 'use_colors': 'True', 'use_experimental_parser': 'False', 'no_print': 'None', 'quiet': 'False', 'warn_error_options': 'WarnErrorOptions(include=[], exclude=[])', 'static_parser': 'True', 'log_format': 'default', 'invocation_command': 'dbt ls -s something', 'target_path': 'None', 'introspect': 'True', 'send_anonymous_usage_stats': 'False'}
19:48:05.135858 [info ] [MainThread]: Registered adapter: snowflake=1.7.1
19:48:05.187251 [debug] [MainThread]: checksum: 248e8aba381ba2d577dce7ab50d2010c2aac45d2b27e6e6214cdd96d2d41fabc, vars: {}, profile: , target: , version: 1.7.4
19:48:05.519577 [debug] [MainThread]: Partial parsing enabled: 0 files deleted, 0 files added, 0 files changed.
19:48:05.520165 [debug] [MainThread]: Partial parsing enabled, no changes found, skipping parsing
19:48:05.703670 [info ] [MainThread]: Found 337 models, 11 snapshots, 444 tests, 10 seeds, 2 operations, 145 sources, 0 exposures, 0 metrics, 683 macros, 0 groups, 0 semantic models
19:48:05.707770 [warn ] [MainThread]: The selection criterion 'something' does not match any nodes
19:48:05.708355 [warn ] [MainThread]: No nodes selected!
19:48:05.709225 [debug] [MainThread]: Resource report: {"command_name": "list", "command_success": true, "command_wall_clock_time": 0.8377126, "process_user_time": 1.595339, "process_kernel_time": 0.210098, "process_mem_max_rss": "137264", "process_in_blocks": "0", "process_out_blocks": "0"}
19:48:05.709759 [debug] [MainThread]: Command `dbt ls` succeeded at 19:48:05.709641 after 0.84 seconds
19:48:05.710292 [debug] [MainThread]: Flushing usage events

And time output is:
real    0m2.220s
user    0m1.814s
sys     0m0.219s

So using the env var config actually results in fewer events logged and a large improvement in runtime.
Expected Behavior
As I understand it, --no-send-anonymous-usage-stats and DBT_SEND_ANONYMOUS_USAGE_STATS=false should be identical.
Steps To Reproduce
Run:
time dbt ls -s something --no-send-anonymous-usage-stats
and
time DBT_SEND_ANONYMOUS_USAGE_STATS=false dbt ls -s something
Compare the logs and runtime.
Relevant log output
No response
Environment
- OS:
- Python:
- dbt:

Which database adapter are you using with dbt?
snowflake
Additional Context
No response

Thanks for reporting this @mbarugelCA 🙏
I was able to reproduce the same thing that you reported.
Reprex
First, I needed to make sure I had all the relevant environment variables unset:
echo $DBT_SEND_ANONYMOUS_USAGE_STATS
echo $DO_NOT_TRACK
unset DBT_SEND_ANONYMOUS_USAGE_STATS
unset DO_NOT_TRACK

From there, I ran all the following commands to see what dbt is doing:
dbt --debug ls -s something --send-anonymous-usage-stats | grep -Eo "'send_anonymous_usage_stats': '(.*)'"
DBT_SEND_ANONYMOUS_USAGE_STATS=true dbt --debug ls -s something | grep -Eo "'send_anonymous_usage_stats': '(.*)'"
dbt --debug ls -s something --no-send-anonymous-usage-stats | grep -Eo "'send_anonymous_usage_stats': '(.*)'"
DBT_SEND_ANONYMOUS_USAGE_STATS=false dbt --debug ls -s something | grep -Eo "'send_anonymous_usage_stats': '(.*)'"
DO_NOT_TRACK=true dbt --debug ls -s something | grep -Eo "'send_anonymous_usage_stats': '(.*)'"

Explanation:

--debug writes debug-level logs to standard out (rather than needing to look in logs/dbt.log.
Then the grep -Eo "'send_anonymous_usage_stats': '(.*)'" part finds the log line that shows the config settings that dbt is using.

The output we'd expect is:
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'

But depending on the config setting for send_anonymous_usage_stats within profiles.yml, I saw different behavior.
When send_anonymous_usage_stats is either set to True (default), None, or not set within profiles.yml:
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'True'   ❌
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'

When send_anonymous_usage_stats is set to False within profiles.yml:
'send_anonymous_usage_stats': 'False'  ❌
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'

Conclusion
👉 So it looks like the --send-anonymous-usage-stats / --no-send-anonymous-usage-stats CLI flags are being ignored rather than adhering to the standard precedence.
Acceptance criteria
The output from the above commands is always the following, regardless if the send_anonymous_usage_stats setting within profiles.yml is None, True, or False):
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'True'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'
'send_anonymous_usage_stats': 'False'

I'm not sure if it affects the behavior described in this issue or not, but we take an additional environment variable DO_NOT_TRACK into consideration here:
https://github.com/dbt-labs/dbt-core/blob/68970d09fa69889fa86023c94c09be8afdc0b103/core/dbt/cli/flags.py#L281-L283
We mostly expose only a single environment variable to control flags / global configs, but this is the rare case (and maybe only one) in which we allow two different environment variable names:

DBT_SEND_ANONYMOUS_USAGE_STATS
DO_NOT_TRACK

According to the click docs here, envvar can also be:

a list of different environment variables where the first one is picked.

Also, click handles the BOOL parameter type like this:

The string values “1”, “true”, “t”, “yes”, “y”, and “on” convert to True. “0”, “false”, “f”, “no”, “n”, and “off” convert to False.

This happens to be exactly what is in the logic below, which would lower the barriers to that portion of the code refactor:
https://github.com/dbt-labs/dbt-core/blob/68970d09fa69889fa86023c94c09be8afdc0b103/core/dbt/cli/flags.py#L282
So we might be able to remove this code in favor of adding this here:
    envvar=["DO_NOT_TRACK", "DBT_SEND_ANONYMOUS_USAGE_STATS"],

At the very least, it would reduce the number of occurrences of os.getenv in the code base (which feels like tech debt). But there's a possibility it would also resolve the undesirable behavior described in this issue.

I just found out this same issue!
I was running dbt-duckdb on a place (no Internet), and saw that any run was hanging for 30s.
I suspected it was due to tracking so I tried adding --no-send-anonymous-usage-stats, but it was still hanging.
Now trying with no internet and DO_NOT_TRACK=1, it doesn't hang.
This seems related to this value: https://github.com/dbt-labs/dbt-core/blob/afb2d61a0876c4b3bafc0ec4fc359458859a7f2b/core/dbt/tracking.py#L59
Should we raise another issue that if the client is not connected to the internet we should not hang for 30 secs?

Should we raise another issue that if the client is not connected to the internet we should not hang for 30 secs?

It seems reasonable to give the anonymous tracking a quick shot and give up after a very short amount of time (1 or 2 seconds total) if it isn't able to establish a connection.
If you want to open an issue for this, I'll label it as help wanted.


Phantomjs Crash Dump.

Crash Dump: https://www.dropbox.com/s/stqkbs6e3dyancm/4d69218e-b873-4437-b10d-1a73ef931f72.dmp
System Info:
Windows Server Datacenter
Service Pack 2
Processor: Intel(R0 Xeon(R) CPU E5507 @ 2.27Ghz 3.19GHz
Memory(RAM):  1.66 GB
System type: 32-bit Operating System

Closing the issue. Fixing some css fixed this issue.


soup.find() returning 'None', resulting in AttributeError: 'NoneType' object has no attribute 'get_text

Traceback (most recent call last):
File "/workspaces/pcpartpickertest/index.py", line 7, in
parts = pcpp.part_search("i7")
File "/home/codespace/.python/current/lib/python3.10/site-packages/pypartpicker/scraper.py", line 232, in part_search
soup = self.__make_soup(f"{search_link}&page={i + 1}")
File "/home/codespace/.python/current/lib/python3.10/site-packages/pypartpicker/scraper.py", line 95, in _make_soup
if "Verification" in soup.find(class="pageTitle").get_text():
AttributeError: 'NoneType' object has no attribute 'get_text'

The error is returned by this example code:
from pypartpicker import Scraper

pcpp = Scraper()
parts = pcpp.part_search("i7")

for part in parts:
    print(part.name)

first_product_url = parts[0].url
product = pcpp.fetch_product(first_product_url)
print(product.specs)`

The issue is with the last line of code in the error message, which tells us that it occurs at scraper.py, line 95, in __make_soup.
The key is in the statement:
if "Verification" in soup.find(class_="pageTitle").get_text():
The call to soup.find should return some object reference on which a call to get_text will return some data. But if soup.find does not succeed it does not return any object (in reality it returns None). So the call to get_text is impossible because there is no object to call it from. Which results in the error message: AttributeError: 'NoneType' object has no attribute 'get_text'
This may be a problem with the scraper.py code or the Soup package itself.

This issue is likely occurring due to a cloudflare bot verification check when the library makes the request to pcpartpicker.
This can be solved by using custom HTTP headers, the same as the ones your browser users in order to bypass the check.
When creating the instance of the Scraper class, pass in a dictionary headers with the same HTTP headers as your browser.

I will think about adding a new CloudflareCheck error to the library to make it more clear that this is the case, as the current error is confusing.

It seems to be working ok now :)


In ALE, Show in Map button has wrong URL

It should point to:
https://bcgov.github.io/ols-devkit/ols-demo/index.html
instead of:
https://ols-demo.apps.gov.bc.ca/index.html?

@mraross PR #199 is ready for your review in ols-devkit


Restructure files

Purpose
Fix #52
Does this introduce a breaking change?
When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.
[ ] Yes
[x] No

Type of change
[ ] Bugfix
[ ] Feature
[ ] Code style update (formatting, local variables)
[x] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist
See CONTRIBUTING.md for more details.

[-] The current tests all pass (python -m pytest).
[-] I added tests that prove my fix is effective or that my feature works
[-] I ran python -m pytest --cov to verify 100% coverage of added lines
[-] I ran python -m mypy to check for type errors
[-] I either used the pre-commit hooks or ran ruff manually on my code.

mypy is unhappy (as usual). I'm placating it.

What's weird is that these tests passed last Thursday.
I upgraded my local mypy version and got the same errors. I think it's the latest release.
Thanks! For fixing it.

@john0isaac Yep latest mypy came out last week, and caused a bunch of issues all over. It broke azure-search-openai-demo CI cuz of https://github.com/python/mypy/issues/17550 which I've also just fixed.


[FEATURE] Disable values starting with a space

Copying the global setting value (or some other value from UI), adds space in front of the copied text, then when pasting I managed to have a template whose name starts with space....

@rhtyd @davidjumani can we have textinput to trim entered text while submitting the form?

@rhtyd I think we need to trim text input after entered?

We can do it both ways, but it would be better to trim it during the submit, to reduce the code changes introduced

@andrijapanicsb it may cause unexpected results in some cases, what is somebody wants to put space in front/end of a string input? Does old UI trim the space too ? I'm tentatively pushing this to 1.1 as it's not a critical/blocker issue.

David checked and it's an issue at backend, fixing this in UI will still allow API to do this - so ideally the backend should trim the space.

Fixed in the backend via https://github.com/apache/cloudstack/pull/4311

Closing based on @davidjumani 's remark, the fix at backend needs to be cautious but will utlimately fix the issue for all kinds of client inputs (UI, API...)


Jv runtime filters protobufs

Description
Draft for gRPC messages for runtime filtering in collector.
Checklist

[x] Investigated and inspected CI test results
- [ ] Unit test and regression tests added
- [ ] Evaluated and added CHANGELOG entry if required
- [ ] Determined and documented upgrade steps
- [ ] Documented user facing changes (create PR based on openshift/openshift-docs and merge into rhacs-docs)

If any of these don't apply, please comment below.
This just adds some protobufs.
Testing Performed
Here I tell how I validated my change
TODO(replace-me)
Use this space to explain how you validated that your change functions exactly how you expect it.
Feel free to attach JSON snippets, curl commands, screenshots, etc. Apply a simple benchmark: would the information you
provided convince any reviewer or any external reader that you did enough to validate your change.
It is acceptable to assume trust and keep this section light, e.g. as a bullet-point list.
It is acceptable to skip testing in cases when CI is sufficient, or it's a markdown or code comment change only.
It is also acceptable to skip testing for changes that are too taxing to test before merging. In such case you are
responsible for the change after it gets merged which includes reverting, fixing, etc. Make sure you validate the change
ASAP after it gets merged or explain in PR when the validation will be performed.
Explain here why you skipped testing in case you did so.
Have you created automated tests for your change? Explain here which validation activities you did manually and why so.
Reminder for reviewers
In addition to reviewing code here, reviewers must also review testing and request further testing in case the
performed one does not seem sufficient. As a reviewer, you must not approve the change until you understand the
performed testing and you are satisfied with it.

To build on the above idea that "status" is feature-specific, I think that we should specify the potential values for it (even if they are not implemented in the MVP).
Quick braindump below:
feature = LISTENING_ENDPOINTS
status = { enabled: <bool> }

feature = PROCESSES
status =
{
  enabled = <bool>,
  max_params = <int>
}

feature = NETWORK_CONNECTIONS
status =
{
  enabled = <bool>,
  aggregate_external = <bool>
}

To build on the above idea that "status" is feature-specific, I think that we should specify the potential values for it (even if they are not implemented in the MVP).
Quick braindump below:
feature = LISTENING_ENDPOINTS
status = { enabled: <bool> }

feature = PROCESSES
status =
{
  enabled = <bool>,
  max_params = <int>
}

feature = NETWORK_CONNECTIONS
status =
{
  enabled = <bool>,
  aggregate_external = <bool>
}

We can do that. We could use oneof for status. I might still have a string be a member of each version of status, which could allow for flexible configuration. It is also reasonable to make EXTERNAL_IPS a feature of NETWORK_CONNECTIONS instead of being its own feature. I think it aggregate_external is a better name than EXTERNAL_IPS. Having said all of that. I think making those changes is probably not a priority right now.

I might still have a string be a member of each version of status, which could allow for flexible configuration.

protobuf types can evolve by adding supplementary fields in objects. I think that adding a string is not necessary.

It is worth considering if CollectorRuntimeConfig should be added to DynamicClusterConfig.

/retest

I created a document for these protobuf definitions here https://docs.google.com/document/d/120SipUJRtFQX2Ll1llCB4Lt6Y4ecUX5Qr49Gr8_9bHc/edit which discussed some alternatives.

I have removed CollectorRuntimeConfigurationACK as it is not clear if it is needed. If it turns out that it is needed, it can be added back in another PR.

/test ocp-4-12-nongroovy-e2e-tests

/retest

/retest

/retest

/retest


Fix decoding issue on Field Display

I'd realised that the previous implementation wasn't deconding the String properly. I believe this fixes it, but do let me know if there's a better way to do so

Any chance you could have a look at these, @blackbeam ?


MWPW-136767 - disable scroll

This PR has been opened because previous PR (https://github.com/adobecom/milo/pull/1419) contains some unwanted changes on footer's appearance on test pages, while code doesn't have any changes which could cause this behaviour.
This PR should resolve the issue of scrollable background when modal dialog (which is authored to have a 'curtain') is opened. In other cases (with 'curtain-off') background should be scrollable per requirements.
Resolves: MWPW-136767
Test URLs:
Before: https://main--milo--adobecom.hlx.page/drafts/rclayton/video-examples?martech=off
After: https://mwpw-136767-disable-scroll--milo--saragajic11.hlx.page/drafts/rclayton/video-examples?martech=off
Commerce dialog (it should work the same as before this PR):
Before: https://main--milo--adobecom.hlx.page/drafts/ruchika/discover1?martech=off
After: https://mwpw-136767-disable-scroll--milo--saragajic11.hlx.page/drafts/ruchika/discover1?martech=off

thank you @saragajic11

@saragajic11 could you please add the required label to the PR.
@spadmasa , thanks for testing the PR https://github.com/adobecom/milo/pull/1419 Could you please add your comments regarding CC page verification here too. Footer issue you noticed in previous PR should be resolved here.

@saragajic11 also please assign this PR to a QE or SME from DC side to verify.

Verified the flow in cc in the below url scrollable background when modal dialog is opened issue is seen fixed .
https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=mwpw-136767-disable-scroll--milo--saragajic11#after-effects

but on the content in the page still see the issue.

 - transperancy on the scroll
share icons missing in the footer

issues are not seen in main branch : https://main--cc--adobecom.hlx.page/creativecloud/animation/discover

Verified the flow in cc in the below url scrollable background when modal dialog is opened issue is seen fixed . https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=mwpw-136767-disable-scroll--milo--saragajic11#after-effects
but on the content in the page still see the issue.   - transperancy on the scroll share icons missing in the footer
issues are not seen in main branch : https://main--cc--adobecom.hlx.page/creativecloud/animation/discover
Thankyou @Ruchika4 , verified the flow in mobile mode as well as advised and do not see the scroll functionality working https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=mwpw-136767-disable-scroll--milo--saragajic11#animate

Thanks for testing @spadmasa
@saragajic11 looks like there is some issues with sending viewport dimensions to iframe (commerce modal) which is causing it to break in all viewports. I tested it in a test branch by removing body css file and just defining the css class name .disable-scroll Test page here https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=disable-scroll--milo--ruchika4#after-effects and it appears to be working fine for commerce modal. @spadmasa could you also please verify this. I see this fixes some other content issue you pointed out earlier too.

Commerce modal is used on homepage too so please get it tested by them also once. Our team (CC team) is testing only commerce modal on CC pages. So please get all the other modals (georouting etc) tested before getting the PR merged.

Verified the flow in cc in the below url scrollable background when modal dialog is opened issue is seen fixed . https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=mwpw-136767-disable-scroll--milo--saragajic11#after-effects
but on the content in the page still see the issue.   - transperancy on the scroll share icons missing in the footer
issues are not seen in main branch : https://main--cc--adobecom.hlx.page/creativecloud/animation/discover
Thankyou @Ruchika4 , verified the flow in mobile mode as well as advised and do not see the scroll functionality working https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=mwpw-136767-disable-scroll--milo--saragajic11#animate

Thanks for testing @spadmasa @saragajic11 looks like there is some issues with sending viewport dimensions to iframe (commerce modal) which is causing it to break in all viewports. I tested it in a test branch by removing body css file and just defining the css class name .disable-scroll Test page here https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=disable-scroll--milo--ruchika4#after-effects and it appears to be working fine for commerce modal. @spadmasa could you also please verify this. I see this fixes some other content issue you pointed out earlier too.

Verified the flow in the url provided in all view ports looks fine.
https://main--cc--adobecom.hlx.page/creativecloud/animation/discover?milolibs=disable-scroll--milo--ruchika4#after-effects

content issue is not seen

Closing this one as new PR has been opened which doesn't affect content:
https://github.com/adobecom/milo/pull/1466


fix: separate before and after cache

:tada: This PR is included in version 1.0.0 :tada:
The release is available on GitHub release
Your semantic-release bot :package::rocket:


aiohttp version update

Hello, could you please mind updating the required aiohttp version requirements.
Lavalink current aiohttp version requirement: 3.6.0
Please update it to version 3.7.3 or later

Let me if I am wrong.

if it isn't broken, don't fix it. ¯_(ツ)_/¯

It is I mean, clashing with other packages which requires newer version of aiohttp.

what about the people who need an older version of x package, at the end of the day it isnt really going to hurt you that much if you have to run a simple command tbh