Revert "Update LedDeviceAPA102.cpp by pcaffardi"

Reverts tvdzwan/hyperion#484
was working before so this merge was not needed
sorry for the inconvenience

Thank you, indeed it reintroduced the EndFrame Bug
Compiled a new verion without it. Great!

Be careful guys, check the source code before commiting a change. Let's hope @tvdzwan merge the commit soon

Build it against latest master branch with these changes however not getting the end frame bug like with the latest official binary.

ok .. thats kind of weird since the last update of binarys was 28 days ago (jan 2016)
and the revert restores the APA102.cpp back to the last commit on Nov 4, 2015
... any ideas whats going wrong with the binarys?

Yeah thought that official binary would have the fix already, went thru the commit log and couldn't find any mention of recent APA102 end frame fixes (after jan 2016 release) so didn't look like it was merged.
Other users ran into the APA102 end frame issue on the Mediaportal forums (Ambient area) so went from there, they had to use the previously patched binary posted by pcaffardi to make it work.
Upgraded tot APA102 2 weeks ago and used the wiki installation manual for Raspbian where after I ran into the last led issue as well, only fix that works at the moment is compiling from master branch or the old patched one.

ok but this reverts back to november 2015 and you confirmed it working (#480) .. so it should be in the binarys since a long time confused
did @tvdzwan get something wrong while compiling ???

Correct also with the master branch from 6 days it worked as expected.
Not sure why there would be a difference compared to binary release :(
Already double checked the number of leds with Arduino because if you input those incorrectly (more than available) you won't notice the end frame bug.

ok i found it ..
following your text - install as described in wiki
i checked the install script - it will download hyperion.tar.gz - which is - hold on - 11 MONTHS old
instead it should maybe point to hyperion_rpi.tar.gz which is 28 days - almost up to date
i will add a pull request to fix that in a minute :p

Awesome, thanks :+1:

@wisc17
Don´t invest that much time in it. I wrote a new Install/update script. Currently i test it with the users from the german raspberrypi forum.

ok you do that ^^ but this is only a "typo" problem - we should correct this first before moving to the next level ;)

the install script points to the old release.
On which image would you point now from the 5 releases?

it should point to the hyperion_rpi.tar.gz
tested and running on my RPI

Yes, you could update, but if you change the install.sh people would install on new devices they have no gpio2spi (cause tar just overwrites and gipo2spi is indeed not deleted)


Exporting: errors with meshes with no faces

Game
Resident Evil 5,
Errors when trying to export meshes:

No vertices -> exception
Vertices, but no faces -> exception
Some vertices with no faces -> artifacts

Vertices by itself use 'degenerate' triangles (e.g. invisible, see pl00ChrisNormal.arc, in mesh 0). However, it might be also a good idea to just skip exporting those kind of meshes.
For 3, it seems that the algorithm for converting triangle lists to triangle strips expects all vertices to be referenced by an index.
For 1, 2, the exception is:
Traceback (most recent call last):
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/blender.py", line 189, in execute
    func(obj, self.filepath)
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/engines/mtframework/blender_export.py", line 78, in export_arc
    exported_mod = export_mod156(child)
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/engines/mtframework/blender_export.py", line 168, in export_mod156
    exported_meshes = _export_meshes(blender_meshes, bounding_box, bone_palettes, exported_materials)
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/engines/mtframework/blender_export.py", line 424, in _export_meshes
    vertices_array = _export_vertices(blender_mesh_ob, bounding_box, mesh_index, bone_palette)
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/engines/mtframework/blender_export.py", line 306, in _export_vertices
    tangents = _get_tangents_per_vertex(blender_mesh)
  File "/home/sysop/.config/blender/2.76/scripts/addons/albam/engines/mtframework/blender_export.py", line 292, in _get_tangents_per_vertex
    blender_mesh.calc_tangents(uv_name)
RuntimeError: Error: Tangent space computation needs an UVMap, "UVMap" not found, aborting

Fixed in Albam Reloaded and backported to Albam.


multithreaded runtests.py

bc.. It should be trivial to run several tests at once using Parallel Python or
something else. py.test offers such a feature already, so maybe we should
use it instead of a own solution.
p. Original issue for "#99":https://github.com/fredrik-johansson/mpmath/issues/99: "http://code.google.com/p/mpmath/issues/detail?id=59":http://code.google.com/p/mpmath/issues/detail?id=59
p. Original author: "https://code.google.com/u/Vinzent.Steinberg<EMAIL_ADDRESS>p. Original owner: "https://code.google.com/u/Vinzent.Steinberg<EMAIL_ADDRESS>

I think we could use pytest-xdist plugin.  We run CI tests with pytest anyway...


Bug: wrong cursor motion on mouse scroll in the right preview window

Description
When opening lazygit window, I try to use mouse scroll down and view git diff on the right preview window.
But the behaviour is different with the terminal lazygit. The cursor just run away, and I have to re-open it.
To Reproduce
nvim config:
set mouse=a

Expected behavior
correct mouse scroll behaviour.
Screenshots

Desktop (please complete the following information):
macOS 13.4.1 (latest stable), M1 MAX chip.

nvim --version: v0.9.1

I found this issue only appears on neovide, a GUI client of neovim, not on native terminal such as iterm2, alacritty.
So maybe this is an issue from neovide.


cannot find package "github.com/SumoLogic/sumoshell/sumojson"

where can I find this package?

Hey David, development on Sumoshell has moved to
https://GitHub.com/rcoh/angle-ginder which is much easier to use, feature
complete and high performance
Let me know if that doesn't work for you.
On Mon, Jul 9, 2018, 8:44 AM Dave Gallant<EMAIL_ADDRESS>wrote:

where can I find this package?
—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/SumoLogic/sumoshell/issues/37, or mute the thread
https://github.com/notifications/unsubscribe-auth/AAeFZyEG7LJ7c9zJMzk6ax2Pdn3W0vCSks5uE3pfgaJpZM4VH8iK
.


Make tools/lint-all runnable on python 3

Since pyflakes behaves differently on python 3, I had to fix some python 3 compatibility issues to make the pyflakes check pass.

Automated message from Dropbox CLA bot
@sharmaeklavya2, it looks like you've already signed the Dropbox CLA. Thanks!

lgtm, merged!


Binding class methods and other primitives at runtime

I'm studying how to implement binding Wren to my classes at runtime based on my own reflection metadata, in a similar fashion as I bound lua. (https://github.com/hugoam/mud/blob/master/src/lang/Lua.cpp)
I'm almost in a dead end, or all designs I can think of will end up way too complex. So I want to know if you can help me or give me some pointers :)
The crux of the problem is that, unlike in lua where there is a function call mechanism with upvalues (for closures) at the core of the library, in Wren everything is the other way around: everything is an object, so you only ever call methods, and the C API method binding mechanism doesn't allow you to store any closure data with it : you only give a bare function pointer, which is biased towards implementing all your foreign methods manually (or generating C++ code).
I can't bind C/C++ primitives at runtime without being able to store some "closed-over" values somehow (like a wren handle) together with the bound method in bindForeignMethod. Which, if I'm starting to understand Wren design, seems likely to be impossible.
So I've been trying to think of ways to store a Wren handle (to a foreign class that I defined to store my callable), but since I have no way to attach it to the bound method in the bindForeignMethod handler, I need an alternative.
Since in Wren closures seem to be a higher level construct based on the former mechanisms (objects, methods, etc...), I can think of some very contrived workarounds where I would define my closure classes, create instances of these closures for each of my primitives, and write boilerplate "proxy" methods that would call these closures in the body of the foreign class, and then compile all this Wren boilerplate. But there are two things about such a design :

I'm still not sure it's entirely possible
that's more an argument towards not using Wren so far, and I really wanted to provide this otherwise very neat language :)

Did I miss something crucial that would help me implement this with Wren ? If I haven't been clear or making sense let me know too :)

I think I found a pattern to maybe get this working :
I need to create these foreign classes, so I only have to bind a handful of foreign functions :

foreign class Function {
    construct create(name) {}

    foreign call(func)
    foreign call(func, _)
    foreign call(func, _, _)
    // etc..
}

foreign class Member {
    construct create(class, name) {}

    foreign set(object, value)
    foreign get(object)
}

foreign class Static {
    construct create(class, name) {}

    foreign set(value)
    foreign get()
}

foreign class Method {
    construct create(class, name) {}

    foreign call(object)
    foreign call(object, _)
    foreign call(object, _, _)
    // etc..
}

And then for each class I need to bind, I just need to redirect to a bunch of these primitives, stored as static members, initialized at startup 👎
foreign class vec3 {

    static init() {
        __x = Member.create("vec3", "x")
        __y = Member.create("vec3", "y")
        __z = Member.create("vec3", "z")
    }

    add(v) { return __add.call(this, { v }) }

    x { return __x.get(this) }
    x=(value) { __x.set(this, value) }

    y { return __y.get(this) }
    y=(value) { __y.set(this, value) }

    z { return __z.get(this) }
    z=(value) { __z.set(this, value) }

    static up { return __up.get() }
    static up=(value) { return __up.set(value) }
}

This will probably add a significant overhead as opposed to a lua-like system of directly binding closures, but if I can get this working, I would probably be okay with that.
I'll try to implement this tomorrow and let you know if I manage :)

I have one blocker.
If I write :
foreign class Constructor{
    construct create(class) {}

    foreign call()
}

foreign class vec3 {
    construct create() { __constructor.call() }
    static init() {
        __constructor = Constructor.create("vec3")
    }
}

Can I call wrenSetSlotNewForeign in the implementation of Constructor.call and have that behave as if I called it inside of the initialize function ? Or will this break everything ?

I don't understand what you are trying to achieve. It looks like reflexion API but not sure.

I'll try to make the problem clear:
I have N C++ classes with members, methods, constructors which I'm binding to Wren.
However, I don't write N*x C function to bind each of these foreign primitives: I have a generic reflection mechanism to do that, so I instead want to bind all of these foreign object members, methods and constructors to one single C function which itselfs calls my runtime reflection. However, this function needs context about the class, but Wren doesn't have any mechanism to either bind closures to foreign methods, and it doesn't have a mechanism to get the class in the handlers either IIUC.
The approach described above would have worked, except for one blocker on which I'm stuck on: the foreign class constructor apparently needs wrenSetSlotNewForeign to be called inside the initialize handler, so my idea of allocating the actual foreign object inside a Constructor.create() method called by the foreign constructor body doesn't work: the constructor body is only called if the foreign object was already allocated (and not doing so yields an error apparently).

Earlier today I started to think that since we have wren_modules in place it was time to import wrenalyzer in the source code, so we can start some automated tooling. With that in place it should be possible to do binding generation. Do you think this could help you ? I mean wrenalyzer has to be updated and and stuff, but it has in its code the tools to produce AST of wren source, so it should be relatively ease to produce a binding generator for your needs.

Otherwise re reading your constructor problem, I don't see any reason why you should not be able to call wrenSetSlotNewForeign anyware in C code. Just avoid to call wrenEnsureSlots since it is still broken frown within a foreign. But it should just work.

I'm trying to avoid generating more code here, so a generator wouldn't really help.
I don't really understand the design of wrenSetSlotNewForeign, it only satisfies a very very narrow use case.

It's meant to be called from C directly but doesn't call the constructor body in this case (so why do foreign classes have constructors in the first place ?)
When called from Wren (which is the only way to create a C/C++ construct from Wren code), it's actually allocating the whole object itself, not just the foreign part. Which means that I can't call it anywhere, I have to call it in initializer, because if I don't there is simply no object. (and the actual constructor then fails to call)

It fits two different uses at the same time but doesn't fit either use well. It also allows to call wrenSetSlotNewForeign outside of the initializer with the same class but a different size than inside the initializer (which doesn't make sense).
It would really need to be split up in wrenSetSlotNewForeign to create an object and wrenAllocateForeign to allocate the data itself. The latter would be called inside of the initializer (or in a handler called by the constructor, which is what I'm trying to do).

Foreign constructors are for the case, where calling the WrenReallocateFn is enought to initialise the object, so you can then do the extra initialisation in wren.
wrenSetSlotNewForeign is basically a memory allocator from the C point of view. So it returns a void* to the memory allocated for the foreign data. You can use that to do a C function call to initialise your data, like you would after a malloc. Or in C++ initialise your object wirth a placement new call, and finialise with a distructor method call to ~Foo() (do not call delete since the memory is owned by wren).

I think you missed one point, wren doesn't require you to have a constructor method to allocate a foreign object. You can have a static foreign method (of any name) that create the foreign object for you. So you can do something like:
foreign class Foo {
  foreign static new() // This is an allocator with the usual name
  foreign static new(a, b, c) // This is an allocator with the usual name, and 3 parameter
  foreign static singleton()  // This is an allocator with an unusual name that returns a singleton instance
  foreign construct new() // Not sure it works, but it should technically call a WrenReallocateFn and then
                                       // a foreign method with "init new()" signature.
}

Thanks for the clarifications, I deleted my comment before you answered because I was guessing I misunderstood something. I'll look into all these options and update this issue soon :)

I found a way to make this work.
But I have a full additional level of indirection on each constructor, function, setter, getter and method call, that wouldn't need to be there if Wren made foreign functions to be plain closures, instead of just a simple C function pointer. I don't know Wren internals enough to estimate how complex of a change that would be, but I would gladly study it if you think it's doable and would be open to a pull request regarding this.

If you mean that the type WrenForeignMethodFn returned by bindForeignMethod is then an object to which I can attach some values, then yes

Actually, I said I managed to make this work, but I still had to add a missing function to Wren.
It's a bit off-topic to this issue, but unless I'm mistaken there is no way in the public interface to assign a value to a variable from C. Which, to do two-ways C/Wren interactions, is kind of a fundamental feature.
So I added this function:
void wrenAssignVariable(WrenVM* vm, const char* module, const char* name, int value_slot);

Should I open a pull request with this patch ?

Try to make pull request, though I should see the code to be sure you don't
create a new variable, else it should be fine.

Can you have a quick look ? I really started looking into Wren 2 days ago, so it's very likely I'm not doing it exactly the way it should be done ^^
https://gist.github.com/hugoam/2483e240573cc9aa90cf7f778c2bc223

I don't want to open new issues for things that might just be due to my misunderstanding of how Wren works, but I have another problem.
I'm trying to use wrenCall(), I have my method handle for init(_,_), and I'm pushing three values of type class A, class B, and class C in slots 0, 1 and 2. (class A is the class that has the init(_,_) method). The call doesn't succeed, with the error being Null does not implement 'init(_,_)'
When I inspect the vm->apiStack, it indeed has three objects of type A,  B, and  C in slots 0, 1 and 2. However when I inspect the values in fiber->stack, it has three objects of type Null, B metaclass, and C metaclass.
Again, I don't want to open an issue for something that might not be a bug, but if you can help me understand what's going on here, I'd be thankful :)

And a last question: is it expected that int currentSize = (int)(vm->fiber->stackTop - vm->apiStack); inside of wrenEnsureSlots (https://github.com/munificent/wren/blob/master/src/vm/wren_vm.c#L1580) sometimes has a negative value ?
Also the documentation of wrenEnsureSlots specifies it's meant to be used inside of foreign methods, but I'm following the documentation about Calling Wren from C and it's using that function, outside of a foreign method, which is what I am doing here.

After investigating, I think I now understand the root of all these issues :
Wren is not at all designed to be called from C outside of foreign functions. So everything related to vm->apiStack will only behave correctly when called inside a foreign functions. You cannot use these straight up from C code.
This is not even about re-entrancy (I saw there is work on this going on), but it's just about simple entrancy. I didn't know that it's not supported. This should probably be specified somewhere ?

I'm investigating, and something is deeply wrong with the plain C -> Wren calling mechanism...
I'm wondering if anyone is actually using this, because it seems to be simply broken. If you are calling Wren repeatedly directly from C, wrenEnsureSlots behavior is broken (the negative value I reported earlier) because vm->apiStack and fiber->stackTop simply don't point to the same chunk of memory.

This issue is known. In some situations wrenEnsureSlot is the source of
stack corruption. But resolving it requires partial reentrancy support, and
Bob takes its time...

Well, with all due respect, there should be a big red disclaimer at the top of the Calling Wren from C section from the docs saying that it doesn't work and you shouldn't use it...
I don't think it requires reentrancy support. This is simply a matter of entrancy. Entrancy doesn't work, in the current state.
I just added a two-liner that solves the problem for me :
void wrenBegin(WrenVM* vm) { 	vm->fiber = wrenNewFiber(vm, NULL); 	vm->apiStack = vm->fiber->stack; }
Also solving the stack corruption should be high priority : a simple way to keep track of it would be to check which fiber apiStack refers too. wrenEnsureSlots should have a big ASSERT checking that that fiber is the same as  vm->fiber.

The source of the corruption is that some internal function backup the
apiStack on the host stack. Meaning there is reentrancy happening.

The behavior of wrenEnsureSlots() is very dangerous and error-prone in any case... It should have a safeguard or an assert to prevent it doing that terribly wrong thing that it is currently doing.

Actually I think this can already be enforced :
just add the following before checking the currentSize inside of wrenEnsureSlots()
  ASSERT(vm->apiStack >= vm->fiber->stack && vm->apiStack < vm->fiber->stackTop,
	     "apiStack must be contained in the current fiber stack");

Unless I overlooked something, this should always be the cases in the situations where wrenEnsureSlots() is safe to execute.

This is it, any resize that results in resize of the stack will trigger the
issue (which defeats the purpose of that function).
Le lun. 6 août 2018 à 18:31, Hugo Amnov<EMAIL_ADDRESS>a écrit :

Actually I think this can already be enforced :
just add the following before checking the currentSize inside of
wrenEnsureSlots()
ASSERT(vm->apiStack >= vm->fiber->stack && vm->apiStack < vm->fiber->stackTop,
"apiStack must be contained in the current fiber stack");
Unless I overlooked something, this should always be the cases in the
situations where wrenEnsureSlots() is safe to execute.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/munificent/wren/issues/593#issuecomment-410768731,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AY7blQ1aXwAQI6htw5WnqDAySK2g_Fk1ks5uOG9igaJpZM4VtwsO
.

That's not really what's happening in the issue I'm triggering here.
Resizing the fiber stack works perfectly fine, the issue I have is triggered by wrenEnsureSlots() measuring it using two pointers that aren't relative to the same stack/memory chunk, hence it tries to resize it with a completely absurd/corrupt value like 34053940.
Adding this check ensure that the apiStack is a pointer relative to the fiber stack and not something completely unrelated relative to another fiber stack.


Use native REST calls into Kube instead of kubectl

We need some guidance on how to get the proper auth token.

FYI, we're working on a proper Go client library.

@MHBauer can't this one be closed now?

If you want. I've got a tracking issue for the full depth of the API
elsewhere.
On Aug 25, 2016 7:11 PM, "Doug Davis"<EMAIL_ADDRESS>wrote:

@MHBauer https://github.com/MHBauer can't this one be closed now?
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/servicebroker/servicebroker/issues/42#issuecomment-242605569,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AMuDAd7Wl6Ai-xr7OMA6_pPtp2-tQX3Cks5qjks-gaJpZM4JTa4G
.


Fix grainy rasterized colorbars

Already tried increasing dPI??? Not sure what else to do. Rasterization necessary RE #81

Now they're scrambled again.


Dockerising!

Hi Derek,
This is my working tweaks for allowing to setup the main config elements via environment variables which in turns allows to use Docker to start prose.
Given that Prose is really client-side only app the setup is slightly strange, with the actual production gulp build being done on container startup so that it can pick up the environment variables for the browserification.
Also in a production setup, we probably wouldn't use serve for serving the static file.
In any case, it works (together with the upcoming gatekeeper dockerisation pull request).
Cheers,
Jun
iilab/prose#2 prose/prose#906 prose/prose#900

@jmatsushita yeah serve might not be the way to go, but will be useful to get set up in the meantime.
At dev seed we're planning to move Prose.io hosting from Github to something like Amazon S3, so we can take care of ssl properly. We'll still be using Travis to push changes to S3, like it's doing now for gh-pages.


Log sink payload format

Forcing log sink payload to be converted to textual format.
Resolves #7

@sobychacko: Thanks for the PR! @markpollack had a suggestion about fixing the problem at the source itself. Should we check to see how time-source is producing data and with format?

@sabbyanandan time-source produces data as text, but its stored in the middleware as byte[] with application/json as the content type. Unless we override the content type, I think that is the expected behavior in the source apps. Any sink or processor application that needs this data can consume it as byte[] and then convert it using the corresponding content type. In the case of log-sink, since we are using a MessageHandler internally, this OOTB conversion will not happen. We need to convert it in the sink itself. @olegz - this is what we discussed when this issue originally came up a while back. Do you have any insights?

@sobychacko , i'm not sure about this statement:

Any sink or processor application that needs this data can consume it as byte[] and then convert it using the corresponding content type.

For example will this DSL still work?
time | transform --expression=payload.toUpperCase() | log

Or will i need to do some additional, magical expressions to convert the byte[] into string?
If the output of the time-source is a text shouldn't the content type be set to application/text?

@sabbyanandan The comment from @sobychacko is exactly correct. And in fact that scenario is explicitly documented in the new user guide (see the 'NOTE' as the last paragraph. . . formatting got screwed up a bit).
There is nothing that must be done a the source. Message Conversion is the responsibility of the Consumer and consumer and for typical user case when the handler signature is handle(Foo foo) we have no issues, but when it is handle(Object foo) or handle(Message<?> foo), then we call it typeless handler and as such can not convert using the internal framework support and at that point it is the responsibility of the application itself. And in this case the actual type semi-communicated by the content-type header.

@tzolov As discussed in other comments, it is the responsibility of the sinks/processors to use an appropriate content-type to convert the data. For our OOTB applications, we can fix for all those needs to be changed, but for custom consumers, it needs to be addressed per consumer basis.


Will the utility help to fill Message Size

Let me take Negotiate200.h generated by real-logic tool as an example. A valid Negotiate200 message to CME should be Packet Header + MDP message as inidicated in the below link.
https://www.cmegroup.com/confluence/download/attachments/170262771/ConflatedTCPPacketStructure.png?version=2&modificationDate=1617896706155&api=v2 1
My code will provide Packet Header. Seems wrapAndApplyHeader() will fill Simple Binary Header part. Who will provide Message Size? Of course I can call sbeBlockAndHeaderLength() to fill it. Is there any better way?
BTW, when to call wrapForEncode()?

Who will provide Message Size?
you should calculate message size by yourself

Of course I can call sbeBlockAndHeaderLength() to fill it
it's not enough for messages with repeating groups or variable length fields

BTW, when to call wrapForEncode()?
don't need to call in case of wrapAndApplyHeader() called


python312Packages.numba: fix compatibility with numpy 2.0.1

Description of changes
numba got broken by the latest numpy update.
While waiting for the next release, this patch should let it build.
Upstream issue: https://github.com/numba/numba/issues/9687
Upstream PR: https://github.com/numba/numba/pull/9683
cc @doronbehar

Things done

Built on platform(s)

[x] x86_64-linux
[ ] aarch64-linux
[ ] x86_64-darwin
[ ] aarch64-darwin

For non-Linux: Is sandboxing enabled in nix.conf? (See Nix manual)

[ ] sandbox = relaxed
[ ] sandbox = true

[ ] Tested, as applicable:

NixOS test(s) (look inside nixos/tests)
and/or package tests
or, for functions and "core" functionality, tests in lib/tests or pkgs/test
made sure NixOS tests are linked to the relevant packages

[ ] Tested compilation of all packages that depend on this change using nix-shell -p nixpkgs-review --run "nixpkgs-review rev HEAD". Note: all changes have to be committed, also see nixpkgs-review usage
[ ] Tested basic functionality of all binary files (usually in ./result/bin/)
24.11 Release Notes (or backporting 23.11 and 24.05 Release notes)

[ ] (Package updates) Added a release notes entry if the change is major or breaking
[ ] (Module updates) Added a release notes entry if the change is significant
[ ] (Module addition) Added a release notes entry if adding a new NixOS module

[x] Fits CONTRIBUTING.md.

Add a :+1: reaction to pull requests you find important.

 How come it didn't fail for you @GaetanLepage locally?
..................................................................................................................................sss......................................................................../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_array_analysis.py", line 990:
        def test_impl():
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_array_analysis.py", line 1053:
        def test_impl():
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
.../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_array_analysis.py", line 1017:
        # these tests shouldn't throw error in array analysis
        def test_impl1():
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_array_analysis.py", line 1023:
        def test_impl2(A, a):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
...s..........sssssssssssssssss...........s.............sssss..ss.............sssssssss..........................ssssssssssssssssssssssssss.........ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss.............ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss/nix/store/wviq85xs5awfp75mj5sjq9mn1nkqhq12-python3-3.12.4/lib/python3.12/multiprocessing/popen_fork.py:66: DeprecationWarning: This process (pid=8) is multi-threaded, use of fork() may lead to deadlocks in the child.
  self.pid = os.fork()
..........s/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 117:
    def test_alias_flat(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 193:
    def test_alias_func_ext(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 110:
    def test_alias_ravel(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 179:
    def test_alias_reshape1(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 186:
    def test_alias_reshape2(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 124:
    def test_alias_transpose1(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 131:
    def test_alias_transpose2(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_remove_dead.py", line 138:
    def test_alias_transpose3(self):
        def func(A, i):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
.......................................s...................s...................................../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_stencils.py", line 75:
    @njit(parallel=True)
    def addone_pjit(a):
    ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
........./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
.s/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numpy/core/fromnumeric.py:3504: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numpy/core/_methods.py:129: RuntimeWarning: invalid value encountered in scalar divide
  ret = ret.dtype.type(ret / rcount)
.s............/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/typed_passes.py:336: NumbaPerformanceWarning:
The keyword argument 'parallel=True' was specified but no transformation for parallel execution was possible.
To find out why, try turning on parallel diagnostics, see https://numba.readthedocs.io/en/stable/user/parallel.html#diagnostics for help.
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_stencils.py", line 521:
        """
        def test_impl(A):
        ^
  warnings.warn(errors.NumbaPerformanceWarning(msg,
./nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/ir_utils.py:2172: NumbaPendingDeprecationWarning:
Encountered the use of a type that is scheduled for deprecation: type 'reflected list' found for argument 'b' of function '__numba_stencil_0xffff6ab2a030_6'.
For more information visit https://numba.readthedocs.io/en/stable/reference/deprecation.html#deprecation-of-reflection-for-list-and-set-types
File "<string>", line 1:
<source missing, REPL/exec in use?>
  warnings.warn(NumbaPendingDeprecationWarning(msg, loc=loc))
....ssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssssss
======================================================================
ERROR: test_wrapper_address_protocol_libm (numba.tests.test_function_type.TestFunctionTypeExtensions.test_wrapper_address_protocol_libm) [<object object at 0xffff6bb515d0>] (jit={'nopython': True})
Call cos and sinf from standard math library.
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_function_type.py", line 625, in test_wrapper_address_protocol_libm
    self.assertEqual(jit_(myeval)(mycos, 0.0), 1.0)
                     ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/dispatcher.py", line 423, in _compile_for_args
    error_rewrite(e, 'typing')
  File "/nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/core/dispatcher.py", line 364, in error_rewrite
    raise e.with_traceback(None)
numba.core.errors.TypingError: Failed in nopython mode pipeline (step: nopython frontend)
non-precise type pyobject
During: typing of argument at /nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_function_type.py (616)
File "../../nix/store/58drm3jnmp1mkw1xvijdl4rkwclblaz8-python3-3.12.4-env/lib/python3.12/site-packages/numba/tests/test_function_type.py", line 616:
        def myeval(f, x):
        ^
----------------------------------------------------------------------
Ran 10500 tests in 1300.432s
FAILED (errors=1, skipped=695, expected failures=22)

Ofborg's CI error with aarch64-linux is weird...

Good question... maybe it has just been flaky on hydra...


Adding vetur support

Vetur can provide autocomplete support for custom components through tag and attributes definition files and by linking to them under a "vetur" key in the package.json. See this page for more details.

Excellent work! Thank you, will publish new version soon.

@all-contributors please add @shadskii for code,doc

@all-contributors please add @shadskii for code and doc


feat(Plugins): remove connector_gcs app

Delete the connector_gcs app from plugins.
Changes
Please list / describe the changes in the codebase for the reviewer(s).

Change 1
Change 2

How/what to test
A few words about what needs to be tested, along with specific testing instructions if needed.
Screenshots / screencast
If relevant, please add some screenshots or a short video.

Step 1 (with migrations) should be deployed in prod before merging this


[Issue-6091] Fix the StandaloneServer that support Windows Index

fix #6091

Codecov Report

Merging #6101 (2df56b0) into dev (dc85e1a) will decrease coverage by 0.03%.
The diff coverage is n/a.

@@             Coverage Diff              @@
##                dev    #6101      +/-   ##
============================================
- Coverage     46.36%   46.33%   -0.04%
+ Complexity     3733     3732       -1
============================================
  Files           608      608
  Lines         25062    25062
  Branches       2866     2866
============================================
- Hits          11621    11613       -8
- Misses        12295    12305      +10
+ Partials       1146     1144       -2

Impacted Files
Coverage Δ

...r/plugin/registry/zookeeper/ZookeeperRegistry.java
39.00% <0.00%> (-5.68%)
:arrow_down:

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update dc85e1a...2df56b0. Read the comment docs.

@CalvinKirs PTAL

Fix the CI failures and it LGTM

@CalvinKirs Is this caused by your code?

@CalvinKirs Is this caused by your code?

Please update your code.

@CalvinKirs Is this caused by your code?

Please update your code.

I have updated it.

The recent code of StandAlone-Server has solved the problem of reading file paths caused by system inconsistencies, so this pr will be closed. :)


Fix apache logging limitation by using correct apache call.

Apache 2.4 brought the option to change the ErrorLogFormat. However, many
fields remain empty, as ModSecurity uses the wrong apache logging function.
This fixes this behaviour with the use of ap_log_rerror.

Hi @dune73 ,
Sent your patch to our buildbots last Friday. It seems like it won't compile to Nginx (32 or 64), same for IIS (Windows 7 and 8, 32 and 64 bits).
Look at the buildbot results here:

Nginx (Linux 32) - http://www.modsecurity.org/developers/buildbot/builders/Linux32 - Nginx (Mainline)/builds/180/steps/make nginx/logs/stdio
Ningx (Linux 64) - http://www.modsecurity.org/developers/buildbot/builders/Linux64 - Nginx (Mainline)/builds/198/steps/make nginx/logs/stdio
Windows 8 - vs2012 - http://www.modsecurity.org/developers/buildbot/builders/Win8 - (VS 2012 64b) - IIS/builds/152/steps/build modsec/logs/stdio

The complete list is available here:
http://www.modsecurity.org/developers/buildbot/waterfall
The problem seems to be the same:
undefined reference to `ap_log_rerror_'

We may need to define the method ap_log_rerror  in standalone as well, like ap_log_error is defined here:
https://github.com/SpiderLabs/ModSecurity/blob/4a49d5b8dc4346d7583c66be4579309d0259827c/standalone/server.c#L263-L286

OK. I'll try and sort this out. Please hold on.

thank you @dune73

So this took a while, but now I added the said declaration to standalone/server.c. NginX compiles for me now.

Thanks @dune73, I will have a look later today.

Thanks @dune73!
Patch merged!

The thanks is all yours, @zimmerle.

@rainerjung, @dune73 What does this mean?:

If you apply the patch, you might also want to remove the "[client %s] " part of your log format (and the corresponding ip address argument) since it is redudant to what ap_log_rerror() already writes out by default.

I'm using 2.9.1-RC1, specifically for this change, and I do notice it's now logging the client IP twice: once as part of the standard apache ErrorLogFormat (with text "client", client ip address and port), and ModSecurity seems to be logging it again in error log just before it's message (with text "client", client ip address and no port):
[Sun Feb 21 23:14:34.453580 2016] [:error] [pid 7979:tid<PHONE_NUMBER>86464] [client <IP_ADDRESS>:52788] [client <IP_ADDRESS>] ModSecurity: Access denied with code 403 (phase 2)...etc.
Now I can of course change this in ErrorLogFormat now thanks to this change, to stop logging the client ip address and port, but then this will remove it from **all **error logging and not just ModSecurity error logging. Surely this duplication should be removed in ModSecurity?
Not a massive deal, and if can't be fixed will live with it for this extra functionality, but a little confusing none-the-less.

This is odd.
There is redundancy in the default logformat, but I do not see a duplication like you. Here is an example from my 2.9.1-rc1 tests with default ErrorLogFormat:
[2016-02-22 05:13:42.442421] [-:error] <IP_ADDRESS>:59593 VsqK9n8AAQEAAB9HHJQAAAAA [client <IP_ADDRESS>] ModSecurity: Warning. Matched phrase "/etc/passwd" at ARGS:y. [file "/home/dune73/data/git/owasp-modsecurity-crs-dune73/rules/REQUEST-30-APPLICATION-ATTACK-LFI.conf"] [line "108"] [id "930120"] [rev "4"] [msg "OS File Access Attempt"] [data "Matched Data: /etc/passwd found within ARGS:y: /etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/3.0.0"] [maturity "9"] [accuracy "9"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-local file inclusion"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"] [hostname "localhost"] [uri "/index.html"] [unique_id "VsqK9n8AAQEAAB9HHJQAAAAA"]

But where is your 1st "[client ..." is coming from? Do you have a custom ErrorLogFormat configured?
I think Felipe did not want to remove the redundancy as it could kill existing installations which take the info out of the ModSec part of the log. But I can not really tell for sure. My patch was only about enabling apache to fill its part of the log.

Well I tried it both with a defined ErrorLogFormat (set to the same as the default) and then with no ErrorLogFormat specified at all.
It's interesting that your log snippet logs the client IP twice. Just the first time without the "client" label and without the square brackets. As these are the defaults in Apache log files I've got to ask - are sure you don't have a custom ErrorLogFormat defined? ;-)

Thanks for following up. Indeed, I edited the wrong config file. I can now confirm your finding:
[Mon Feb 22 09:29:19.502412 2016] [:error] [pid 13241:tid<PHONE_NUMBER>40096] [client <IP_ADDRESS>:33041] [client <IP_ADDRESS>] ModSecurity: ...
@zimmerle : What do we do with this? I suggest to remove the ModSec repetition of the client IP, but maybe that is risky as apache is adding the port number as well. Maybe we can cope with that by proposing a custom ErrorLogFormat in the CHANGES file which brings the apache part of the entry without the port number. I guess this is what @rainerjung proposed all along.

since 2.9.1-RC1 i get for each entry in the audit log the following error.
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
Is this a side effect of this change?

@MichaelHaas I do not think so. But then your problem was not there with 2.9.0 (?), so it must have to do with 2.9.1-rc1.
Could you provide more infos?

Apache Version / Build infos
Full ErrorLog Line (possibly more logs like a full request on LogLevel trace8)
Config File

Problem started with 2.9.1-rc1. It was not there with 2.7.7 or 2.9.0.
Apache 2.4.17 (tested also 2.4.18)
I only get this error when Loglevel is set >=error  so if i set it to crit the error/message is not there.
--a7c7313d-B--
GET /?select * from SYSTEM.TEST HTTP/1.1
User-Agent: curl/7.19.7 (x86_64-suse-linux-gnu) libcurl/7.19.7 OpenSSL/0.9.8j zlib/1.2.7 libidn/1.10
Host: xxxxx
Accept: /
--a7c7313d-F--
HTTP/1.1 403 Forbidden
Accept-Ranges: bytes
Connection: close
Content-Type: text/html
--a7c7313d-H--
Message: Match of "rx ^(?i:(?:[a-z]{3,10}\s+(?:\w{3,7}?://[\w\-\./](?::\d+)?)?/[^?#](?:\?[^#\s])?(?:#[\S])?|connect (?:\d{1,3}\.){3}\d{1,3}\.?(?::\d+)?|options \)\s+[\w\./]+|get /[^?#](?:\?[^#\s])?(?:#[\S])?)$" against "REQUEST_LINE" required. [file "xxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "52"] [id "960911"] [rev "2"] [msg "Invalid HTTP Request Line"] [data "GET /?select * from SYSTEM.TEST HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"]
Message: Match of "within %{tx.allowed_http_versions}" against "REQUEST_PROTOCOL" required. [file "xxxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "78"] [id "960034"] [rev "2"] [msg "HTTP protocol version is not allowed by policy"] [data "* from SYSTEM.TEST HTTP/1.1"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/PROTOCOL_NOT_ALLOWED"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A6"] [tag "PCI/6.5.10"]
Message: Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:960911-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ-REQUEST_LINE. [file "xxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "26"] [id "981176"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5, SQLi=1, XSS=0): Last Matched Message: HTTP protocol version is not allowed by policy"] [data "Last Matched Data: GET /?select * from SYSTEM.TEST HTTP/1.1"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "xxxxxx/conf/modsecurity_crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "37"] [id "981204"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5, SQLi=1, XSS=0): HTTP protocol version is not allowed by policy"]
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
Action: Intercepted (phase 2)
Stopwatch:<PHONE_NUMBER>799877 3072 (- - -)
Stopwatch2:<PHONE_NUMBER>799877 3072; combined=1597, p1=348, p2=1120, p3=0, p4=0, p5=129, sr=0, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.1-RC1 (http://www.modsecurity.org/); OWASP_CRS/2.2.8.
Server: Apache
Engine-Mode: "ENABLED"
--a7c7313d-Z--
If i set the loglevel to trace8 i get more errors/messages.
--585cd533-H--
Message: Match of "rx ^(?i:(?:[a-z]{3,10}\s+(?:\w{3,7}?://[\w\-\./](?::\d+)?)?/[^?#](?:\?[^#\s])?(?:#[\S])?|connect (?:\d{1,3}\.){3}\d{1,3}\.?(?::\d+)?|options \)\s+[\w\./]+|get /[^?#](?:\?[^#\s])?(?:#[\S])?)$" against "REQUEST_LINE" required. [file "xxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_20_protocol_violations.conf"] [line "52"] [id "960911"] [rev "2"] [msg "Invalid HTTP Request Line"] [data "GET /?select * from SYSTEM.TEST HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"]
Message: Match of "within %{tx.allowed_http_versions}" against "REQUEST_PROTOCOL" required. [file "xxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_30_http_policy.conf"] [line "78"] [id "960034"] [rev "2"] [msg "HTTP protocol version is not allowed by policy"] [data "* from SYSTEM.TEST HTTP/1.1"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.8"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/POLICY/PROTOCOL_NOT_ALLOWED"] [tag "WASCTC/WASC-21"] [tag "OWASP_TOP_10/A6"] [tag "PCI/6.5.10"]
Message: Access denied with code 403 (phase 2). Pattern match "(.*)" at TX:960911-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ-REQUEST_LINE. [file "xxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_49_inbound_blocking.conf"] [line "26"] [id "981176"] [msg "Inbound Anomaly Score Exceeded (Total Score: 5, SQLi=1, XSS=0): Last Matched Message: HTTP protocol version is not allowed by policy"] [data "Last Matched Data: GET /?select * from SYSTEM.TEST HTTP/1.1"]
Message: Warning. Operator GE matched 5 at TX:inbound_anomaly_score. [file "xxxxxx/apache_test/conf/modsecurity_crs/activated_rules/modsecurity_crs_60_correlation.conf"] [line "37"] [id "981204"] [msg "Inbound Anomaly Score Exceeded (Total Inbound Score: 5, SQLi=1, XSS=0): HTTP protocol version is not allowed by policy"]
Apache-Error: [file "ssl_engine_kernel.c"] [line 245] [level 7] AH02034: %s HTTPS request received for child %ld (server %s)
Apache-Error: [file "http_request.c"] [line 322] [level 11] Headers received from client:
Apache-Error: [file "http_request.c"] [line 326] [level 11]   %s: %s
Apache-Error: [file "http_request.c"] [line 326] [level 11]   %s: %s
Apache-Error: [file "http_request.c"] [line 326] [level 11]   %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "request.c"] [line 293] [level 10] request authorized without authentication by access_checker_ex hook: %s
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
Apache-Error: [file "request.c"] [line 349] [level 10] fixups hook gave %d: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "request.c"] [line 293] [level 10] request authorized without authentication by access_checker_ex hook: %s
Apache-Error: [file "http_filters.c"] [line 1006] [level 10] Response sent with status %d%s
Apache-Error: [file "http_filters.c"] [line 1013] [level 12]   Date: %s
Apache-Error: [file "http_filters.c"] [line 1016] [level 12]   Server: %s
Apache-Error: [file "http_filters.c"] [line 835] [level 11]   %s: %s
Apache-Error: [file "http_filters.c"] [line 835] [level 11]   %s: %s
Apache-Error: [file "http_filters.c"] [line 835] [level 11]   %s: %s
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
Configure option are normal i think.
./configure
--prefix=$installdir
--with-apxs=$apachedir/bin/apxs
--with-apr=$apachedir/bin/apr-1-config
--with-apu=$apachedir/bin/apu-1-config
--disable-mlogc
--with-curl=no
You don't get these Messages at all with 2.9.2rc1?

I changed  ap_log_rerror(APLOG_MARK, APLOG_ERR | APLOG_NOERRNO, 0, r, back to  ap_log_error(APLOG_MARK, APLOG_ERR | APLOG_NOERRNO, 0, r->server,
and now the message is gone.

Too bad.
I planned to do exactly that, but a customer project keeps me busy.
The patch was built a long time ago and I do not have the logs ready. So I can not really tell what happened when I enabled it. Based on your report and given the ModSec dev community meeting talking about 2.9.1 release will be tomorrow (care to join?) it might be best to pull this change out of the release. After all it's a final release for 2.9 and we do not have the option to fix it in an eventual 2.9.2.

Disappointing, and didn't see that error myself but probably right call since there are a few outstanding questions and getting late in the day.
I was looking for this change to start logging the real IP address (in X-FORWARDED-FOR header) in error logs as our Apache sits behind a load balancer. Any other thoughts on how I could do this?
Thanks,
Barry

On 1 Mar 2016, at 13:44, Christian Folini<EMAIL_ADDRESS>wrote:
Too bad.
I planned to do exactly that, but a customer project keeps me busy.
The patch was built a long time ago and I do not have the logs ready. So I can not really tell what happened when I enabled it. Based on your report and given the ModSec dev community meeting talking about 2.9.1 release will be tomorrow (care to join?) it might be best to pull this change out of the release. After all it's a final release for 2.9 and we do not have the option to fix it in an eventual 2.9.2.
—
Reply to this email directly or view it on GitHub.

Seems like i'm not the only one  issue #1073 was opened by @lifeforms.
Have tried to change the ap_log_rerror statement and removed all %s but it didn't help. I think it's best to revert the change because i'm not a programmer and can't fix it.

Actually you are correct. That is there in the audit logs - hadn't noticed this before as thought you meant it wasn't logging the real error.
You can fix this by commenting out or removing line 1989 of msc_logging.c to stop logging this to the audit log since it really is a useless message:
    for(i = 0; i < msr->error_messages->nelts; i++) {
        error_message_t *em = (((error_message_t **)msr->error_messages->elts)[i]);
        text = apr_psprintf(msr->mp, "Apache-Error: %s\n",
            format_error_log_message(msr->mp, em));
           /* sec_auditlog_write(msr, text, strlen(text)); */
    }

I would submit a pull request but not that familiar with Git to be honest!
Looks like this was always there but is only now being called as a side effect to this release.

Hold on, you are suppressing the Apache-Error messages from the audit log, are not you? Also the ones which are correctly formatted.

wouldn't this remove all Apache-Error messages?
Sometimes they are helpful, for example if apache denied a request and you have it logged in the audit log you see something like this
Apache-Error: [file "mod_authz_host.c"] [line 311] [level 3] client denied by server configuration: ...
So you can see immediately that apache denied the request.

Hmmm yes it would. A little bit of knowledge is a dangerous thing!
Looks like they are not correctly formatted anyway. For example:
Apache-Error: [file "core.c"] [line 4302] [level 3] AH00126: Invalid URI in request %s

But if people find them useful then I'm easy either way. Leave in and ignore this extra ModSecurity error, or exclude and lose all the messages. Personally I'd probably leave in.

So I sat down and tried to nail this occurrence of "%s" artifacts in the auditlog. I was able to isolate them from this pull request.
Here is the audit log of ModSecurity 2.7.5 on Apache 2.4.18:

		  --30a40174-H--
		  Message: Warning. Pattern match "(?:(?

AFAIK it was always there for the other modules - just not for ModSecurity, but that's not what @MichaelHaas is raising. There is one extra line now for the ModSecurity error in the audit log.
So, prior to 2.9.1-RC1 it used to say:
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "core.c"] [line 4433] [level 6] AH00128: File does not exist: %s

with the %s not being filled in.
and now it says:
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "mod_authz_core.c"] [line 809] [level 7] AH01626: authorization result of %s: %s
Apache-Error: [file "core.c"] [line 4433] [level 6] AH00128: File does not exist: %s
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s

with the %s not being filled in and an extra line for ModSecurity now, since it is now going through the proper apache error logging thanks to this change.
To me if you're dealing with the other messages already then one extra message for ModSecurity is not a big deal. But that's just me, because I never noticed them or used them before (and hence didn't even see what @MichaelHaas was talking about initially!). And because I want this change for other reasons :-)

These are new
Apache-Error: [file "apache2_util.c"] [line 271] [level 3] [client %s] ModSecurity: %s%s [uri "%s"]%s
I don't have them in the audit log before 2.9.1.
I found now some other with 2.9.0
Apache-Error: [file "proxy_util.c"] [line 3366] [level 4] AH01092: no HTTP 0.9 request (with no host line) on incoming request and preserve host set forcing hostname to be %s for uri %s
I think the problem with the new one is that it logs them with each message so you have them always in the log.
But if it's only a problem for me i can live with it ;).

I think i didn't see them in my logs because for example this messages is only shown when you are not denying the request.
Apache-Error: [file "core.c"] [line 4433] [level 6] AH00128: File does not exist: %s
If mod_security is blocking the request then you don't see them.

Depends on the phase and the request.
For example ModSecurity can actually **start **to block based on these messages - see rule 981227 in OWASP CRS for example:
#
# Identify Invalid URIs Blocked by Apache
#
# -=[ Rule Logic ]=-
#
# There are some request violations that Apache will handle internally, prior to the
# ModSecurity phase:1 POST-READ-REQUEST hook.  For these requests, we can still get
# visibility by running a check in phase:5 logging to look for the Apache error msg.
#
# -=[ References ]=-
#
SecRule WEBSERVER_ERROR_LOG "@contains Invalid URI in request" \
  "msg:'Apache Error: Invalid URI in Request.', \
  severity:'4', \
  id:'981227', \
  ver:'OWASP_CRS/2.2.9', \
  rev:'1', \
  maturity:'9', \
  accuracy:'9', \
  logdata:'%{request_line}', \
  phase:5, \
  pass, \
  t:none, \
  tag:'OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ', \
  tag:'CAPEC-272', \
  setvar:'tx.msg=%{rule.msg}', \
  setvar:tx.anomaly_score=+%{tx.notice_anomaly_score}, \
  setvar:'tx.%{rule.id}-OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ-%{matched_var_name}=%{matched_var}'"

Which can result in this:
--8896924b-H--
Message: Warning. String match "Invalid URI in request" at WEBSERVER_ERROR_LOG. [file "/www/apache24/conf/modsecurity.d/crs/owasp-modsecurity-crs/base_rules/modsecurity_crs_20_protocol_violations.conf"] [line "82"] [id "981227"] [rev "1"] [msg "Apache Error: Invalid URI in Request."] [data "GET //../../../../../../../../../../../../../../winnt/win.ini HTTP/1.1"] [severity "WARNING"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/PROTOCOL_VIOLATION/INVALID_REQ"] [tag "CAPEC-272"]
Apache-Error: [file "core.c"] [line 4302] [level 3] AH00126: Invalid URI in request %s
Stopwatch:<PHONE_NUMBER>814065 1162 (- - -)
Stopwatch2:<PHONE_NUMBER>814065 1162; combined=548, p1=236, p2=0, p3=14, p4=83, p5=215, sr=0, sw=0, l=0, gc=0
Producer: ModSecurity for Apache/2.9.0 (http://www.modsecurity.org/); OWASP_CRS/2.2.9.
Server: Apache
Engine-Mode: "ENABLED"

Also proxy errors can be intercepted to prevent data leakage:
Message: Access denied with code 403 (phase 4). Pattern match "^5\\d{2}$" at RESPONSE_STATUS. [file "/www/apache24/conf/modsecurity.d/crs/owasp-modsecurity-crs/base_rules/modsecurity_crs_50_outbound.conf"] [line "53"] [id "970901"] [rev "2"] [msg "The application is not available"] [data "Matched Data: 502 found within RESPONSE_STATUS: 502"] [severity "ERROR"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "WASCTC/WASC-13"] [tag "OWASP_TOP_10/A6"] [tag "PCI/6.5.6"]
Message: Warning. Operator GE matched 4 at TX:outbound_anomaly_score. [file "/www/apache24/conf/modsecurity.d/crs/owasp-modsecurity-crs/base_rules/modsecurity_crs_60_correlation.conf"] [line "40"] [id "981205"] [msg "Outbound Anomaly Score Exceeded (score 4): The application is not available"]
Apache-Error: [file "mod_proxy_http.c"] [line 1310] [level 3] [status 70007] AH01102: error reading status line from remote server %s:%d
Apache-Error: [file "proxy_util.c"] [line 397] [level 3] AH00898: %s returned by %s
Action: Intercepted (phase 4)

But yeah I agree it's not common to see these errors in the Audit Log and now it is.
Changing line 1989 of msc_logging.c to below seems to block the ModSecurity messages only:
    /* Apache error messages */
    for(i = 0; i < msr->error_messages->nelts; i++) {
        error_message_t *em = (((error_message_t **)msr->error_messages->elts)[i]);
        text = apr_psprintf(msr->mp, "Apache-Error: %s\n",
            format_error_log_message(msr->mp, em));
        /* Don't log ModSecurity messages themselves */
        if(strstr(text, "ModSecurity") == NULL) {
            sec_auditlog_write(msr, text, strlen(text));
        }
    }

But a bit of a hack to just search for ModSecurity and I've not coded in C for many a year so not sure if someone with more knowledge can think of a better way...

Hi,
If I understood correctly, the '%s' in the H section was not being filled correctly even before that patch. Therefore, the only two problems of this patch/pull request are:

Double client entry on the logs: [client <IP_ADDRESS>:33041] [client <IP_ADDRESS>]
An extra ModSecurity line in the "H" section of the auditlogs

Is that right? If so, I would say that we don't have any major bug, rather, just features that needs some improvements/polishment.
The "double client entry" problem can be circumvented by the correct configurations, right @dune73? The extra line in the section "H" is just an extra data, that might not be a problem for the most of the users.
I can remove the patch, but I would prefer not to modify anything on the patch, as it may lead us to a -RC2 version. As we want to have a final release, my suggestion is to release the 2.9.1 as it is on -RC1 and as part of 2.9.2-RC1 we will take into consideration the modifications to suppress that extra message and to remove the double client entry. How that sounds?

You have summed it up well and I would be keen to keep this in.
In the interests of honesty though I will point out two issues with your suggestion that we can remove the "double client entry" through configuration:

This is not available to Apache 2.2 users (the ErrorLogFormat option was only added in Apache 2.4).
The only option to do this is to remove the default Apache logging of [ip:port] using a custom ErrorLogFormat but this will remove that from **all **error logs and not just ModSecurity error logs. Which is unlikely to be desirable.

So realistically I don't think there is a work around for the "double client entry" problem if that does cause anyone any issues.
However personally I would still agree with the statement that "we don't have any major bug, rather, just features that needs some improvements/polishment" and would vote to keep this in.

Hi guys, sorry for taking weeks to actually understand what the deal is.
I think Felipe summed it up nicely. And while I think it won't win a beauty contest with #840, it's better than before. But let's say this allows for improvement in eventual 2.9.2.

Hi Guys, thanks for the feedback.
I was looking to the issue and I managed to make a very simple patch: I simple removed the client entry. If you guys have time to test it, it will be very good. I have two releases ready: one with the patch and another one without it. So, if you manage to test it on time, and there is no other issue, we can release 2.9.1 without the double-client-entry problem.
Here is the branch with the patch applied:
https://github.com/SpiderLabs/ModSecurity/tree/release_2.9.1-final

Hold on. This does not work.
2.9.1-final compiles just right and my quick tests blocked correctly. However, the logformat is messed up:
2.9.1rc1 on apache 2.4.8, Ubuntu 14.04, default ErrorLogFormat
[Wed Mar 09 15:36:26.497222 2016] [:error] [pid 21821:tid<PHONE_NUMBER>18912] [client <IP_ADDRESS>:37688] [client <IP_ADDRESS>] ModSecurity: Warning. Pattern match "(?:(?<!\\\\w)(?:\\\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\\\.asa|httpd\\\\.conf|boot\\\\.ini)\\\\b|\\\\/etc\\\\/)" at ARGS:a. [file "/core-rules/modsecurity_crs_40_generic_attacks.conf"] [line "205"] [id "950005"] [rev "3"] [msg "Remote File Access Attempt"] [data "Matched Data: /etc/ found within ARGS:a: /etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"] [hostname "localhost"] [uri "/index.html"] [unique_id "VuA06n8AAQEAAFU9o3IAAAAA"]

2.9.1-final on apache 2.4.8, Ubuntu 14.04, default ErrorLogFormat:
[Wed Mar 09 15:37:12.657681 2016] [:error] [pid 21955:tid<PHONE_NUMBER>52032] [client <IP_ADDRESS>:37697] ModSecurity: <IP_ADDRESS>Warning. Pattern match "(?:(?<!\\\\w)(?:\\\\.(?:ht(?:access|passwd|group)|www_?acl)|global\\\\.asa|httpd\\\\.conf|boot\\\\.ini)\\\\b|\\\\/etc\\\\/)" at ARGS:a. [file "/core-rules/modsecurity_crs_40_generic_attacks.conf"] [line "205"] [id "950005"] [rev "3"] [msg "Remote File Access Attempt"] [data "Matched Data: /etc/ found within ARGS:a: /etc/passwd"] [severity "CRITICAL"] [ver "OWASP_CRS/2.2.9"] [maturity "9"] [accuracy "9"] [tag "OWASP_CRS/WEB_ATTACK/FILE_INJECTION"] [tag "WASCTC/WASC-33"] [tag "OWASP_TOP_10/A4"] [tag "PCI/6.5.4"] [uri " [hostname "localhost"]"]/index.html

Sorry for the bad news.

My bad. It is now fixed. The fixed version is on the same branch. Do you mind to test it again?

Right, I am cooking the final release without the fix. It will be integrated as part of v2.9.2

Can someone please summarize what exactly if any are the changes to the format of the audit log from this patch? Would anyone who has consumers of the existing audit log format now break after this patch?

The patch is aimed at the error-log and fixes a bug, where apache did not get the correct information from ModSec and some fields in the default apache ErrorLogFormat were thus empty. They are now filled.
Because of this, there is a duplication of the client ip field, which is now present in the apache-part of the errorlog line and also in the modsecurity-part of the errorlog line. Attempts to fix this did not succeed during the test phase and 2.9.1 was thus released with this duplication.
While testing, it was also discovered that there is an additional line in the "H"-Parts of the Audit-Log and this line has some tags of the strformat string not replaced with the values (-> %s and the like). We could prove that this misbehaviour has been in the Audit-Log for a long time. What is new is the additional log line.
So we believe this is save for consumption, but of course, nobody can guarantee you that. And depending on the consumer it could fail - at least in theory.
If you are unsure, I suggest you pick up on one of the audit log examples above and fill it into the consumer in question. Then you should know.
Sorry for the inconvenience. This is not what I had in mind, when I filed the pull request (after testing it in production for a few months).


Request: Add "saveMessages" option to capture Midjourney prompts

I only use ddg for Midjourney. I only have it connected to channels that are used exclusively for Midjourney's bot.
Midjourney's file names begin with username, then part of the prompt, and then end with a job ID, but the prompt will be abbreviated to make room for the job ID-- it will also be completely stripped of URLs for image prompts.
However, the image attachment is attached to a message (I think always) that contains the full Midjourney prompt-- having an option to save the content of this message into a text file (.prompt?) that has the exact same name/location as the image file attached to it would be a really big help. This might make sense, also, for non-Midjourney users.
As a technical note, Midjourney updates the in-progress image using .webp, so I filtered the extension out so that I only get the finished .png-- it would make sense (perhaps as an option) to only save the message when really saving the attachment.
As it is now, often when I'm working, Midge will give me ephemeral messages for my images, so if I do almost anything, they disappear from Discord-- I also find it comes up a LOT that when I'm going through my images, that I want to quickly know what the whole prompt was. I may have hundreds of nearly-identical images with just slightly different prompts, but the differences in the prompts are truncated in the image name. I can dig them out of the archive, but that adds steps if we're not dealing with the most-recent images.
Where this is coming up a lot now is with the release of v6, I find I am going back to re-run prompts I ran under 5.2. I find these by flipping through the images in the file system, but I have 11,000+ images-- if I want to find the prompt now (particularly if I want to see image prompts), I have to use the job ID off the image file, then show the job from Midjourney in Discord. That might sound simple, but if I want to see a bunch of related prompts at once (to see how they/if they differ), I can't do it in any easy way.
Very advanced:
Save image prompts when the URL isn't recognized in the current directory. For example, if I have https://s.mj.run/oV1fLNrILibD as part of my prompt, look for https://s.mj.run/oV1fLNrILibD in the directory where the image would go. If it's not there, save that image, using a standard filesystem-friendly name (silly example): https£¤¤s.mj.run¤oV1fLNrILibD.

I think I've added an okay solution for what you're needing. https://github.com/get-got/discord-downloader-go/commit/f9727c2267a348a57a8b50dbd3a5f0b59edc1cdf & https://github.com/get-got/discord-downloader-go/commit/57675470e2a31d559ede438de4f3af139d34bf49
I added a "content" setting to logLinks and logMessages. This currently provides no actual utility for logMessages, but in the case of logLinks, it defaults to "{{link}}" but accepts {{msgContent}} or {{embedDesc}} (since I recall Midjourney images/prompts are embedded), you should be able to use the settings below to accomplish what you're wanting.
logLinks is called after the download request finishes, because part of the original intention was to log download failures and whatnot. It also takes the same data keys as downloads, so we can use this to mirror your actual download.
"logLinks": {
    "destination": "*match downloads*",
    "subfolders": [ "*match downloads*" ],
    "filenameFormat": "{{downloadFilename}}.prompt",
    "prefix": "",
    "suffix": "",
    "content": "{{embedDesc}}",
    "logFailures": false
}

should result in
file_A.jpg
file_A.prompt - "Aaaaaaa aaaaaaaaa aaaaaaaa"
file_B.jpg
file_B.prompt - "Bbbbbbb bbbbbbbb bbbbbbbb"

Well, that sounds like a pretty decent solution. Are you going to push an update?

Given that tens of thousands of images have already been downloaded, how would this handle the history part for prompts?
I don't want to redownload the images, of course, but I would like allllllllll the prompts.

Close your active instance of the bot, copy your active settings file to a temporary backup file, change the settings.json to use "save": false, and run the history query you're needing. It'll reprocess everything, logging and whatnot, but won't save files.

I didn't fully get the instructions for capturing the log stuff.
The sample XML you gave-- where should that go? Is that inside the channel?
Secondly, does it save the prompts as you described-- alongside the images?
(I'm being extra cautious, given how many files there are).

logLinks and logMessages exist globally and per-source. For your uses here, it should be used per-source (inside the channel) as formatted above.
Something like this will achieve the result you're wanting. Matching the destination and subfolders will cause them to be downloaded alongside the images.


[BUG]: If authorization proxy server is installed in an alternate namespace by csm-operator, the deployment fails

Bug Description
If authorization proxy server is installed in an alternate namespace by csm-operator, the deployment fails. This is because there are a couple of hardcoded namespaces in one of the config files.
Logs
No logs for this one
Screenshots
No response
Additional Environment Information
No response
Steps to Reproduce
Try to install auth proxy server in an alternate namespace with csm-operator v1.4.2 or earlier, and the install will fail.
Expected Behavior
The authorization proxy server should be installed in an alternate namespace if the customer desires.
CSM Driver(s)
NA
Installation Type
CSM-Operator v1.4.2
Container Storage Modules Enabled
Authorization v1.9.1
Container Orchestrator
Kubernetes v1.27.2
Operating System
RHEL 8.9

Fixed in above PR


Allow intra-workspace links to open files

Hello,
I work at a company where we have an internal code visualization application as a static HTML file. This is located inside a code repo that engineers check out to their workstations. It's most convenient for us to view this page within VSCode using this extension, which works really well. However, it would be even more useful if it was possible to have links in the static HTML file that, when clicked, would open a file within the repository, while still staying within VSCode. Is this possible somehow...?
Thanks!

If you enable server logging (Start Server Logging), then it will give you links that you can click on whenever it serves a file. It should follow whatever configuration you have for clicking on links in the terminal, so alt+click or ctrl+click to open a link.
https://github.com/microsoft/vscode-livepreview?tab=readme-ov-file#persistent-server-task-with-server-logging

Ah, that's useful, thank you! It seems to only open up the file in the same browser tab, though -- I was hoping it'd be possible to open up a file using a normal VSCode editor tab. I think I need to investigate the vscode:// URL handler, which seems out of scope for an issue filed against this extension. Closing.

oh, does it not open the editor?

When I ctrl+click on this, it opens the file in the editor. Is it not the same for you?


Add Ruby 2.5 support

Hi there!
Accordingly, to this commit https://github.com/ruby/ruby/commit/0078e40115fd53e2cafd4e8cfb6ff5fd7f87c3db, multiline headers are no more acceptable.  It's a pity, but the library generates User-Agent  with OS version which contains trailing newline.
This PR solves the issue and add explicit Ruby 2.4 and 2.5 support to test against on CI.

I signed it!

Any resolution to this issue?

Hey @grant, sorry for bugging you! Any chance that it could be merged? A lot of production projects are suffering...


MSVS 2017 is recognized as 2015

Node Version: 8.4.0
NPM Version: 5.3.0
Platform: Windows 7
Compiler: MSVS 2017

I was trying to customize the Platform Toolset variable for a legacy project I'm building when I ran into a strange problem. Essentially I need to set each Visual Studio version to use the _xp variant of the toolset. Everything works except for MSVS 2017, which is recognized as MSVS 2015 (Because it sets it to v140_xp).
"conditions":
[
	[ "MSVS_VERSION == '2012'", { "msbuild_toolset": "v110_xp" }],
	[ "MSVS_VERSION == '2013'", { "msbuild_toolset": "v120_xp" }],
	[ "MSVS_VERSION == '2015'", { "msbuild_toolset": "v140_xp" }],
	[ "MSVS_VERSION == '2017'", { "msbuild_toolset": "v141_xp" }]
]

Doing
npm config set msvs_version 2017

and
node-gyp configure --msvs_version=2017

didn't work so I'm wondering if maybe this is something that isn't supported yet?

What does node-gyp -v print?  VS 2017 support was added in v3.6.0.

Hi, it prints out 3.6.2.

@dkrutsko node-gyp is using a compatibility trick to make GYP work since the version that's embeded doesn't have official VS2017 support:
https://github.com/nodejs/node-gyp/blob/7245415296c5f0a97e10dba6dafbaf2332a67578/lib/configure.js#L153-L160
Try patching that, it should work for you.
After we land https://github.com/nodejs/node-gyp/pull/1183 we should remove that kludge.

I have the same problem and can't make it work with 2015 either.
...File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\__init__.py", line 98, in Load     generator.CalculateVariables(default_variables, params)   File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\generator\msvs.py", line 1916, in CalculateVariables     generator_flags.get('msvs_version', 'auto'))   File "C:\Program Files\nodejs\node_modules\npm\node_modules\node-gyp\gyp\pylib\gyp\MSVSVersion.py", line 434, in SelectVisualStudioVersion     versions = _DetectVisualStudioVersions(version_map[version], 'e' in version) KeyError: '2017' gyp ERR! configure error gyp ERR! stack Error:gyp failed with exit code: 1 ...

So how to make node-gyp work with VS2017?

^ answered in #1364- @chen4393 needs to upgrade.

I have the same problem.
node-gyp configure --msvs_version=2017
But it still uses 2015 version, while I have 2017 version installed. So I got an error during node-gyp build:
...\MSVC\14.12.25827\include\variant(16): fatal error C1189: #error:  class template variant is only available w
ith C++17.

node-gyp v3.6.2 still doesn't seem to support VS2017.  Taking a quick look at the code: https://github.com/nodejs/node-gyp/blob/master/gyp/pylib/gyp/MSVSVersion.py
There's no reference of 2017, hence the key error someone called out above.

That file isn't used with VS 2017, the logic is here.

Please run node-gyp configure --verbose and check the output for
gyp verb find vs2017 Found installation at: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community
gyp verb find vs2017   - Found Microsoft.VisualStudio.Component.VC.Tools.x86.x64
gyp verb find vs2017   - Found Microsoft.VisualStudio.Component.Windows10SDK.16299.Desktop
gyp verb find vs2017   - Found Microsoft.VisualStudio.VC.MSBuild.Base
gyp verb find vs2017   - Using this installation with Windows 10 SDK
gyp verb find vs2017 using installation: C:\Program Files (x86)\Microsoft Visual Studio\2017\Community

VS2017 needs to have a few components installed to be able to compile C++, otherwise node-gyp will ignore it and try to find a another version with a C++ compiler. The easiest way to get those components is to install the "Desktop development with C++" workload.

I don't know what exactly the problem has the originator of the ticket mentioning "MSVS 2017, which is recognized as MSVS 2015", but I had a problem that NodeJS scripts call MSBuild which in turn calls build tools from VS 2015, despite I had VS 2017 installed on my machine.
The solution for me was to update my *.vcxproj file and specify there explicitly v141 as a PlatformToolset. And of course, MSBuild.exe must be available in the %PATH% from the VS 2017 directory. If the current ticket is about the same problem, please see my answer here as well: https://stackoverflow.com/a/51132581/4807875.

@dkrutsko Make sure that "msbuild.exe" in your %PATH% targets MSBuild from VS 2017, e.g. C:\Program Files (x86)\Microsoft Visual Studio\2017\Enterprise\MSBuild\15.0\Bin\MSBuild.exe. That worked for me.

My guess is that any issues here have probably been resolved with recent releases of node-gyp and the additional Windows work that's gone into it. Please open an new issue if there's something still to be taken care of.


test: Create an integration test for interfacing with the verification server

TL;DR
Create an integration test for interactions with the verification server.  The verification server can be mostly mocked.

This is going to need a positive and negative case, one where the exposure is verified, and one where it is not verified.

/assign @argetlam-cs

/assign @argetlam-cs

/open

/open

/reopen

https://github.com/google/exposure-notifications-server/pull/829 added the scenario of exposure is verified, but not the case of exposure not verified yet, so there is still some work left to do

#868 added the scenario for the exposure not yet verified

Cool
/close


Support change tracking of non-primitive types

ChangeTracker does not detect changes to properties of type Dictionary<,>. In provider for PostgreSQL database database column type hstore is mapped to .net type Dictionary<string,string> but this feature can not be used properly if we do not have change tracking for those properties. One solution would be to use ImmutableDictionary but we want to avoid that.
Steps to reproduce
public class HStoreContext : DbContext
{
   public DbSet<SomeEntity> SomeEntities { get; set; }
}

public class SomeEntity
{
   public int Id { get; set; }
   public Dictionary<string, string> Tags { get; set; }
}

using (HStoreContext ctx = CreateContext())
{
   var entity = new SomeEntity()
   {
       Tags = new Dictionary<string, string>()
   };
   ctx.SomeEntities.Add(entity);
   var num = ctx.SaveChanges();

   Assert.Equal(1, num);

   var entry = ctx.Entry(entity);
   Assert.Equal(EntityState.Unchanged, entry.State);

   entity.Tags.Add("kind", "new");
   ctx.ChangeTracker.DetectChanges();

   Assert.Equal(EntityState.Modified, entry.State); // This fails
}

Further technical details
EF Core version: 2.0.0
Database Provider: Microsoft.EntityFrameworkCore.PostgreSQL
Operating system:
IDE: Visual Studio 2017

@dejvid-smth The ability to extend equality checks beyond ".Equals" is something that is not implemented yet. We intended to do it as part of type conversions (#242), since it is often needed then, although it is really a different feature. It's also being tracked as part of type mapping here: #8537

Hmm... Unless I'm mistaken Dictionary's .Equals() should compare the items, so EF Core should properly detect changes in a dictionary's contents... @dejvid-smth I haven't had time yet to properly review and test your PR which requires this, but could you double-check what's happening?

I don't know how exactly ChangeTracker works but we are just updating the same instance of Dictionary and EntityEntry from ChangeTracker only tracks if we actually change instance so both OriginalValue and CurrentValue are pointing to the same instance of Dictionary and therefore are the same.
We would have to somehow make a copy of original Dictionary when EntityEntry's state is set to Unmodified and save that as OriginalValue in EntityEntry so we could later compare OriginalValue with CurrentValue.

@roji Pretty sure Dictionary just does reference equality on the dictionary instance.
@dejvid-smth Correct, the snapshot would need to have a copy of the dictionary.

@roji You will need to generate mappings with the appropriate ValueComparers for the types that Postgres supports. It's not clear to be that a deep snapshot/comparison is always the best choice for this kind of property depending on how it is used, but I think you would know better how people typically use these kinds of properties with Postgres and can therefore make a decision as to which is the appropriate behavior. Let's discuss if I'm missing anything.

Thanks @ajcvickers! I'll take a look at the relevant types and modify their mappings accordingly for 2.1. I think that for arrays and for hstore (a native Dictionary<string,string> PostgreSQL datatype) a deep value comparison is the right thing to do, but we'll see what users say.
For anyone following, I've opened https://github.com/npgsql/Npgsql.EntityFrameworkCore.PostgreSQL/issues/305 for array comprison behavior and https://github.com/npgsql/Npgsql.EntityFrameworkCore.PostgreSQL/issues/306 for hstore.

Just to report that I've value comparison for hstore seems to work perfectly... I'll be adding comparers to array and other relevant types for 2.1 as well.
Thanks!

@ajcvickers if you're interesting in seeing a complex/interesting comparer implementation, take a look at the array comparer, it will also illustrate some of the issues detailed in #11072.


first request doesen't work

I'm making a request to this site: https://www.slamjam.com and i get the waiting page but the second time that I make the same request in the same session i get response 200
There is an example of my code:
header={
'authority': 'www.slamjam.com',
'scheme': 'https',
'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.9',
'accept-encoding': 'gzip, deflate, br',
'accept-language': 'it-IT,it;q=0.9,en-US;q=0.8,en;q=0.7',
'cache-control': 'max-age=0',
'sec-fetch-dest': 'document',
'sec-fetch-mode': 'navigate',
'sec-fetch-site': 'same-origin',
'sec-fetch-user': '?1',
'upgrade-insecure-requests': '1',
'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36',
}
session=request.Session()
s=cloudscraper.create_scraper(sess=session, browser={'browser': 'chrome', 'mobile': False}, interpreter='nodejs')
t=0
while t == 0:
r=s.get('https://www.slamjam.com', headers=header)
print(r)
if r.status_code == 200:
t=1

Ok....  lets get a couple simple things out of the way first and get you a little educated..
HTTP/2 Pseudo Headers

authority
scheme

The above are http/2 pseudo headers, they are not part of the http/1.1 framework .. https://http2.github.io/http2-spec/#rfc.section.<IP_ADDRESS> , there is ZERO requirement for them unless you're using h/2
Fundamental Code Structure Problems
Onto the next lot of problems...
FWIW: You are setting everything up wrong .. if you RTFM you would understand that a more viable approach would have been something more along the lines of..
import cloudscraper

s=cloudscraper.create_scraper(
    browser={
        'custom': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/80.0.3987.132 Safari/537.36'
    }
)

while True:
    r = s.get('https://www.slamjam.com')
    print(r)
    if r.status_code == 200:
        break

Lastly: Ordered Dictionaries
Cloudflare expects its headers to be in a specific order, by using a standard dictionary it does not hold its order, you need to use an OrderedDict from collections... but the create_scraper does this all for you....

got it, thank you very much, I didn't mean to sound rude sorry

No , you sound rude (well.. you didnt say hi or nothign ;P), its more the fact that I'm not stackoverflow... instead its time out of my day to respond to tickets because people don't bother to read the documentation 🙉, and yea that gets frustrating.
But anyway... I'm really glad its working for you mate, least you can move forward 😄, enjoy the rest of your day ❤


Create HttpBlog.md

Tagging @lukehoban @mehzer @joeduffy .  Feel free to tear apart.  Also, this is just a PR.  So if you feel strongly about anything, just go in and make the edits yourself :)   We can treat this as a collaborative document.
Thanks!

lgtm. i'll add closing links to serverless bits and pieces at the bottom before publish.

Hey Cyrus, I put some minor editing changes up there in a commit.  All of the changes were pretty nit so please feel free to ignore.  There were a couple of mispellings I probably should have just called out separately.  It is a really good post, game changing stuff!

Oh no.  Please just correct mispellings as you see them.  it's much faster than having to write them up and have someone else go edit it :D
Thanks!

Closing out as Marc has moved to appropriate online location.


jobDoc.pJobId is memset to zero before dispatching to OtaAppCallback

Probably can memset the jobId after calling OtaAppCallback? As user application unable to retrieve the jobId.
static OtaErr_t processDataHandler( const OtaEventData_t * pEventData )
{
    // ...

    /* Set the job id and length from OTA context. */
    jobDoc.pJobId = otaAgent.pActiveJobName;
    jobDoc.jobIdLength = strlen( ( const char * ) otaAgent.pActiveJobName ) + 1U;
    jobDoc.fileTypeId = otaAgent.fileContext.fileType;

    if( result == IngestResultFileComplete )
    {
        /* Check if this is firmware update. */
        if( otaAgent.fileContext.fileType == configOTA_FIRMWARE_UPDATE_FILE_TYPE_ID )
        {
            jobDoc.status = JobStatusInProgress;
            jobDoc.reason = JobReasonSigCheckPassed;

            otaJobEvent = OtaJobEventActivate;
        }
        else
        {
            jobDoc.status = JobStatusSucceeded;
            jobDoc.reason = JobReasonAccepted;
            jobDoc.subReason = ( int32_t ) otaAgent.fileContext.fileType;

            otaJobEvent = OtaJobEventUpdateComplete;
        }

        /* File receive is complete and authenticated. Update the job status. */
        err = otaControlInterface.updateJobStatus( &otaAgent, jobDoc.status, jobDoc.reason, jobDoc.subReason );

        // otaAgent.pActiveJobName set to 0 before dispatch to OtaAppCallback
        dataHandlerCleanup();

        if( otaAgent.statistics.otaPacketsProcessed < UINT32_MAX )
        {
            /* Last file block processed, increment the statistics. */
            otaAgent.statistics.otaPacketsProcessed++;
        }

        /* Let main application know that update is complete */
        otaAgent.OtaAppCallback( otaJobEvent, &jobDoc );

Thank you for reporting this issue.  It looks like some adjustments can be made here.  We are taking a look at it.

Pr to move the memset was merged here: https://github.com/aws/ota-for-aws-iot-embedded-sdk/pull/419

done a quick test, and able to get the jobID back. Thanks for the prompt response!


write docs

Now that https://dux.cldellow.com/ is a thing, what should go there?

[x] Introduction

[x] Getting started

[x] Quick start

[x] how to install datasette and dux
[x] yolo mode

[x] Getting help

[x] Submit an issue
[x] Ask on Discord

[x] Installation

[x] Minimum requirements
[x] WAL mode

[x] Publish to the web

[x] Walk through how to set up Fly with a mutable database
[x] Enable backups (maybe we need to write a datasette-cloud-backups plugin?)

[x] Configuration

[x] Authentication, authorization, auditing
[x] SQL schema support
[x] metadata.json
[x] edit-row.html

[ ] Use case guides (these should also explain useful patterns for setting up auth, eg github-auth-plugin, maybe google-auth-plugin)

[ ] Traditional data entry
[ ] Turker mode
[ ] Form mode

[ ] API Reference

[x] Hooks

[x] edit_control hook
[x] redirect_after_save hook

[x] Built-in edit controls

[x] StringControl
[x] StringAutocompleteControl
[x] all the rest

[x] Endpoints

[x] dux-autosuggest-column

[x] Contributing

[x] Architecture

[x] Relationship to Datasette
[x] How we communicate data to JavaScript
[x] JavaScript and CSS extensions
[x] The dux_column_stats index tables

[x] How to contribute

[x] How to set up env
[x] How to run tests
[ ] How to run Cypress tests

Will add use case guides and Cypress once we have more features / Cypress integrated


Issues labels

Hello,
I saw that you don't really use badges for your issues.
I think you should use them to classify these issues and then help potential contributors (like me) to find issues they can solve with categories.

You are probably right.

I felt the same. 2 Years later there are around 20/200 issues tagged. I would be glad to contribute in your project as well.


Indicators colors

Hello,
Thank you very much for sharing this, i am using 4EMA in my strategy, it displays all the indicators as needed, i just would like to know how to change the colors of each indicator ?
Any help please ?
Many Thanks.

Hi, at the moment this is not possible since all colors are automativally set by the chart.

:+1 for features


add an input

I think it would be interesting to add an entry to this node so that it can be triggered with any event.

added