NeoPixel RGBW Support

Adafruit is selling a new NeoPixels strip that have 4 LEDs in them (RGB + W). What changes need to be made to BiblioPixel to support these strips?
https://www.adafruit.com/products/2842

Sorry, I responded to this via email and it should've gone here automatically but that didn't happen for some reason... in case you missed it:
So... it's possible, but I'm going to guess it would be complicated.
The problem is that normal strips use 3 bytes per pixel and BiblioPixel is VERY wrapped around that fact, but the RGBW strips use 4 bytes per pixel.
Most of the time, new strips mean just a new driver... but all the drivers assume that what they are getting IN from BiblioPixel is 3 bytes per pixel and then they can output whatever they want. So I can't just update the driver. I have to update the entire framework.
Second, BiblioPixel doesn't have a direct driver for the NeoPixel chipset... it's actually SK6812 in the RGBW variety instead of WS2812. So I assume you are talking about running through our AllPixel driver board. And that definitely doesn't support RGBW. It would require new firmware... but first it would require the library we use (FastLED) to support RGBW in the first place.
Short answer... it's something I'm going to look into, but it's going to be a long time until RGBW is supported, if ever.
Sorry it's not better news.

Do you have any news regarding RGBW, as FastLED was building support for since last year: https://github.com/FastLED/FastLED/issues/106

It's unfortunately going to be a fairly long time before we do RGBW - even
though this is on my personal list of things I desperately want to do.
The issue is simple - in more than a few many places, we have unfortunately
hardcoded the fact that pixels have exactly three components, R, G and B.
This not only prevents us from doing RGBW now, it also makes working with
monochromatic lights a bit annoying (though it's possible).
It won't be a huge amount of work, but it also won't be a tiny amount of
work.  I do plan to do this but realistically it won't be till early next
year or so.  More interest from users would definitely change our
priorities regarding this issue.
I filed an issue just for this -
https://github.com/ManiacalLabs/BiblioPixel/issues/366.
Might I ask - which hardware are you using or planning to use?  Is it the
NeoPixels, or something else?
Just for the entertainment of readers, I had a chance to chat to a
professional who specializes in just this.  Apparently, there are at least
these additional possible colors that are available commercially.

W: white
A: amber (used to get "natural" light, more like an incandescent)
L: lime (basically the same)
U: ultraviolet (for black lights!)

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

We've built a rather large big one with 1600 NeoPixels RGBW  last year - should be the SK controller but WS2812 protocol works. At that time we've thought RGBW will be a could idea as it's only an other initialisation for the Arduino library. But I have to admit we were a bit to naive.
Right ow the stripe is controlled with an ATMega 2560 via a W5100 network Interface for http. We currently only display solid colors. Animating the colors depending on setting was our goal. If the interface is http, udp oder mqtt is not important as we hook it up to our home automation system via node-red.
I'm searching for an solution to either drive the Pixel "as one" (Bibliopixel on RPi Zero with 74HCT125) or segment them in smaller lenghts (ESP8266) and sync (optional) these in any way.
But all thing's I'm aware of have no RGBW support... Maybe you can point me in a good direction :-)

As far as I know, no one has four component color support yet.  :-/  I did
a bunch of searching at the start of the year for this, and found nothing.
On the other hand, this means if we did get it working on BP, we'd be "best
of show".  :-D
On Tue, Aug 15, 2017 at 11:38 AM, Patrik Mayer<EMAIL_ADDRESS>wrote:

We've built a rather large big one with 1600 NeoPixels RGBW last year -
should be the SK controller but WS2812 protocol works. At that time we've
thought RGBW will be a could idea as it's only an other initialisation for
the Arduino library. But I have to admit we were a bit to naive.
Right ow the stripe is controlled with an ATMega 2560 via a W5100 network
Interface for http. We currently only display solid colors. Animating the
colors depending on setting was our goal. If the interface is http, udp
oder mqtt is not important as we hook it up to our home automation system
via node-red.
I'm searching for an solution to either drive the Pixel "as one"
(Bibliopixel on RPi Zero with 74HCT125) or segment them in smaller lenghts
(ESP8266) and sync (optional) these in any way.
But all thing's I'm aware of have no RGBW support... Maybe you can point
me in a good direction :-)
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-322424831,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdskprffscqVVcYU47iMerU462BeL0ks5sYWccgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

I've cam accross this hack for the FastLED library: http://www.partsnotincluded.com/neopixels/fastled-rgbw-neopixels-sk6812/

Ouch, lots of work for them!
I think this would be much easier in Python (or for that matter in modern
C++, which if I were FastLED I'd be strongly thinking of porting to).
I'm fairly certain that we'll be doing this in the medium-term.  We need to
fiddle with the structure of a pixel anyway to do alpha channels, and once
we're there, we might as well go all the way.  :-D
On Tue, Aug 15, 2017 at 12:19 PM, Patrik Mayer<EMAIL_ADDRESS>wrote:

I've cam accross this hack for the FastLED library:
http://www.partsnotincluded.com/neopixels/fastled-rgbw-neopixels-sk6812/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-322432218,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdslb7R86tti-GzzyFW9tePy4QOws8ks5sYXDIgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

There are a few libraries that support it... mainly one-offs from Adafruit
to support the stuff they sell.
Funny you should mention single color displays though... I just hacked that
together last night:
https://twitter.com/ManiacalLabs/status/897274713642221568
That's the first display I ever built... it has no brightness control at
all. Originally it just showed text.
But getting it working was a complete hack on my part... I copied the BP
Serial driver and hacked at it so that I threw out everything but red. Then
I too that and turned it into single bytes for each column of 8 pixels
where each bit is a single pixel. Anything in the red channel > 0 meant the
pixel is on.
Code here:
https://github.com/ManiacalLabs/HT1632_Serial/tree/master/HTSerial
One of the reasons it's done that way is that the Arduino controlling it
doesn't have the RAM to take more data than one byte per column :P I'm in
no way saying this is how we should handle anything in the future... But it
was at least nice that I could very easily just remove the driver, hack at
it a bit and then call out to it from a project file.
On Tue, Aug 15, 2017 at 6:53 AM, Tom Swirly<EMAIL_ADDRESS>wrote:

Ouch, lots of work for them!
I think this would be much easier in Python (or for that matter in modern
C++, which if I were FastLED I'd be strongly thinking of porting to).
I'm fairly certain that we'll be doing this in the medium-term. We need to
fiddle with the structure of a pixel anyway to do alpha channels, and once
we're there, we might as well go all the way. :-D
On Tue, Aug 15, 2017 at 12:19 PM, Patrik Mayer<EMAIL_ADDRESS>wrote:

I've cam accross this hack for the FastLED library:
http://www.partsnotincluded.com/neopixels/fastled-rgbw-neopixels-sk6812/
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
<https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-
322432218>,
or mute the thread
<https://github.com/notifications/unsubscribe-auth/AAPdslb7R86tti-
GzzyFW9tePy4QOws8ks5sYXDIgaJpZM4HOuUs>
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com
—
You are receiving this because you modified the open/close state.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-322437602,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6a6vLiw21DVrxWKEYDdB6MSFx3qE9mks5sYXjCgaJpZM4HOuUs
.

@adammhaile: Wow, nice one.
But you've understood me wrong. We are displaying solid colors (no animations) in one single stripe. But we have the same problem regarding the memory of the microcontroller... 1605 Pixel x 4 Byte = ~6,5kb RAM for "one" state of the complete stripe. Not to speak from animations where you need at least the last state of the pixel to calculated the next state ;-)
The goal ist to show animations on that long stripe. Segmentation is possible but I don't know how to acomplish this. I thought about an ESP8266 oder a PiZero for that... but every usable software I find has no support for the RGBW stripes.
So, I'm running circles :-P

oh... gotcha.
What's your reason for needing the W channel? Just curious... just what you
had on hand?
On Tue, Aug 15, 2017 at 11:06 AM, Patrik Mayer<EMAIL_ADDRESS>wrote:

@adammhaile https://github.com/adammhaile: Wow, nice one.
But you've understood me wrong. We are displaying solid colors (no
animations) in one single stripe. But we have the same problem regarding
the memory of the microcontroller... 1605 Pixel x 4 Byte = ~6,5kb RAM for
"one" state of the complete stripe. Not to speak from animations where you
need at least the last state of the pixel to calculated the next state ;-)
The goal ist to show animations on that long stripe. Segmentation is
possible but I don't know how to acomplish this. I thought about an ESP8266
oder a PiZero for that... but every usable software I find has no support
for the RGBW stripes.
So, I'm running circles :-P
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-322495635,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AA6a6kFOfmXxbuuuR8cEBAPzBk_9B047ks5sYbQRgaJpZM4HOuUs
.

We've thought it was a good idea to have a "normal" warmwhite ambiet light, as the pixels are installed in a lounge area of a bar. And as I wrote, I was a bit naive for driving the pixels... thought I had only change the Library Init to acomplish RGBW.

The very reasonable desire to get a "normal" warm light is the driving
force behind W, Amber and Lime channels, from what I have heard!
On Tue, Aug 15, 2017 at 5:40 PM, Patrik Mayer<EMAIL_ADDRESS>wrote:

We've thought it was a good idea to have a "normal" warmwhite ambiet
light, as the pixels are installed in a lounge area of a bar. And as I
wrote, I was a bit naive for driving the pixels... thought I had only
change the Library Init to acomplish RGBW.
—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-322504652,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdshXtH_zqYjW9tQe84XDvtSO1nZI1ks5sYbvygaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

I've done this with https://github.com/jgarff/rpi_ws281x and converting rgb to rgbw.
for (var i = 0; i < NUM_LEDS; i++) {
   pixelData[i] = rgb2Int(amp*w,amp*r,amp*g,amp*b);
}

ws281x.render(pixelData);

function rgb2Rgbw(r,g,b){
	var m = r < g ? (r < b ? r : b) : (g < b ? g : b);
	return {
		r:r-m,
		g:g-m,
		b:b-m,
		w:m
	};
}

Worked out pretty great, although it assumes you'll never need all channels on since you're basically replacing the overlap between the channels with a dedicated white channel.

Wow... I have to try this after my vacation. Much appreciated ^^

@rec @drewcovi
I was able to get BiblioPixel to work with RGBW now that rpi_ws281x supports it. The change was simply to the final hardcoded strip_type parameter in the PiWS281X driver to the 0x18100800 hex code for SK6812.
That hardcoded hex for ws2811 is out of date as of https://github.com/jgarff/rpi_ws281x/commit/1ee85ad1c334102d71b0794f993591695273af9c#diff-a75872860cfdbe91d6131e7de40b1c88

Thanks for this!
Based on that, I just sent out a pull request to allow you to set this from
the BP driver: https://github.com/ManiacalLabs/BiblioPixel/pull/423.
On Sat, Sep 9, 2017 at 8:25 AM, Reid Harrison<EMAIL_ADDRESS>wrote:

@rec https://github.com/rec @drewcovi https://github.com/drewcovi
I was able to get BiblioPixel to work with RGBW now that rpi_ws281x
supports it
https://github.com/jgarff/rpi_ws281x/blob/bd32bedf446a609528da9a797a1a7e4852decd5a/ws2811.h#L45.
The change was simply to the final hardcoded strip_type parameter in the PiWS281X
driver
https://github.com/ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/bibliopixel/drivers/PiWS281X.py#L60
to the 0x18100800 hex code for SK6812.
That hardcoded hex for ws2811 is out of date as of jgarff/rpi_ws281x@
1ee85ad#diff-a75872860cfdbe91d6131e7de40b1c88
https://github.com/jgarff/rpi_ws281x/commit/1ee85ad1c334102d71b0794f993591695273af9c#diff-a75872860cfdbe91d6131e7de40b1c88
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328258045,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsjHqhKkrn9rSgysR6iRbnwAoX9EDks5sgi9KgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

@rec
Awesome!! My only other thought is to add a white byte to the packet data. The rpi_ws281x Color now supports it as a parameter but the driver only expects RGB. I'm pretty new to this still and don't quite understand how the packet data is getting set or else I'd contribute the change myself.
https://github.com/jgarff/rpi_ws281x/blob/bd32bedf446a609528da9a797a1a7e4852decd5a/python/neopixel.py#L8
https://github.com/rec/BiblioPixel/blob/4ab981b3800ab9ae7318fb4022c12754ebf2dbe1/bibliopixel/drivers/PiWS281X.py#L83

Interesting.
That's doable.  Having separate white components for each pixel we can't do
just yet...
On Sat, Sep 9, 2017 at 7:56 PM, Reid Harrison<EMAIL_ADDRESS>wrote:

@rec https://github.com/rec
Awesome!! My only other thought is to add a white byte to the packet data.
The rpi_ws281x Color now supports it as a parameter but the driver only
expects RGB. I'm pretty new to this still and don't quite understand how
the packet data is getting set or else I'd contribute the change myself.
https://github.com/jgarff/rpi_ws281x/blob/bd32bedf446a609528da9a797a1a7e
4852decd5a/python/neopixel.py#L8
https://github.com/rec/BiblioPixel/blob/4ab981b3800ab9ae7318fb4022c127
54ebf2dbe1/bibliopixel/drivers/PiWS281X.py#L83
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328293558,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsjNG7iu6NeQI1QnSqNnjJnbAoM7Xks5sgtFMgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

Is this logic preventing the change? https://github.com/ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/bibliopixel/drivers/driver_base.py#L117
Could the "magic number" of 3 be variablized? Here too
Otherwise, it looks to me like the PiWS281X driver logic needs slight modification to how it breaks up the buffer data: https://github.com/rec/BiblioPixel/blob/4ab981b3800ab9ae7318fb4022c12754ebf2dbe1/bibliopixel/drivers/PiWS281X.py#L83
Please bear with me if I'm missing something here!

Everything you write is correct.
Unfortunately, there are several other spots in the code that assume the
number 3, some of them not so obvious.
And worse, we have issues with ideas like "colors" - which are now all
3-tuples.  We'd need to have more bullet-proof code to handle setting a
pixel in 4-component color layout with a 3-component color.
This is on the cards to do but probably won't get done for several weeks if
not more.
On Sat, Sep 9, 2017 at 9:22 PM, Reid Harrison<EMAIL_ADDRESS>wrote:

Is this logic preventing the change? https://github.com/
ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f8
5dad3346dc/bibliopixel/drivers/driver_base.py#L117
Could the "magic number" of 3 be variablized? Here too
https://github.com/ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/bibliopixel/drivers/driver_base.py#L59
Otherwise, it looks to me like the PiWS281X driver logic needs slight
modification to how it breaks up the buffer data: https://github.com/rec/
BiblioPixel/blob/4ab981b3800ab9ae7318fb4022c12754ebf2dbe1/bibliopixel/
drivers/PiWS281X.py#L83
Please bear with me if I'm missing something here!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328298288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsv_CnVKDPZZTmNlrAf3iJMowzN6Zks5sguWLgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

However, I can start to make these changes in the locations you
identified and a few more, just to get the ball rolling.  It won't affect
correctness anywhere else - of course, it won't be any good till its
everywhere, but it's a start.
On Sat, Sep 9, 2017 at 9:34 PM, Tom Swirly<EMAIL_ADDRESS>wrote:

Everything you write is correct.
Unfortunately, there are several other spots in the code that assume the
number 3, some of them not so obvious.
And worse, we have issues with ideas like "colors" - which are now all
3-tuples.  We'd need to have more bullet-proof code to handle setting a
pixel in 4-component color layout with a 3-component color.
This is on the cards to do but probably won't get done for several weeks
if not more.
On Sat, Sep 9, 2017 at 9:22 PM, Reid Harrison<EMAIL_ADDRESS>wrote:

Is this logic preventing the change? https://github.com/ManiacalLab
s/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/
bibliopixel/drivers/driver_base.py#L117
Could the "magic number" of 3 be variablized? Here too
https://github.com/ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/bibliopixel/drivers/driver_base.py#L59
Otherwise, it looks to me like the PiWS281X driver logic needs slight
modification to how it breaks up the buffer data:
https://github.com/rec/BiblioPixel/blob/4ab981b3800ab9ae7318
fb4022c12754ebf2dbe1/bibliopixel/drivers/PiWS281X.py#L83
Please bear with me if I'm missing something here!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328298288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsv_CnVKDPZZTmNlrAf3iJMowzN6Zks5sguWLgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

Sent a pull request out:
https://github.com/ManiacalLabs/BiblioPixel/pull/424
This is just a start - I only picked up the low-hanging fruit and even then
I didn't see a good immediate solution in every case.  But it puts it in
the radar.
On Sun, Sep 10, 2017 at 10:39 AM, Tom Swirly<EMAIL_ADDRESS>wrote:

However, I can start to make these changes in the locations you
identified and a few more, just to get the ball rolling.  It won't affect
correctness anywhere else - of course, it won't be any good till its
everywhere, but it's a start.
On Sat, Sep 9, 2017 at 9:34 PM, Tom Swirly<EMAIL_ADDRESS>wrote:

Everything you write is correct.
Unfortunately, there are several other spots in the code that assume the
number 3, some of them not so obvious.
And worse, we have issues with ideas like "colors" - which are now all
3-tuples.  We'd need to have more bullet-proof code to handle setting a
pixel in 4-component color layout with a 3-component color.
This is on the cards to do but probably won't get done for several weeks
if not more.
On Sat, Sep 9, 2017 at 9:22 PM, Reid Harrison<EMAIL_ADDRESS>wrote:

Is this logic preventing the change? https://github.com/ManiacalLab
s/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/
bibliopixel/drivers/driver_base.py#L117
Could the "magic number" of 3 be variablized? Here too
https://github.com/ManiacalLabs/BiblioPixel/blob/ea3455b58ab741ac1d5319373a87f85dad3346dc/bibliopixel/drivers/driver_base.py#L59
Otherwise, it looks to me like the PiWS281X driver logic needs slight
modification to how it breaks up the buffer data:
https://github.com/rec/BiblioPixel/blob/4ab981b3800ab9ae7318
fb4022c12754ebf2dbe1/bibliopixel/drivers/PiWS281X.py#L83
Please bear with me if I'm missing something here!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328298288,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsv_CnVKDPZZTmNlrAf3iJMowzN6Zks5sguWLgaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com

@reid-harrison That's really weird that it needs the SK hex code. Thanks for finding that.
And yeah, we really need a way to allow arbitrary byte counts for all this. Honestly, BP shouldn't care at all. I think that's doable. I build a display the other week that is 1-bit per pixel even. But what I did was write a totally custom driver that throws away all but red and then bitshifts everything down to 8 pixels per bye. But that's an extreme example.

One byte per pixel, I think.
On Tue, Sep 12, 2017 at 3:01 AM, Adam Haile<EMAIL_ADDRESS>wrote:

@reid-harrison https://github.com/reid-harrison That's really weird
that it needs the SK hex code. Thanks for finding that.
And yeah, we really need a way to allow arbitrary byte counts for all
this. Honestly, BP shouldn't care at all. I think that's doable. I build a
display the other week that is 1-bit per pixel even. But what I did was
write a totally custom driver that throws away all but red and then
bitshifts everything down to 8 pixels per bye. But that's an extreme
example.
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ManiacalLabs/BiblioPixel/issues/31#issuecomment-328700524,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAPdsig1JE_hv65v5HqUkzlbeDgk-2x_ks5shdf6gaJpZM4HOuUs
.

--
/t
https://tom.ritchford.com https://tom.ritchford.com
https://tom.swirly.com https://tom.swirly.com


Update example in readme to compile and run with v0.73.0

Update example in readme to compile and run with v0.73.0

 Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.You have signed the CLA already but the status is still pending? Let us recheck it.

@willcain happy to get this merged in if you could update the commit message to be docs: <same as current message> so the linter will pass. Info on commit messages is available here: https://github.com/newrelic/newrelic-client-go#commit-messages

Thanks, updated.


validation checks error - possible improvement

Can you improve the error messages? When a cert does not match the domain queried the word Error is the same color as the domain

Could the full error not be displayed when pressing enter with the list just giving highlights?
When running ssl-checker without any values there is no output or help printed.

Thank you for your feedback @daniejstriata. I believe changed in #8 should address all of your suggestions.


Aarch 64 version

Am examining this on Aarch 64.  Would it be helpful to have a separate branch using sse2neon or have sse2neon supported in the main branch? Codes could also be kept separate, but having them in one place may ease comparison, especially if other vector extensions are added.

I'm so sorry for late reply.
Thank you for your feedback!
We are currently preparing the NEON implementation of AREION based on the following paper.
https://eprint.iacr.org/2023/794
We plan to release the NEON version soon, so please give it a try once it's available!


Tweak OO chapter for the essentials version

Here are the illustrations I used in the morning lecture.
I also removed the two final parts. Let's do this in the advanced course.

Did you kill the CI? I don't see any passed checks.


GraphQLResolverOf not working as expected with code-first

Describe the bug
I expect to be able to use GraphQLResolverOf as described in the docs here: https://hotchocolate.io/docs/resolvers#resolver-types

If you have two attributes GraphQLResolverOf  on a file, it throws here https://github.com/ChilliCream/hotchocolate/blob/d5595996a6baf1dcd948174cacc6d078af48fa60/src/Core/Types/Configuration/TypeInitializer.cs#L288-L289

This snapshot seems wrong as the query clearly doesn't contain the logic from the new resolver: https://github.com/ChilliCream/hotchocolate/blob/master/src/Core/Types.Tests/snapshots/SchemaBuilderTests.AddType_TypeIsResolverTypeByType_QueryContainsBazField.snap

The operation in a simple case (shown below) doesn't work:

public class QueryRoot
{
  // ...
}

[GraphQLResolverOf(typeof(QueryRoot))]
public class QueryRootResolvers
{
  public string GetTestString => "foobar";
}

public class Startup
{
    public void ConfigureServices(IServiceCollection services)
    {
                // ...
		services.AddGraphQL(
                    schemaFactory: sp => SchemaBuilder.New()
			// ...
			.AddQueryType<QueryRoot>()
                        .AddType<QueryRootResolvers>()
               // ...
    }
}

Expected behavior
Expected QueryRoot to have a new field TestString
Desktop (please complete the following information):

OS: Windows 10
Version 10.3.0-preview5

I have checked this one now and it works like the following:
[GraphQLResolverOf(typeof(QueryType))]
public class QueryResolverOnType
{
    public string GetFoo([Parent]object o) => "QueryResolverOnType";
}

[GraphQLResolverOf("Query")]
public class QueryResolverOnName
{
    public string Baz { get; }
}

So, the type has to be a schema type.
The question is if this is still a useful feature? This feature dates back to the pre SchemaBuilder versions of HC.
My gut feeling is that we should deprecate it and move to introduce type extension descriptor attributes.
[ExtendObjectType("Query")]
public class QueryResolverOnType
{
    public string GetFoo([Parent]object o) => "QueryResolverOnType";
}

The attribute would mark this type an extension of a specific type.
what do you guys think? @tunurgitr @PascalSenn

Also just as a side note to this. With version 11 there is already now support for the extensions like in 10 and then there are type interceptors which lets you intercept the type initialization on a broader scale. With that you can do things like ... for all types that have an Id field make the field of type Uuid and things like that.
https://github.com/ChilliCream/hotchocolate/blob/master/src/Core/Types/Configuration/TypeInitializationInterceptor.cs

My vote is for deprecating this in favor of the other ways of extending types...personally I think it's better to be more opinionated here unless there's some compelling reason to stick with this other behavior.

I agree with you. The GraphQLResolverOfAttribute is I think fully replaced by our new ways to extend things.
OK, I will move this issue to 11 and we will deprecate it there.
We can then start removing it with version 12.

So, I have marked this as a bug and will close it and create a follow up issue that will deprecate it.

#1298


Add to metadata a name for a column that contains unique id

Carry in metadata a name of a column that contains unique survey id identifier.
This is because we expect that (https://github.com/astronomy-commons/hipscat/issues/345) our internal name for an object is not necessary unique, and we want to have a unique identifier for various association tables and similar applications.

Possible via kwargs with #358. To be implemented in import with https://github.com/astronomy-commons/hipscat-import/issues/393


licensing related, can we include this with our UWP app which we are selling

We are developing and selling a UWP app for our client as per his requirement. He wants to change the theme easily, so found this theme changing functionality of 'fluent-xaml-theme-editor' interesting.
Can we bundle this along with our uwp app and sell (or) is this any license related issue?
Can anyone give clarity.
@kikisaints

IANAL, but, as per the license, yes.

Yes, technically you can.


deb installer

so this adds an offline deb installer, there is one built from the master branch already, to build it run build_deb.sh in installer/linux/offline
tested on ubuntu, it works, the post install and post remove scripts in it are complete and utter hacks btw, cuz turns out vrpathreg runs a browser
rn it needs openvrlib installed, cuz again vrpathreg doesnt work without it even tho steamvr does
dont merge this just yet

it is surprising how long i didnt get back to my linux
im back now, dep installer coming soon:tm:

one last test and i'll merge this


Updated Mapzen.js docs location to release tarball

@hanbyul-here and I have moved the documentation for Mapzen.js to a tarball generated as part of the release and deploy process.

looks ok in testing


Create a Github page for Picnic Machine Learning

Create a single page website: picnicml.github.io.

I can help you 😉👍

Hey @munirozkul. That's great, thanks! I've created a repo: https://github.com/picnicml/picnicml.github.io. I was thinking of something minimalistic with a title Picnic Machine Learning, an additional label Immutable machine learning in Scala. and a link to the doddle-model repo. What do you think?

Ok, cool;)
Let me fork your picnicml.github.io repo and try something to show you.

That sounds great, thanks :).

Yes, here is an example:
https://munirozkul.github.io/picnicml.github.io/

What do you think about centering everything that's on a white background and making the github icon bigger? Nitpicking, but <title>PicnicML</title> should probably be <title>picnicml</title>.

Of course;) Let me try..

Here it is: https://munirozkul.github.io/picnicml.github.io/

Thanks, @munirozkul. I've discussed this with @novoselrok and we'll take a different approach for GH pages. The main difference is that we'll add documentation automatically generated from scaladoc. Nevertheless, we appreciate your contribution, if you are still willing to help, LMK :D.

No, thanks. Bye


[refactor-prompt] add wallet token history

What current issue(s) does this address, or what feature is it adding?
move #623 forward by adding token history
How did you solve this problem?
port to new command
How did you make sure your solution works?
make test
Are there any special changes in the code that we should be aware of?
Yes. When I added the get_token utility function in neo.Prompt.Utils I was facing circular import errors. To get around them I changed some import statements to not import functions directly, but import the module and call from there.
Please check the following, if applicable:

[X] Did you add any tests?
[X] Did you run make lint?
[X] Did you run make test?
[X] Are you making a PR to a feature branch or development rather than master?
[ ] Did you add an entry to CHANGELOG.rst? (if not, please do)

Coverage increased (+0.004%) to 85.254% when pulling 410ef5e59d76c864e3ddd2f5180dde054d4f5f03 on ixje:token-history into 1ea0b83dc4790445bb52de3e4874a8b96538c913 on CityOfZion:refactor-prompt.


Use goroutines and sync.Map in List API to increase performance

Proposed Changes
Since the volume backups are stored in double hashed folders.
For 1 volume backup, 1 + 1 + 1 = 3 s3 list command queries.
For 32 volume backups, the worst case is 1 + 32 + 32 = 65 s3 list command queries.
In this PR, we propose to use goroutines and sync.Map to increase the performance and make sure the thread-safe by making the queries in parallel.
With 32 volume backups, the command time be:

w/o this PR: 2m53.696s
w/ this PR: 0m10.179s

NOTE:

my cluster has 4 CPUs per host
the latency between LH cluster to local in-cluster minio server is 1 second

Linked Issues
https://github.com/longhorn/longhorn/issues/1955

Good idea parallelizing the deep lookups, but instead of spawning an unbounded amount of go routines.
Consider implementing a fixed worker pool + queue approach.
Then we can pass a function to the workers + path param.
We can then use this for similar deep functions, getBlockNamesForVolume,
The DeleteObjects call can also benefit from parallel deletion.

Good idea parallelizing the deep lookups, but instead of spawning an unbounded amount of go routines.
Consider implementing a fixed worker pool + queue approach.
Then we can pass a function to the workers + path param.
We can then use this for similar deep functions, getBlockNamesForVolume,
The DeleteObjects call can also benefit from parallel deletion.

Sounds good, I'll implement it and test the worker + queue pool performance.

This is ready to review.

@jenting Can you do a another benchmark run to see how the performance is with the fixed worker pool?

@shuo-wu any outstanding feedback? if not feel free to merge.


enh: update native tmq api

Codecov Report

Merging #205 (ed8247a) into 3.0 (949f3fe) will decrease coverage by 0.13%.
The diff coverage is 72.85%.

@@            Coverage Diff             @@
##              3.0     #205      +/-   ##
==========================================
- Coverage   81.54%   81.42%   -0.13%
==========================================
  Files          20       20
  Lines        2585     2649      +64
==========================================
+ Hits         2108     2157      +49
- Misses        477      492      +15

Files Changed
Coverage Δ

taos/tmq.py
87.97% <70.00%> (-3.05%)
:arrow_down:

taos/cinterface.py
81.25% <75.00%> (-0.01%)
:arrow_down:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more


How can I play it in background

Is that possible to play sound in the background, and show the audio in notification bar?

You tried or not?

I tried. yes it runs in the background

How to does it work bro?


[Bug] 输入BUG标题 /mj are not working

Describe the bug
command /mj are not working getting one answer anyway .
Prompt: test
Task ID: undefined
[10/3/2023, 2:03:39 PM] - Task submitted successfully: Please wait a moment
To Reproduce
Steps to reproduce the behavior:

Go to '...'
Click on '....'
Scroll down to '....'
See error

Expected behavior
A clear and concise description of what you expected to happen.
Screenshots
If applicable, add screenshots to help explain your problem.
Deployment

[*] Docker
[ ] Vercel
[ ] Railway
[ ] Server

Desktop (please complete the following information):

OS: [e.g. iOS]
Browser [e.g. chrome, safari]
Version [e.g. 22]

Smartphone (please complete the following information):

Device: [e.g. iPhone6]
OS: [e.g. iOS8.1]
Browser [e.g. stock browser, safari]
Version [e.g. 22]

Additional Logs
Add any logs about the problem here.

you need input : /mj [prompt]

@Licoy You are right. This is a useful information but the author don't mention it anywhere. The doc is so bad almost make me crazy.


Add --predicate cli flag

Fixes #21

Thanks!
Do we plan to expose this in the API too?
I've built a reusable workflow for https://github.com/google/ko and I need exactly this feature. But I think I'd rather use an API

Do you think it's not appropriate to do something like what we've done here and called HostedActionsProvenance and just use the Predicate field?
p, _ := slsa.HostedActionsProvenance(someWorkflowRun)
fmt.Println(p.Predicate)

this seems good enough to me. We can do the same to set the material sections (I was mostly curious if you thought it should be treated differently - I don't see why :-))


Export absolute value function

merging after pass if no comment

There is a built-in function abs, corresponding to the __abs__ special method, so you didn't really need this. No harm, though.


Incompatibility with Zellij

I've encountered a compatibility issue between this plugin and Zellij, the terminal multiplexer. Pressing  doesn't revert to a previous state.
Is there any known workaround?
Thanks!

You could always change the mapping around and see what works through Zellij's escaping. I expect that you'll find another mapping than the default working fine, but I'm afraid there'll be no workaround preserving the default mappings. I'm open to additional default mappings though, let me know if you find a ergonomic/mnemonic solution and I'll include it. Especially <Cr> has caused trouble before...


Missing FAQ for issues

I'm trying to run it with baggage-loader and each time get
Cannot resolve module 'html'

try npm install html-loader --save-dev

@WearyMonkey it helped! thanks a lot!


cmor file not created from high frequency var

Hi all,
please forgive me if this is a trivial issue but I am not able to see the solution by myself: I am trying to cmorize some PRIMAVERA data with the PRIMAVERA tables: first question, is anyone still using this? Should I use the PRIMAVERA branch (where there are no PRIMAVERA tables)?
Back to my problem: I have noticed that with PRIMAVERA tables I only obtain daily and monthly cmor data. All the high frequency postprocessed data (3hr, 3hrPt, 6hr, 6hrPt) does not show up.
For example, if I have a very simple varlist.json test file, i.e.
{     "Prim3hr": [         "sfcWind",         "evspsbl",         "sfcWindmax"     ] }
when I run the cmorization, Prim3hr data are not created and this is the log I get: apparently there is an error but it not clear where this is coming from.
INFO:taskloader:Found 3 cmor target variables in input variable list.
INFO:taskloader:Created 3 ece2cmor tasks from input variable list.
INFO:ece2cmorlib:Selected 3 IFS tasks from 3 input tasks
INFO:ifs2cmor:Inspecting 3 tasks.
INFO:ifs2cmor:Validated 3 tasks for processing.
INFO:ifs2cmor:Executing 3 IFS tasks...
INFO:ifs2cmor:Post-processing mask variables...
INFO:ifs2cmor:Post-processing 0 IFS tasks...
INFO:ifs2cmor:Post-processed batch of 0 tasks.
INFO:ifs2cmor:Post-processing surface pressures...
INFO:ifs2cmor:Post-processing 0 IFS tasks...
INFO:ifs2cmor:Post-processed batch of 0 tasks.
INFO:ifs2cmor:Cmorizing surface pressures...
INFO:ifs2cmor:Cmorizing 0 IFS tasks...
INFO:ifs2cmor:Post-processing regular variables...
INFO:ifs2cmor:Post-processing 3 IFS tasks...
ERROR:postproc:Unsupported combination of frequency 3hr with time operators [u'mean'] encountered
ERROR:postproc:Unsupported combination of frequency 3hr with time operators [u'mean'] encountered
ERROR:postproc:Unsupported combination of frequency 3hr with time operators [u'maximum'] encountered
INFO:postproc:Post-processing target sfcWind in table Prim3hr from file /marconi_scratch/userexternal/pdavini0/tmp_cmor/sta1_30928/tmp_1951_1/filtered/3hr/ICMGGsta1+195101 with cdo command -expr,'var214=sqrt(sqr(var165)+sqr(var166))' -setgridtype,regular -selcode,165,166
ERROR:cmor_task:Attempt to increase failed status for task ignored
INFO:postproc:Post-processing target sfcWindmax in table Prim3hr from file /marconi_scratch/userexternal/pdavini0/tmp_cmor/sta1_30928/tmp_1951_1/filtered/3hr/ICMGGsta1+195101 with cdo command -expr,'var214=sqrt(sqr(var165)+sqr(var166))' -setgridtype,regular -selcode,165,166
ERROR:cmor_task:Attempt to increase failed status for task ignored
INFO:postproc:Post-processing target evspsbl in table Prim3hr from file /marconi_scratch/userexternal/pdavini0/tmp_cmor/sta1_30928/tmp_1951_1/filtered/3hr/ICMGGsta1+195101 with cdo command -setgridtype,regular -selcode,182
ERROR:cmor_task:Attempt to increase failed status for task ignored
INFO:ifs2cmor:Post-processed batch of 0 tasks.
INFO:ifs2cmor:Cmorizing regular variables...
INFO:ifs2cmor:Cmorizing 0 IFS tasks...

! ------
! All files were closed successfully.
! ------

Is there any way to increase the verbosity? I am running with python 2.7.12 and cdo 1.8.2, which is a pretty standard configuration I think. The weirdest thing is that if I run for example
cdo expr,'var214=sqrt(sqr(var165)+sqr(var166))' -setgridtype,regular -selcode,165,166
it runs smooth and a grib file is created. What I am doing wrong?
Any suggestions welcome!
Cheers,
Paolo

The problem is that in your version of the tables the high frequency variables are denoted with the mean operators which ece2cmor3 does not accept. I have fixed this in the nocdo branch but it is not yet merged. One way would be to edit the tables and replace 'mean' with 'point' for these. I am currently not able to fix it in the code because I am on vacation.

Thanks for the prompt reply (especially since you are on vacation). This is a very good hint!
I modified this for the two mean variables (I let the maximum one as it is for the moment) but it crashes again blaming time bounds. Is there any official repository for the Primavera tables repository? Perhaps I am using a wrong version.
 Warning: Table Prim3hr, unknown variable attribute: >>>>primavera_priority<<<< value: (1)
!
!!!!!!!!!!!!!!!!!!!!!!!!!

 Warning: Table Prim3hr, unknown variable attribute: >>>>primavera_priority<<<< value: (1)
!
!!!!!!!!!!!!!!!!!!!!!!!!!

 Warning: Table Prim3hr, unknown variable attribute: >>>>primavera_priority<<<< value: (1)
!
!!!!!!!!!!!!!!!!!!!!!!!!!

INFO:ifs2cmor:Creating time axes for table Prim3hr...
INFO:ifs2cmor:Creating time axis using variable sfcWind...

C Traceback:
! In function: cmor_axis
!

!!!!!!!!!!!!!!!!!!!!!!!!!
!
! Error: axis: time (table: Prim3hr) must have bounds, you did not pass any when creating it via cmor_axis
!
!!!!!!!!!!!!!!!!!!!!!!!!!

Traceback (most recent call last):
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ece2cmor.py", line 75, in <module>
    main()
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ece2cmor.py", line 68, in main
    maxsizegb = args.tmpsize)
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ece2cmorlib.py", line 139, in perform_ifs_tasks
    ifs2cmor.execute(ifs_tasks,cleanup = cleanup)
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ifs2cmor.py", line 132, in execute
    cmorize(processedtasks)
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ifs2cmor.py", line 344, in cmorize
    create_time_axes(tskgroup)
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ifs2cmor.py", line 458, in create_time_axes
    tid = create_time_axis(freq = task.target.frequency,path = getattr(task,"path"),name = tdim,hasbnds = (timop != ["point"]))
  File "/marconi_work/Pra13_3311/ecearth3/post/ece2cmor3/ece2cmor3/ece2cmor3/ifs2cmor.py", line 624, in create_time_axis
    return cmor.axis(table_entry = str(name),units = "hours since " + str(ref_date_),coord_vals = times)
  File "/marconi_work/Pra13_3311/opt/anaconda/envs/ece2cmor3/lib/python2.7/site-packages/cmor/pywrapper.py", line 444, in axis
    type, cell_bounds, cbnds, interval)
_cmor.CMORError: Problem with 'cmor.axis'. Please check the logfile (if defined).

Ok that's an issue from our side. Can you try the primavera branch of ece2cmor3? After switching remember to type git submodule update --init --recursive to checkout the primavera tables of John seddon

Thanks. I tried this way right now, but apparently there is no PRIMAVERA_grids.json in the Jon Seddon tables, which leads to a crash of the code.

Hi sorry for my late answer. I have had this issue too and solved it my creating symlink to CMIP6_grids.json named PRIMAVERA_grids.json or a copy will do as well.

Following the discussion at #146, I think this is deprecated.


file: add patch for CVE-2019-18218

Motivation for this change
https://nvd.nist.gov/vuln/detail/CVE-2019-18218
Upstream patch https://github.com/file/file/commit/46a8443f76cec4b41ec736eca396984c74664f84.patch doesn't apply directly, debian have a version which has been adapted for 5.37.
Massive list of reverse dependencies - I don't have the capacity to do the full build.
Things done

[x] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS)
Built on platform(s)

[ ] NixOS
[x] macOS
[x] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[ ] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[ ] Fits CONTRIBUTING.md.

Notify maintainers
cc @

This probably should be submitted to staging branch.

Good call. Rebased.

@GrahamcOfBorg build file

The patch looks reasonable. I think we should go for Hydra to build all reverse deps -- it's unlikely that any developer will do that on their own machine.


CodeHilite - custom pygments lexer

Is it possible to load a custom pygment lexer for the CodeHilite extension ?

Ok, i will have to go the "long" way then, i will try :)
Thank you for your support


[xerces-c] produces strange files in installed/x64-linux/lib

Host Environment

OS: CentOS 7
Compiler: g++ (GCC) 8.2.1 20180905 (Red Hat 8.2.1-3)

To Reproduce
./vcpkg install xerces-c
Failure logs
$ ls -l installed/x64-linux/lib/
-rw-r--r--. 1 user user 12969802 Jul 31 15:16 libxerces-c-3.2.a
lrwxrwxrwx. 1 user user       17 Jul 31 15:16 libxerces-c.so -> libxerces-c-3.2.a
drwxrwxr-x. 2 user user       24 Jul 31 15:16 pkgconfig

$ ls -l installed/x64-linux/lib/pkgconfig/
-rw-r--r--. 1 user user 483 Jul 31 15:16 xerces-c.pc

Also, config-*-err.log files contain strange entry:
CMake Warning at /home/user/vcpkg/scripts/buildsystems/vcpkg.cmake:107 (message):
  There are no libraries installed for the Vcpkg triplet x64-linux.

In source\cmake\XercesDll.cmake:23 :
set(BUILD_SHARED_LIBS ON CACHE BOOL "Build shared libraries")

Forced to set a dynamic library caused this issue.

What is the proper way to fix it -- should I patch it so it sets this variable only if VCPKG_LIBRARY_LINKAGE is static? (assuming this variable is visible at that point)
Trying to avoid breaking other triplets...

Just delete this line should fix this issue.

Hmm... nope -- it doesn't help. Trying to figure out who creates these symlinks...
On a side note: is it ok that some packages (zstd/xerces-c/etc) create .pc files in installed/x64-linux/lib/?

Found it in src/CMakeLists.txt:
elseif(UNIX)
  # For strict libtool compatibility on Unix.  It's a horrible hack to
  # set the version in the filename, and create the symlink at install
  # time.  Note: could be dropped when the SONAME is updated and
  # libtool compatibility is no longer required.
  set_target_properties(xerces-c PROPERTIES OUTPUT_NAME "xerces-c-${INTERFACE_VERSION_D}")
  file(GENERATE
    OUTPUT "${CMAKE_CURRENT_BINARY_DIR}/InstallLibrarySymlink.cmake"
    CONTENT "execute_process(COMMAND ln -sf \"$<TARGET_FILE_NAME:xerces-c>\" \"\$ENV{DESTDIR}${CMAKE_INSTALL_FULL_LIBDIR}/libxerces-c.so\")")
  install(SCRIPT "${CMAKE_CURRENT_BINARY_DIR}/InstallLibrarySymlink.cmake")

somebody was very naughty...

Is there any way to link agains a dynamic [xercec-c] library on Linux? Now the one installed by vcpkg is a static one on linux by default

@hailIulius here is one way to do it: https://github.com/microsoft/vcpkg/blob/master/docs/examples/overlay-triplets-linux-dynamic.md


Not working with some sites

Thanks a lot for developing and sharing this solution, it works nicely for a lot of sites.
However there are some sites where it doesn't solve the challenge and gives up with {"code":500,"message":"Request Timeout"}
This is one of the sites, can you please check and advise
https://apnetv.to/Hindi-Serials

It seems to be caused by devtools detector. Normally the browser has been tested on devtools detector and is not caught, but additional precautions must have been taken. I will investigate this issue in more detail, but you can scrape it without any problem with the following method.
Step 1:
https://github.com/zfcsoftware/cf-clearance-scraper/blob/4a3bb4e86d084bf4a426eced5645d5d46cec6eeb/module/scrape.js#L93
Replace this line with the following.
 waitUntil: 'domcontentloaded'
Step 2:
Interfere with the request. Add the following code in the field below.
https://github.com/zfcsoftware/cf-clearance-scraper/blob/4a3bb4e86d084bf4a426eced5645d5d46cec6eeb/module/browser.js#L113
   const { RequestInterceptionManager } = await import('puppeteer-intercept-and-modify-requests')
       const client = await page.target().createCDPSession()
       const interceptManager = new RequestInterceptionManager(client)
       await interceptManager.intercept({
           urlPattern: `https://apnetv.to/Hindi-Serials`,
           resourceType: "Document",
           modifyResponse({ body }) {
             return {
                      body: body.replace(`window.location.href = 'https://apnetv.to/indexnow.html';`,''),
             };
           },
         });

Thanks a lot for taking the time to look at this issue.
I made the changes and ran on my ubuntu server vm. Looks like even though i did npm install it is missing some dependencies
gujal@tux:~/cf-clearance-scraper$ npm run start

><EMAIL_ADDRESS>start
> node index.js

Server running on port 3000
Failed to launch the browser process! undefined
[1590:1590:0609/185637.942529:ERROR:ozone_platform_x11.cc(243)] Missing X server or $DISPLAY
[1590:1590:0609/185637.942605:ERROR:env.cc(258)] The platform failed to initialize.  Exiting.

TROUBLESHOOTING: https://pptr.dev/troubleshooting

Anyway i will wait for you to check and release the docker image, then will check with the image as it will fully self contained. Thanks

puppeteer-intercept-and-modify-requests

npm i puppeteer-intercept-and-modify-requests
Just run it. After making the changes and running it, scraping will be available on the relevant site.
Thank you for your feedback.

I added your code and tried it and it get's cf_clearance for that site. You think there is a way to intercept dev tool detector for any site, rather than hard-coding the intercept and modify for specific sites?

Can you try again with the latest version?

Can you try again with the latest version?

Yes, tried with the latest version, with the following python code.
Even though cf-clearance-scraper get the cookie, subsequent call with the cookie results in 403
import requests

url = 'https://apnetv.to/Hindi-Serials'
cs_url = 'http://localhost:3000/cf-clearance-scraper'
data = {'url': url}
res = requests.post(cs_url, json=data)
result = res.json()
if res.status_code == 200:
    h = result.get('headers')
    headers = {
        'User-Agent': h.get('user-agent'),
        'Referer': url,
        'Cookie': h.get('cookie')
    }
    res2 = requests.get(url, headers=headers)

Can you try again with the latest version?

Yes, tried with the latest version, with the following python code. Even though cf-clearance-scraper get the cookie, subsequent call with the cookie results in 403
import requests

url = 'https://apnetv.to/Hindi-Serials'
cs_url = 'http://localhost:3000/cf-clearance-scraper'
data = {'url': url}
res = requests.post(cs_url, json=data)
result = res.json()
if res.status_code == 200:
    h = result.get('headers')
    headers = {
        'User-Agent': h.get('user-agent'),
        'Referer': url,
        'Cookie': h.get('cookie')
    }
    res2 = requests.get(url, headers=headers)

Cloudflare does not only check user-agent. It also checks values such as accept-language. If you use the whole header, the problem will be solved.

Yes I tried with the compete headers from cf-clearnce scraper like this
res2 = requests.get(url, headers=h)

It is still getting 403.
That site seems to be doing something more.
If i open the page in browser and then take just the cookie and ua and make a python call it works, so something additional happening on this site. Thanks for looking.

Yes I tried with the compete headers from cf-clearnce scraper like this
res2 = requests.get(url, headers=h)

It is still getting 403. That site seems to be doing something more. May not be in the scope of your project Thanks for looking.
EDIT: Tried with Flaresolverr, even there using headers from the response for subsequent calls doesnt work. so this is some other protection on site probably. However Flaresolverr returns the retrieved page html code in the response so could use that, but not ideal putting every request through Flaresolverr

The library had a bug in the last update and didn't send the header. I don't know how he did it, but that site is very good at bot detection. When I wanted to analyse the request with burp, it blocked it instantly. Cloudflare settings are very good. When sending a request, it somehow catches it, probably doing tls fingerprint check. I am closing this because the problem with the browser and library has been solved. Good luck scraping this site.

Yes I tried with the compete headers from cf-clearnce scraper like this
res2 = requests.get(url, headers=h)

It is still getting 403. That site seems to be doing something more. May not be in the scope of your project Thanks for looking.
EDIT: Tried with Flaresolverr, even there using headers from the response for subsequent calls doesnt work. so this is some other protection on site probably. However Flaresolverr returns the retrieved page html code in the response so could use that, but not ideal putting every request through Flaresolverr

Hello, this problem is related to tls fingerprint. Since I just met tls fingerprint, I could not offer a solution. I searched for it because some of my projects had problems. You can engrave with the code below without any problem.
const initCycleTLS = require('cycletls');

(async () => {
    const cycleTLS = await initCycleTLS();

    const response = await cycleTLS('https://apnetv.to/Hindi-Serials', {
        ja3: '772,4865-4866-4867-49195-49199-49196-49200-52393-52392-49171-49172-156-157-47-53,13-0-18-16-65281-45-43-51-5-27-23-17513-65037-35-11-10,25497-29-23-24,0',
        userAgent: 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36',
        method: "get",
        headers: {
            "authority": "apnetv.to",
            "host": "apnetv.to",
            "origin": "https://apnetv.to",
            "referer": "https://apnetv.to/Hindi-Serials",
            "accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
            "accept-language": "tr-TR,tr;q=0.9,en-US;q=0.8,en;q=0.7",
            "cache-control": "no-cache",
            "pragma": "no-cache",
            "priority": "u=0, i",
            "sec-ch-ua": "\"Chromium\";v=\"127\", \"Not)A;Brand\";v=\"99\"",
            "sec-ch-ua-mobile": "?0",
            "sec-ch-ua-platform": "\"Linux\"",
            "sec-fetch-dest": "document",
            "sec-fetch-mode": "navigate",
            "sec-fetch-site": "none",
            "sec-fetch-user": "?1",
            "upgrade-insecure-requests": "1",
            "cookie": "",
            "user-agent": "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/<IP_ADDRESS> Safari/537.36"
        }
    }, 'get');

    console.log(response.status);
    cycleTLS.exit();
})();


WIP: Reduce load on the Kubernetes API server by enabling the WatchList (Streaming Lists) feature

The WatchList feature of the kube-apiserver is available since v1.27.
The ClientWatchList feature of client-go is available since v0.28.0 enabled with a non-standard ENABLE_CLIENT_GO_WATCH_LIST_ALPHA environment variable.
In client-go 0.31, the client watch list feature of the reflector will be enabled with a new ClientWatchList feature flag instead of the ENABLE_CLIENT_GO_WATCH_LIST_ALPHA variable.
I've measured a significant memory reduction in both the controller and in the kube-apiserver when the WatchList feature is enabled.

Fixes: #3748
kubectl logs -n cert-manager deploy/cert-manager | fgrep -v -e "starting worker" -e "leaderelection"

I0712 16:40:35.025446       1 round_trippers.go:553] GET https://<IP_ADDRESS>:443/api/v1/secrets?allowWatchBookmarks=true&labelSelector=controller.cert-manager.io%2Ffao%3Dtrue&resourceVersionMatch=NotOlderThan&sendInitialEvents=true&timeout=7m37s&timeoutSeconds=457&watch=true 200 OK in 2502 milliseconds
I0712 16:40:35.025528       1 round_trippers.go:553] GET https://<IP_ADDRESS>:443/api/v1/secrets?allowWatchBookmarks=true&labelSelector=!controller.cert-manager.io%2Ffao&resourceVersionMatch=NotOlderThan&sendInitialEvents=true&timeout=8m7s&timeoutSeconds=487&watch=true 200 OK in 2501 milliseconds
I0712 16:40:35.655423       1 reflector.go:798] exiting k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232 Watch because received the bookmark that marks the end of initial events stream, total 6 items received in 3.13200872s
I0712 16:40:35.655514       1 reflector.go:359] Caches populated for *v1.PartialObjectMetadata from k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232
I0712 16:40:35.655697       1 reflector.go:798] exiting k8s.io/client-go@v0.30.2/tools/cache/reflector.go:232 Watch because received

Background

https://github.com/kubernetes/enhancements/blob/master/keps/sig-api-machinery/3157-watch-list/README.md

Enable the WatchList / Streaming list feature in the cert-manager controller to avoid overloading the Kubernetes API server at startup.

@wallrj we are now using the v0.31.0 client-go libraries.

/kind bug

/kind feature


Make util functions to replace tokenSymbol ?? "token" all over the codebase

What does this PR do and why?
The tokenSymbol ?? "token" and its 3 other counterparts weren't being translated properly. I made functions for the 4 different variations in /utils and implemented them throughout the codebase. This shouldn't affect anything in the UI, except that a bunch of 'token' and 'tokens' text will now translate properly.
Apologies for the branch name, initially planned to do the redeem rate slider, then to do these changes and the slider together, but then thought it'd be best to get these changes out in a single PR.
Acceptance checklist

[x] I have evaluated the Approval Guidelines for this PR.
[x] I have tested this PR in all supported browsers.

Beautiful! I hadn't done this before only because it seemed complicated to handle the plural/capital/long/short cases, but I think these util functions work nicely 👏

Looks like there was some conflicts with translations. Just pushed a rebase with main 👍


terraform-providers: bump versions

Motivation for this change
Upgrade TF providers, in particular AWS one
Things done

[ ] Tested using sandboxing (nix.useSandbox on NixOS, or option sandbox in nix.conf on non-NixOS linux)
Built on platform(s)

[x] NixOS
[x] macOS
[ ] other Linux distributions

[ ] Tested via one or more NixOS test(s) if existing and applicable for the change (look inside nixos/tests)
[x] Tested compilation of all pkgs that depend on this change using nix-shell -p nix-review --run "nix-review wip"
[ ] Tested execution of all binary files (usually in ./result/bin/)
[ ] Determined the impact on package closure size (by running nix path-info -S before and after)
[ ] Ensured that relevant documentation is up to date
[x] Fits CONTRIBUTING.md.

Notify maintainers
cc @

Something seems to be wrong with terraform-providers.scaleway.
nix-review log:
[104 built (1 failed), 78 copied (2045.6 MiB), 458.3 MiB DL]
error: build of '/nix/store/c5p4pfx24a5s03v8wix9ivwm6fj5hmaa-env.drv' failed
https://github.com/NixOS/nixpkgs/pull/72782
4 package failed to build:
terraform-full terraform-providers.scaleway terraform_0_11-full terragrunt

50 package were build:
terraform-providers.acme terraform-providers.alicloud terraform-providers.archive terraform-providers.aws terraform-providers.azuread terraform-providers.azurerm terraform-providers.azurestack terraform-providers.bigip terraform-providers.cloudflare terraform-providers.datadog terraform-providers.digitalocean terraform-providers.docker terraform-providers.fastly terraform-providers.flexibleengine terraform-providers.github terraform-providers.google terraform-providers.google-beta terraform-providers.hcloud terraform-providers.heroku terraform-providers.huaweicloud terraform-providers.kubernetes terraform-providers.local terraform-providers.mailgun terraform-providers.mysql terraform-providers.nomad terraform-providers.ns1 terraform-providers.nutanix terraform-providers.oci terraform-providers.openstack terraform-providers.opentelekomcloud terraform-providers.opsgenie terraform-providers.oraclepaas terraform-providers.packet terraform-providers.pagerduty terraform-providers.panos terraform-providers.postgresql terraform-providers.powerdns terraform-providers.random terraform-providers.secret terraform-providers.selectel terraform-providers.spotinst terraform-providers.tencentcloud terraform-providers.tfe terraform-providers.tls terraform-providers.triton terraform-providers.ucloud terraform-providers.vault terraform-providers.vsphere terraform-providers.yandex terraform_plugins_test

Thanks @d-goldin , it got stale and the old version of scaleway is not compatible with a newer version of go in the master. I've bumped them all again and now nix-review passes for me.

With #72939 merged, that one can be abandoned now.


re-evaluating native module sources is not supported

Reinstelled the mirror with the latest node version, now I get:
(node:4214) fs: re-evaluating native module sources is not supported. If you are using the graceful-fs module, please update it to a more recent version.

:( Node 6.0 isn't supported by many packages yet.
Maybe we should use a node 5.X version in the installer and mention it in the readme?

Probably :/


Can't add file to project

Describe the bug
I have a .NET 6 console add. I can't use the "Add Current File to Project" feature.
Steps to reproduce

Create a new console app
Create a new .fs file.
Navigate to the file.
Choose the aforementioned action.

You get an error as follows:

Expected behaviour
The file is added to the project.
Machine info

OS: Windows 11
.NET SDK version: 6.0.23
Ionide version: 6.0.6

Missing project explorer info (and presumably by extension project info at all due to failures during project cracking/workspace initialization) would lead to this behavior. It would be nice if the we could add a condition like 'projects are loaded' to the commands to hide them until we're initialized.

Just wanted to add that I have the same issue on Mac.

OS: macOS Monterey 12.4
.NET SDK Version: 6.0.104
Ionide version: v6.0.6

@mhays118 is your project explorer also not showing up?
@baronfel if you want me to use a "diagnostics" version of Ionide to get you debug info, just let me know.

@isaacabraham Both the normal project/file explorer in VSCode and the F# Solution Explorer are showing up for me.

@mhays118 ok, so this is a separate thing for me - i'll raise a separate issue

Linked to #1753

I can't even see the option in Ionide any more:


IMvcCoreBuilder not accessible from modules

Hi guys,
I'm creating a Swagger module for my site built on the Orchard Core Framework. This requires me to register the ApiExplorer that normally is added by app.AddMvc(), but since we use app.AddMvcCore() I have to add it manually.
That's fine but the only thing I find a bit messy is that I can't access the instance of IMvcCoreBuilder that is created in Orchard.Mvc.ModularServiceCollectionExtensions.AddMvcModules().
So my module's Startup class looks like this...
...
public override void ConfigureServices(IServiceCollection services)
{
        // We are using the AddMvcCore for a more paired-down MVC stack, so need to explicitly add the Api Explorer services
        new MvcCoreBuilder(services, null).AddVersionedApiExplorer();

        // Register the Swagger generator, defining one or more Swagger documents
        services.AddSwaggerGen(c =>
        {
           ...
}

Seems a bit messy to initialize a new internal MvcCoreBuilder to me. I'm using the new Versioned API Explorer .
FYI I believe it should be possible to call AddMvcCore again such as,
services.AddMvcCore().AddVersionedApiExplorer();
But when i try that i get issues with a duplicate 'exists' route constraint being added from the internal Mvc framework.
ArgumentException: An item with the same key has already been added. Key: exists
System.ThrowHelper.ThrowAddingDuplicateWithKeyArgumentException(object key)
System.Collections.Generic.Dictionary.Insert(TKey key, TValue value, bool add)
Microsoft.Extensions.Options.OptionsCache.CreateOptions()
System.Threading.LazyInitializer.EnsureInitializedCore<T>(ref T target, ref bool initialized, ref object syncLock, Func<T> valueFactory)
Microsoft.Extensions.Options.OptionsCache.get_Value()
Microsoft.AspNetCore.Routing.DefaultInlineConstraintResolver..ctor(IOptions<RouteOptions> routeOptions)
System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitConstructor(ConstructorCallSite constructorCallSite, ServiceProvider provider)
Microsoft.Extensions.DependencyInjection.ServiceLookup.CallSiteRuntimeResolver.VisitTransient(TransientCallSite transientCallSite, ServiceProvider provider)
Microsoft.Extensions.DependencyInjection.ServiceProvider+<>c__DisplayClass16_0.<RealizeService>b__0(ServiceProvider provider)
Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService(IServiceProvider provider, Type serviceType)
Microsoft.Extensions.DependencyInjection.ServiceProviderServiceExtensions.GetRequiredService<T>(IServiceProvider provider)
Microsoft.AspNetCore.Builder.MapRouteRouteBuilderExtensions.MapRoute(IRouteBuilder routeBuilder, string name, string template, object defaults, object constraints, object dataTokens)
Microsoft.AspNetCore.Builder.MvcAreaRouteBuilderExtensions.MapAreaRoute(IRouteBuilder routeBuilder, string name, string areaName, string template, object defaults, object constraints, object dataTokens)
Microsoft.AspNetCore.Builder.MvcAreaRouteBuilderExtensions.MapAreaRoute(IRouteBuilder routeBuilder, string name, string areaName, string template, object defaults)
Company.Admin.Startup.Configure(IApplicationBuilder builder, IRouteBuilder routes, IServiceProvider serviceProvider) in Startup.cs
+
            routes.MapAreaRoute(
Microsoft.AspNetCore.Modules.ModularTenantRouterMiddleware.BuildTenantPipeline(ShellSettings shellSettings, IServiceProvider serviceProvider)
Microsoft.AspNetCore.Modules.ModularTenantRouterMiddleware+<Invoke>d__4.MoveNext()

To the best of my ability I think this occurs because  Microsoft.AspNetCore.Mvc.Internal.MvcCoreRouteOptionsSetup  is doing this twice:
public static void ConfigureRouting(RouteOptions options)
{
      options.ConstraintMap.Add("exists", typeof(KnownRouteValueConstraint));
}

This I believe is because the normal checks for stopping services from being registered twice in AddMvcCore (services.TryAddEnumerable) are being bypassed, since in Orchard we create a new ServicesCollection for every module's Startup class. (See ShellContainerFactory) So if two modules call AddMvcCore the framework will add multiple route constraints since the assumption that there is a single service collection is no longer true.
The troublesome lines:
Line 97: ShellContainerFactory
startup.ConfigureServices(startupServices);
tenantServiceCollection.Add(startupServices);

Anyway thought I'd document my journey, even though I have a workaround, if anyone else sees value in exposing IMvcCoreBuilder to module's startup classes or thinks there's value in allowing multiple calls to AddMvcCore I'm happy to help resolve.

Hey @jrestall I agree, the extension points to this subsystem could be better, and thanks for your write up.
I will take a look and see if we cant make this area more extensible.

Thanks @Jetski5822 , it turns out my workaround doesn't work anyway so I'm going to need to figure out a way to allow multiple calls to AddMvcCore after all. I don't think an extension point to access IMvcCoreBuilder will be enough since 3rd party libraries such as Microsoft.AspNetCore.Mvc.Versioning call AddMvcCore again internally.
In my case adding a call to app.AddApiVersioning() caused the same issue with constraints being added twice due to the package's code:
public static IServiceCollection AddApiVersioning(this IServiceCollection services, Action<ApiVersioningOptions> setupAction)
{
services.AddMvcCore((Action<MvcOptions>) (mvcOptions =>
      {
        mvcOptions.Conventions.Add((IApplicationModelConvention) new ApiVersionConvention(options.DefaultApiVersion, options.Conventions));
      }));

If we want to support libraries such as Swashbuckle and Microsoft.AspNetCore.Mvc.Versioning and likely many others, we will have to look at how ShellContainerFactory is re-creating service collections for each Startup class.
So to be clear what I'm asking is that putting a call such as the below in a Startup class would just work:
services.AddMvcCore().AddVersionedApiExplorer();

I'm currently thinking something along these lines as a fix, haven't tried it though, will hopefully have time to test and submit a pull request soon.
            foreach (var startup in moduleServiceProvider.GetServices<Microsoft.AspNetCore.Modules.IStartup>())
            {
                ...

                if (!featureServiceCollections.TryGetValue(feature, out featureServiceCollection))
                {
                    featureServiceCollections.Add(feature, featureServiceCollection = new ServiceCollection());
                }

                int previousTenantServiceCount = tenantServiceCollection.Count;
                startup.ConfigureServices(tenantServiceCollection);

                // Determine and store any new services the startup class has added for this feature
                var newTenantServices = tenantServiceCollection.Skip(previousTenantServiceCount);
                featureServiceCollection.Add(newTenantServices);
            }

Basically instead of passing a new collection into each startup class, I pass the same tenant collection of services. To determine the feature's services I just pull out the newly added items.

Hey @jrestall heres my Swagger module for O2 btw... https://github.com/Jetski5822/ECom/blob/master/src/ECom.Modules/ECom.Swagger/Startup.cs

Thanks @Jetski5822, the code for including XML comments is useful.
I'm really happy with how much easier it is to include swagger with Orchard in V2 as compared with V1. I remember having to do some serious hacks to get API Explorer to find the correct shell routes previously.

Yeah we need this

Hi @Jetski5822, I have good news then! Sébastien just merged my pull request that adds IStartupFilter support https://github.com/OrchardCMS/OrchardCore/commit/fa4ac675fdeabbfcf6d92841f44588cae15065a1.
I'm happy to close this issue now as we've fixed both the ability for libraries to call AddMvcCore multiple times and added support for libraries to initialize themselves via IStartupFilter e.g. Microsoft.AspNetCore.Mvc.Versioning.
Let me know if you disagree and we can re-open or we can create a new more specific issue.

@jrestall Thanks mate, your solution saved the day!


Disabled Progress should not start attached Live

Type of changes

[x] Bug fix
[ ] New feature
[ ] Documentation / docstrings
[ ] Tests
[ ] Other

Checklist

[x] I've run the latest black with default args on new code.
[x] I've updated CHANGELOG.md and CONTRIBUTORS.md where appropriate.
[x] I've added tests for new code.
[x] I accept that @willmcgugan may be pedantic in the code review.

Description
If you make a Progress that is disabled, it will still display an initial version of itself because it is the renderable of its attached Live, and the Live is started unconditionally in Progress.start. Example:
from rich.progress import Progress

with Progress(disable=True) as progress:
    task = progress.add_task("foobar", total=100)
    progress.update(task, advance=1)

print("done")

Running this produces this output:

With this patch, it produces this:

I added some tests for this, but they're poking around the internals to do it - let me know if this isn't your preferred style.

Thanks for this. Would you mind also testing the output? i.e. confirm that when disable=True nothing gets written. Just to catch any regression.

Thanks.


zoneinfo.zip: no such file or directory error when using OffPeakTimezone

I had the following issue, documented here: https://docs.gitlab.com/runner/faq/#zoneinfozip-no-such-file-or-directory-error-when-using-offpeaktimezone
Used the newest AMI as described in this comment

Closing as #9 has been merged. Thanks for your contribution!


Table on this page has excess empty lines

Table on "Creation of a console" page has some extraneous empty lines.

Document Details
⚠ Do not edit this section. It is required for docs.microsoft.com ➟ GitHub issue linking.

ID: a0577daa-0a60-0a8a-e156-f04480e69ba8
Version Independent ID: 09fe6e63-1060-7661-6891-d909a86b8ba8
Content: Creation of a Console - Windows Console
Content Source: docs/creation-of-a-console.md
Product: console
GitHub Login: @bitcrazed
Microsoft Alias: richturn

Complete.


Make logging work well for parallel processes

Make logging work well for parallel processes when n_jobs>0 in cross_val_score and GridSearch. Currently warnings for GaussianProcessSk and NeuralNetworkSk are suppressed.

@mastoffel I need some more context here, would you be able to provide it? What's the current problem? How would we size this issue?

actually, I think we should close this for now. Logging currently works in parallel, but likely to be revisited when we tackle #36

This is already marked as an issue that leads to the closing of #36 so I'd leave it open but take off ai-uk label perhaps?


[TACHYON-467] Client may still use a small remote read buffer size

https://tachyon.atlassian.net/browse/TACHYON-467
tachyon.user.remote.read.buffer.size.byte is set to 8MB as a default value. This is set in the tachyon-default.properties. But in RemoteBlockInStream.java, it still uses 1MB as a default value if this property is not set. A client process without the tachyon-default.properties on the class path may still use 1MB as the remote read buffer size.

Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins/job/Tachyon-Pull-Request-Builder/3249/
Test PASSed.

+1 LGTM

LGTM, thanks


Adding ESLint and Prettier

I suggest adding Prettier for consistent formatting and ESLint for code quality and linting to the project. I can take this up if you’d like! Please let me know.

@ranmerc Yes, sure. Go for it :)


[SE-000007] Extension "" is installed but unused

I have five lines that does not show the extension name. Makes it hard to remove...
The only non-Microsoft app installed, if Sentinel.

That's not ideal, let me have a look 😅

New version deployed to AppSource.


Flops calculation

Hi,
I followed your instruction in GETTING_STARTED.md to replace the conv.py in spconv package. But I got error: AttributeError: 'SparseConvTensor' object has no attribute 'flops'
Do we also need to modify SparseConvTensor class in core.py as well?
Thanks!

I added an attribute in SparseConvTensor class, and it works.

Cool!


Database structure

To improve both storage size and performance of the database its table layout needs to be redesigned.
This issue should also address

[ ] #76
[ ] #77

Closed in favor of #167.


⚠ Samsung has degraded performance

In 0c0a25c, Samsung (https://samsung.com) experienced degraded performance:

HTTP code: 200
Response time: 2519 ms

Resolved: Samsung performance has improved in a537a56.


Unreachable max zoom

The one line uncovered in the coverage testing doesn't look actually coverable, even with a tiny tiny bbox upstream

Coverage is back to 100 ¯_(ツ)_/¯!


Identity Server 4 as a background service

Sorry, this is a broad question but I'm writing it as a question/potential feature request,  I just want some feed back and thoughts on it.
I'm wondering if IdentityServer 4 could be run as a background service and whether that would be safe.
I've been switching back and forth on my project between using IdentityServer, and not.  I was just thinking it might be cool if Identity server could be run as a background application, and hosted in my current application.
There are a few benefits of it being hosted as a background service.
I can deploy a single application and the token server is not dependent on my application code.
and It would be easy to move to another instance of Identity server if the logic wasn't integrated directly with the project I'm using.
However I'm not sure how practical it would be, or whether there would be any security implications etc

ID4 server is framework for building an Identity server, however it heavily relies on use of .net core 3.1 as the underlying SDK.  While it is inherently written to be an asp.net web API, it could also be a console app, or a self hosted windows app, or even run on other OS like Linux, or Dockerized for easy portability.  There are lots of options.  I think you need to refine your requirements, and then see what you are comfortable with, because you will have to write code at some point to get a working solution.


TimeoutError is not exported to the outside world

This definition of TimeoutError in Promise+.js is not visible from userland anymore (
$$$$$cizccCCCCCCCCCcccc$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$?????)>>J$CLccc$??""
$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$????ii?iiJJJ$$??"""
$$$$$$$$$$$$$??$$$$????P$$$???iiiiicccccc<<????)Cicc$P""      ..   .,;;!!!
$$$$$CCCCCCC>>J>>>>cccccc>>>??C????CC>cccJ$??"""""         -``!!;!'  .!!!'
$$$$$??CCCCCCCCCCCff>>>>>J$$$P""""""""            ..,;;;;;;;!'`.,;;!'''
??????????????"""""'' `'              .,..,;;;;!!!'```..```' .,.,;;;- `,;'
                    .,.    ,;;----'''''''```````'  `''`,;;!!'''`..,;;'' ,;
---;;;;;;;-----'''''''''``'  --- `'  .,,ccc$$hcccccc,.  `' ,;;!!!'``,;;!!'
;;;;,,.,;-------''''''' ,;;!!-    .zJ$$$$$$$$$$$$$$$$$$$c,. `' ,;;!!!!' ,;
  ```'    -;;;!'''''-  `.,..   .zJ$$$$$$$$$$$$$$$$$$$$$$$$$$c, `!!'' ,;!!'
!!-  ' `,;;;;;;;;;;'''''```' ,c$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$c,  ;!!'' ,;
,;;;!!!!!!!!''``.,;;;;!'`'  z$$$$$$$$???"""""'.,,.`"?$$$$$$$$$$$  ``,;;!!!
;;..       --''```_..,;;!  J$$$$$$??,zcd$$$$$$$$$$$$$$$$$$$$$$$$h  ``'``'
```'''   ,;;''``.,.,;;,  ,$$$$$$F,z$$$$$$$$$$$$$$$$$$$c,`""?$$$$$h
!!!!;;;;,   --`!'''''''  $$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$$h.`"$$$$h .
`'''``.,;;;!;;;--;;   zF,$$$$$$$$$$?????$$$$$$$$$$$$$?????$$r ;?$$$ $.
!;.,..,.````.,;;;;  ,$P'J"$$$$$$P" .,c,,.J$$$$$$$$$"',cc,_`?h.`$$$$ $L
'``````'    .,..  ,$$". $ $$$$P",c$$$$$$$$$$$$$$$$',$$$$$$$$$$ $$$$ $$c,
!!!!!!!!!!!!!'''  J$',$ $.`$$P c$$$$$$$$$$$$$$$$$$,$$$$$$$$$$$ $$$$ $$$$C
   ``            J$ ,$P $$ ?$',$$$$???$$$$$$$$$$$$$$$??"""?$$$ <$$$ $$$$$
c           ;,  z$F,$$  `$$ $ ?$"      "$$$.?$$$ $$$P c??c, ?$.<$$',$$$$$F
$$h.  -!>   ('  $" $F ,F ?$ $ F ,="?$$c,`$$F $$"z$$',$' ,$$P $h.`$ ?$$$$$r
$$$$$hc,. ``'  J$ $P J$ . $$F L ",,J$$$F <$hc$$ "$L,`??????,J$$$.` z$$$$$
$$$$$$$$$$c,'' ?F,$',$F.: $$ c$c,,,,,c,,J$$$$$$$ ?$$$c,,,c$$$$$$F. $$$$$$
`"$$$$$$$$$$$c, $$',$$ :: $$$$$$$$F"',$$$$$$$$$$h ?$$$L;;$$$??$$$$ $$$$$$
   "?$$$$$$$$$$ $$$$$$ : .`F"$$$$$$$$$$$$""""?"""h $$$$$$$"$,J$$$$ $$$$$'
      "?$$$$$$$ $$$$$$.`.` h `$$$$$$$$$$$cccc$$c,zJ$$$$$P' $$$$$P',$$$$P
$.       `""?$$ $$$$$$$  ` "$c "?$$$$$$$$$$$$??$$$$$$$$" ,J$$$P",J$$$$P
..           `" ?$$$$$$h    ?$$c.`?$$$$$$$$$' . <$$$$$' ,$$$"  ,$$$$$"
!!>. .          `$$$$$$$h  . "$$$c,"$$$$$$$' `' `$$$P  ,$$$' ,c$$$$$'   ;!
```<!!!>         `$$$$$$$c     "$$$c`?$$$$$  : : $$$  ,$$P' z$$$$$$'   ;!!
$hc ```'  ;       `$$$$$$$.      ?$$c ?$$$$ .: : $$$  $$F ,J$$$$$$'   ;!!
.,..      '        `$$$$$$$       "$$h`$$$$ .' ' $$$ ,$$ ,J$$$$$$'    !!!
????P               `$$$$$$L       $$$ $$$F :.: J$$P J$F J$$$$$P     ;!!
-=<                  ?$$."$$       `$$ ?$$' `' z$$$F $P  $$$$$$'     !!'
cc                   `$$$c`?        ?$.`$$hc, cd$$F ,$'  $$$$$$     ;!!
                      $$$$c         `$$c$$$$$$$$$",c$'   $$$$$$     `!!
                      $$$$$          `?$$$$$$$$$$$$P'    $$$$$$> ..
                      $$$$$            `"?$$$$$$$P"      $$$$$$L $$c,
          !!         <$$$$$            zc,`"""',         <$$$$$$.`$$$$cc,
          !!         J$$$$P            `$$$$$$$' !'       $$$$$$L `$$$$$$h
         ;,          $$$$$L          `! J$$$$$',!!        $$$$$$$  `$$$$$$
          '         <$$$$$.           ! $$$$$$ !!         ?$$$$$$   `$$$$$
                   ,$$$$$$$c          `,`???? ;'         c,?$$$$'    `?$$$
                   $$$$$$$??           `!;;;;!     .     `h."?$P      `$$$
                  ,$$$$$$$h.            `'''      `'      `$$$P        `?$
                   $$$$$$$$h                      `!'      `"'           `
                  `$$$$$$$$F          !;     !    ;,
                   `$$$$$$$'         `!!>         `!
c,        ;,        `?$$$$P           !!>             .
$F        !!>         `""'            `!!            ;!>    <-
$F       `!!'                      ;!; '   `!        <!>    ;

Actually, everything declared in Promise+ is invisible from outside, including the TaskPool...

TaskPool and TimeoutError should be visible now...


Correct misspelling of 'Authenticate'

XmppClient contains a method named Autenticate, a misspelling of Authenticate.
As altering the API surface of the class would break backwards-compatibility, I opted to decorate the existing method with [Obsolete] instead of deleting it.
Documentation was moved to the correctly-spelled method; the existing method is no longer documented.
Additionally, there was a single internal usage of the misspelled method, which I have updated.

@pgstath how come this pull request has not gone in since dec 2016? Is this repository dead or is there a reason this pull request is not acceptable?

@pgstath how come this pull request has not gone in since dec 2016? Is this repository dead or is there a reason this pull request is not acceptable?


Noto Emoji not displaying?

I followed the steps to use Noto Emoji as the font for my workspace icons, but they are still not appearing. How can I fix this or figure out what's going wrong? Thanks.

What steps did you follow?

I can replicate this issue. It occurs for me when I attempt to use the following emoji: 🔇 (U+1F507). I've traced the problem down to parsing the configuration file.
Specifically, parser::text() refuses to emit any character that is larger than the size of a uint16_t. This emoji lies outside of that, so it instead writes 0xfffd (the replacement character).
Why is write_text_unicode a value_signal<write_text_unicode, uint16_t>? I feel like you actually need the full uint32_t here to support all possible UTF codepoints, right?

@skystrife 4-byte sequences aren't supported (goes beyond the sized integral type).. I'm almost done with the new implementation though, available on the refactor-cairo branch

I've pushed the changes to the master branch, so please try it out.

Yes, that seems to fix the issue.
Is it expected, though, that the new renderer makes fonts smaller than they were before? Using the same font settings, for example, produces

Where the top one is the master branch and the bottom one is v2.5.2. (I can create a separate issue for this if you'd like, though, since the emoji now render properly for me.)

No it's not intended but there's currently no scaling of point sizes. Will look into it.

Could you post the output of
$ xrdb -q | grep Xft

No output (empty). This is all that's there:
$ xrdb -q
rofi.color-active:	#191a16, #ffb52a, #21211b, #33332b, #ffb52a
rofi.color-enabled:	true
rofi.color-normal:	#191a16, #dbdbd5, #21211b, #33332b, #dbdbd5
rofi.color-urgent:	#191a16, #f9266a, #21211b, #33332b, #f9266a
rofi.color-window:	#191a16, #002b37, #33332b
rofi.font:	Meslo LG M DZ 12
rofi.separator-style:	solid

What about
$ xdpyinfo | grep -B2 resolution

$ xdpyinfo | grep -B2 resolution
screen #0:
  dimensions:    3120x1920 pixels (852x513 millimeters)
  resolution:    93x95 dots per inch

(I have two monitors, one rotated left.)
$ xrandr
$ xrandr
Screen 0: minimum 8 x 8, current 3120 x 1920, maximum 16384 x 16384
DVI-I-0 disconnected (normal left inverted right x axis y axis)
DVI-I-1 disconnected (normal left inverted right x axis y axis)
DVI-I-2 connected primary 1920x1200+0+0 (normal left inverted right x axis y axis) 518mm x 324mm
   1920x1200     59.95*+
   1600x1200     60.00
   1280x1024     75.02    60.02
   1280x960      60.00
   1152x864      75.00
   1024x768      75.03    70.07    60.00
   800x600       75.00    72.19    60.32    56.25
   640x480       75.00    72.81    59.94
HDMI-0 disconnected (normal left inverted right x axis y axis)
DVI-I-3 connected 1200x1920+1920+0 left (normal left inverted right x axis y axis) 518mm x 324mm
   1920x1200     59.95*+
   1920x1080     60.00
   1680x1050     59.95
   1600x1200     60.00
   1280x1024     60.02
   1280x960      60.00
   1024x768      60.00
   800x600       60.32
   640x480       59.94

I think I see how to fix this... working on a patch.


BigQuery - InsertRow - Not supporting UInt64

Hi,
Big Query Numeric column type max value should be larger than UInt64 (ulong) max value, but unfortunatly it is not being supported in the code:
https://github.com/googleapis/google-cloud-dotnet/blob/12bdab84277957b5581388ab0e8d0de92cdd9413/apis/Google.Cloud.BigQuery.V2/Google.Cloud.BigQuery.V2/BigQueryInsertRow.cs#L80
and I get this error:

Unable to use value of type System.UInt64 in BigQueryInsertRow

Is there any reason for that?
Thanks.

Mapping all the other integer types to BigQueryDbType.Int64, but just ulong to BigQueryNumeric would be quite error-prone in my view.
It's very easy to do it explicitly though - just cast your value to BigQueryNumeric. That makes your intent clear, and the existing code will handle that.

Thanks for the reply, I'll use the explicit cast as mentioned.


Failed to create group for the Default item read access role - sometime

When I tried to remove Anonymous from default item read access. But not for all collections.

Link ﹟417


Review of full field stitched data

Now we have the full field stitching pipeline, it would be great to review the algorithm.
Dealing with sun/shade:

Could we do a better job of accounting for exposure when converting 16 bit .bin to tif files?
Where there is overlap, can we select the best or do something more sophisticated when merging
Or would it help if the scans were done at a time during the day when there was less shade (e.g. in the AM)?
Or if the scans were done 2x during the day to make sure we have the entire area in only sun or only shade?

terraref/sites/ua-mac/Level_1/fullfield/2017-04-27/

Could we do a better job of accounting for exposure when converting 16 bit .bin to tif files?
a) We could.  First, if the data is going to be used directly for downstream processing, we should produce 16 bit tiff files so we don't lose data there.  Second, we could do a variant of high-dynamic range rendering --- IF the areas are washed out are just because the screen shows 8 bits of resolution, this is a way to try to show 16 bits of data on a regular screen.  I don't think that is the case here; because things are washed out, the data is really missing.
Where there is overlap, can we select the best or do something more sophisticated when merging
We can select the best, or, probably have some simple rule like "take the darkest pixel".  There are more clever things we could do if we wanted to make this montage look amazing and hide the boundaries, but those steps could introduce artifacts that make downstream processing harder.
Or would it help if the scans were done at a time during the day when there was less shade (e.g. in the AM)? ... Or if the scans were done 2x during the day to make sure we have the entire area in only sun or only shade?
Both of these would keep the data more consistent.  My belief is that it would be hard to be entirely consistent, so any algorithm will eventually have to deal with harsh shadows in the images.  If that is the case anyway, perhaps it isn't so important? --- or at least this is not a hard requirement that mandates the scheduling of the entire gantry.

@pless and @ZongyangLi What are the next steps?
There was a related discussion on using feature matching starting at https://github.com/terraref/computing-pipeline/issues/265#issuecomment-292232377

@pless and @ZongyangLi can you comment on the compressed files that @jterstriep generated? Some have 25x size reduction.

@ZongyangLi @pless @dlebauer I generated a VRT, TIFF and 10% resolution TIFF for 05/09. There are plants in this one. More interesting light effects too, in the north half this time.

@ZongyangLi can you please create a new issue that covers the implementation of these changes, and then close this one?

@ZongyangLi - reminder

further work will discuss in #326


Introduce a section on best practices

Since this is a fairly generic idea, for starts we could introduce a simple slide linking to Core Guidelines (https://isocpp.github.io/CppCoreGuidelines/CppCoreGuidelines). Maybe we could cherry-pick a few guidelines that are commonly considered good, for eg:

Interfaces -I.3 (singletons) as well as I.11 & I.13 (avoid raw_ptr ownership transfer, avoid passing arrays without length)
Errors - E.6 use RAII
Classes - C.131 Avoid trivial getters/setters

(Fairly opinionated list ;)

Some more ideas:

[ ] If you have a debate about how to do something, check core guidelines
[ ] When to use free functions (anonymous namespace) and when member functions
[ ] Keep headers clean, low number of includes?

And on the repetition side:

[ ] Use smart pointers
[ ] Pass by reference

Some more ideas:
* [ ]  If you have a debate about how to do something, check core guidelines

Definitely. I feel like for new comers this should give some pointers on doing things in mostly correct way

* [ ]  When to use free functions (anonymous namespace) and when member functions

* [ ]  Keep headers clean, low number of includes?

:+1:

And on the repetition side:
* [ ]  Use smart pointers
* [ ]  Pass by reference

Yeah this would be nice, esp. understanding ownership, ie. naked pointers as a view only resource

As we discussed just now in our meeting, here are the ATLAS coding standards. Which could be pointed to, together with those of the other experiments. :wink:
http://atlas-computing.web.cern.ch/atlas-computing/projects/qa/draft_guidelines-0.2.html

Decided to make a box/design for making good practices easily recognisable, and put the good practices right next to the places where the rationales can be explained.

I think we should discuss the SOLID principles at the end of the OOP chapter to give some guidance: https://en.wikipedia.org/wiki/SOLID

Closing in favour of #507


.htaccess rewrite to wrong url

if sodler is put as a subfolder of the host eg http://hostname.tld/solder/
when trying to access hostname.tld/solder/api/ you get redirect to hostname.tld/api
RewriteRule ^(.*)/$ /$1 [L,R=301]
rewrites the url incorrectly and is not needed in the .htaccess file as solder/api/ works for interaction with the technic platform.
I have tested this with a now fully function solder installation.

Solder is not intended to be setup as a subfolder of a host, but rather a subdomain with its own virtual host.


#Patterns Interceptor for error handling in your handlers@frontend and routes@backend

@mai-soup As a more advanced idea to handle your errors in your handlers - what a word play - use the concept of interceptors.
As illustrated here How to implement Interceptors in JS
It allows you to wrap an interceptor around each of your calls performing standard operations for e.g. HTTP 200 = Yeah success log and HTTP 403 = You're a naughty, naughty boy!

@mai-soup Looking into your backend I think the routers could use an interceptor as well. The error handling makes e.g. libraries.js very hard to read. And I assume u haven't even implemented all the error cases.

Closing as split up into #87 and #88


add "push" perk

lets you push players on the ground, rag dolling them, then you can pick up their bodys into your inventory

or just add this feature to never skip leg day


http: server gave HTTP response to HTTPS client

When I use docker registry as a chart storage tool, I get this error: "http: server gave HTTP response to HTTPS client".
# helm registry login <IP_ADDRESS>:5000 Username: coapps Password:  INFO[0010] Error logging in to endpoint, trying next endpoint  error="Get \"https://<IP_ADDRESS>:5000/v2/\": http: server gave HTTP response to HTTPS client" Error: Get "https://<IP_ADDRESS>:5000/v2/": http: server gave HTTP response to HTTPS client
But docker login can login normally
docker login <IP_ADDRESS>:5000 Username: coapps Password:  Login Succeeded
when use flag --insecure, login success, But push chart also failed.
helm push chartmuseum-3.6.2.tgz oci://<IP_ADDRESS>:5000/helm-charts Error: failed to do request: Head "https://<IP_ADDRESS>:5000/v2/helm-charts/chartmuseum/blobs/sha256:058e7bc5403d6a66519fd5b87372be382e935d0427d0fb142890d323d07726f3": http: server gave HTTP response to HTTPS client
Output of helm version:
version.BuildInfo{Version:"v3.8.1", GitCommit:"5cb9af4b1b271d11d7a97a71df3ac337dd94ad37", GitTreeState:"clean", GoVersion:"go1.17.5"}
Output of kubectl version:
Cloud Provider/Platform (AKS, GKE, Minikube etc.):

You probably need to download registry certificate and put it to trusted certs. Assuming name of cert is ca.crt and you are on linux follow the below:
cp [path_to_cert] /tmp/ca.crt
sudo cat /tmp/ca.crt | tee -a /etc/ssl/certs/ca-certificates.crt

I have same issue.


Lint command fixes

We'd probably want to use this in the build process so we need to do some more testing here.

Sorry wrong repo


environment variable for postgres hostname, removing hardcoded 'localhost'

Bug report
Describe the bug
The database connection string for the command supabase db reset appears to include a hardcoded 'localhost' for the hostname, and I am not finding anywhere I can override this.
I'm not really great at go or familiar with the project but searching the codebase for 'localhost' seems to reveal hardcoded connection strings with no environment variable option in a couple spots.
https://github.com/supabase/cli/blob/aa6a5121cbc52201a31d6308d987cebd34e46aad/internal/db/reset/reset.go#L54
To Reproduce
Using a VSCode dev container, I install the supabase cli, init and start a database (no problems), and then run supabase db reset
Expected behavior
Able to set a postgres connection string or minimally, host so that supabase db reset runs.
Current result:
# supabase db reset
Resetting database...
Initialising schema...
Error: failed to connect to `host=localhost user=postgres database=main`: dial error (dial tcp [::1]:54322: connect: cannot assign requested address)

System information

OS: linux, docker container
Version of supabase-cli: 1.5.0

Hi @ryanalta, could you share with us your devcontainer.json file to reproduce the error? I tried the following which works
{
  "name": "Node.js & TypeScript",
  "image": "mcr.microsoft.com/devcontainers/typescript-node:16-bullseye",
  "features": {
    "ghcr.io/devcontainers/features/docker-in-docker:1": {}
  },
  "postCreateCommand": "npm install -g supabase"
}

Here's the output from supabase db reset running inside dev container
node ➜ /workspaces/supabase-action-example (update-ci ✗) $ supabase db reset
Resetting database...
Initialising schema...
Applying migration<PHONE_NUMBER>4537_new_employee.sql...
Finished supabase db reset on branch main.

It doesn't seem like there's a way to specify the hostname in 1.12.2 either-- I'm using a devcontainer that extends an existing docker-compose with a custom Dockerfile for the main container that the devcontainer runs in (can't share), but I can force localhost for the container to resolve to my host IP with extra_hosts. I've added this to my docker-compose file and seems to work well enough:
    extra_hosts:
      - "localhost:host-gateway"

It would still be nice to be able to set the DB connection hostname via something in config.yml though, I'm not seeing if that exists but I do see a port.

Possibly related to https://github.com/supabase/cli/issues/270. Would be helpful to remove the hardcoded hostnames for other services too.

+1 that the extra_hosts addition works for me as well.
I have historically had a host.docker.internal:host-gateway in the extra_hosts for other reasons, but had not seen an impact on supabase until recently.

I'd also like the option of passing in a DB connection string so that I could use the reset command on hosted Supabase instances, too.

e option of passing in a DB connection string so that I could use the reset command on hosted Supabase instances, too.

I would agree as well. It's very frustrating not being able to reset hosted Supabase instances - a use case is having a clean instance to apply seeds to for UAT for our product team.

This is now supported on latest version of CLI. For eg.
supabase db reset --linked