Update/2x2 params 1575

Resolves https://github.com/VEuPathDB/web-eda/issues/1575 and I believe the current implementation would obviate https://github.com/VEuPathDB/web-eda/issues/1610.
The mockups request that:

The X and Y variables/values are stacked
The reference values are grayed out until X and Y variables are selected
Stratification variables wrap beneath the X and Y variables/values selections

Regarding 1 and 3, I applied these changes to all visualizations, not just 2x2 mosaic. I also stacked stratification variable/toggle selectors.
Regarding 2, I relied on CSS to gray out and disable those selections but it raises the need to handle disabling in the Select components' APIs in CoreUI. I will open an issue in CoreUI to do so.
Regarding 3 alone, I find that the toggle's label can get longer than necessary. I found myself wanting to limit its width by passing a styleOverride into the CoreUI Toggle component, but its API doesn't allow it as far as I can tell. I will open another issue in CoreUI to address this limitation, if others agree.
Testing the UI with the updated backend can be done on my dev site.
Screenshots (1: shows 2x2 without wrapping; 2: shows 2x2 with wrapping; 3: shows line plot potentially obviating the aforementioned issue):

I think the screenshots look pretty good. In the case where the stratification variable selection wraps below the x- and y-axis variable sections, I find it a little hard to process. I wonder if adding some indentation (left margin) to the selectors would help, to make it more clear that they are subordinate to the heading.
I also think the no data toggle should go above the variable selectors in the stratification section, but it's possible that's been discussed before. It just seems a little lost being beneath them.
I'll take a look at the implementation shortly.


LIME-45 Relay PersistentSessionId and ClientSessionId to AuditEventUser.

Proposed changes
What changed
Relays PersistentSessionId and ClientSessionId from SessionService/SessionRequest to AuditService/AuditEventUser via SessionItem.
Why did it change
The allows PersistentSessionId and ClientSessionId to appear in the user section of AuditEvents.
PersistentSessionId= "persistent_session_id"
ClientSessionId = "govuk_signin_journey_id"
ClientSessionId follows IPVAuthorisationService for variable naming.
Linked with Common-Lamdas 20
Issue tracking

LIME-45
ADR-0039
RFC-0023

Please ensure you manually test these changes by deploying a fraud CRI stack to AWS and set the POWERTOOLS_LOG_LEVEL environment variable for your lambda function to DEBUG so that the audit event will be output in the logs

Thanks!, this is much better than polling the sqs queue and trying to find the latest messages.


Description of PSAvoidUsingWriteHost needs tweaking

File 'Test.ps1' uses Write-Host. This is not
recommended because it may not work in some hosts or
there may even be no hosts at all. Use Write-Output
instead.

What is meant by or there may even be no hosts at all?  There is always an app that is hosting the PowerShell engine, right?  Now that app might now support PSHostUserInterface or even if it did it might send Write-Host data to /dev/nul.  But that is covered by the statement  it may not work in some hosts.
I propose this description be simplified to:

File 'Test.ps1' uses Write-Host. This is not
recommended because it may not work in some hosts.
Use Write-Output instead.

Keith.
Thanks for the recommendation.
We worked with June to update some of the documentation.
Here is the update:
Avoid using Write-Host because it might not work in
all hosts, does not work when there is no host, and
(prior to PS 5.0) cannot be suppressed, captured, or
redirected. Instead, use Write-Output,
Write-Verbose, or Write-Information.

That is a lot better! :+1:

I definitely like the update to the guidance!


[Event Request] codeunit 7041 "Price Asset - Item" - OnAfterFilterPriceLines

Please add new event in codeunit 7041 "Price Asset - Item"
procedure FilterPriceLines(PriceAsset: Record "Price Asset"; var PriceListLine: Record "Price List Line") Result: Boolean;
begin
    PriceListLine.SetRange("Asset Type", PriceAsset."Asset Type");
    PriceListLine.SetRange("Asset No.", PriceAsset."Asset No.");
    PriceListLine.SetFilter("Variant Code", '%1|%2', PriceAsset."Variant Code", '');
    OnAfterFilterPriceLines(PriceAsset, PriceListLine, Result);
end;

[IntegrationEvent(false, false)]
procedure OnAfterFilterPriceLines(PriceAsset: Record "Price Asset"; var PriceListLine: Record "Price List Line"; var Result: Boolean)
begin
end;

Thanks for reporting this. We agree, and we’ll publish a fix asap, either in an update for the current version or in the next major release. Please do not reply to this, as we do not monitor closed issues. If you have follow-up questions or requests, please create a new issue where you reference this one.
Build ID: 47651.


Highlight certain words of a paragraph to bold

I've multiple paragraphs in which i need to make few words as bold and insert into word. For which i had tried the below method for a single paragraph.
import docx
from docx import Document
start_bold = "\033[1m"
end_bold = "\033[0m"
doc = docx.Document()
IntroPara = "This document details the results of the  Quarterly Update Testing. As per Oracle’s timetable for quarterly releases for wave , the upgrade took place on non-production environments on the <day(dd/mm/yy)>, and into the production environment on the <day(dd/mm/yy)>."
words_to_highlight = ["", "", "<day(dd/mm/yy)>"]
for word in words_to_highlight:
IntroPara = IntroPara.replace(word, f"{start_bold}{word}{end_bold}")
IntPara = IntroPara
print(IntPara)
doc.add_paragraph(IntPara)
doc.save('Report.docx')
this gives me highlted words but when i try to save it as a paragraph in word it is throwing below error.
### errors:
This document details the results of the  Quarterly Update Testing. As per Oracle’s timetable for quarterly releases for wave , the upgrade took place on non-production environments on the <day(dd/mm/yy)>, and into the production environment on the <day(dd/mm/yy)>.
Traceback (most recent call last):
File "C:\Users\TSTUSER\PycharmProjects\Test\parag.py", line 21, in
doc.add_paragraph(IntPara)
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\document.py", line 69, in add_paragraph
return self._body.add_paragraph(text, style)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\blkcntnr.py", line 55, in add_paragraph
paragraph.add_run(text)
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\text\paragraph.py", line 41, in add_run
run.text = text
^^^^^^^^
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\text\run.py", line 213, in text
self._r.text = text
^^^^^^^^^^^^
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\text\run.py", line 129, in text
_RunContentAppender.append_to_run_from_text(self, text)
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\text\run.py", line 248, in append_to_run_from_text
appender.add_text(text)
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\text\run.py", line 254, in add_text
self.flush()
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\text\run.py", line 275, in flush
self._r.add_t(text)
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\text\run.py", line 41, in add_t
t = self._add_t(text=text)
^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\TSTUSER\PycharmProjects\test.venv\Lib\site-packages\docx\oxml\xmlchemy.py", line 300, in _add_child
setattr(child, key, value)
File "src\lxml\etree.pyx", line 1063, in lxml.etree._Element.text.set
File "src\lxml\apihelpers.pxi", line 749, in lxml.etree._setNodeText
File "src\lxml\apihelpers.pxi", line 737, in lxml.etree._createTextNode
File "src\lxml\apihelpers.pxi", line 1530, in lxml.etree._utf8
ValueError: All strings must be XML compatible: Unicode or ASCII, no NULL bytes or control characters
Please help me with a fix or alternate way to highlighted words inside the paragraph instead of addrun.

for example using a self made method add_text() with the necessary parameters and the code within it
`
    if not paragraph:
        paragraph: Paragraph = self.doc.add_paragraph()

    style = self.doc.styles[heading_style]

    paragraph.paragraph_format.space_after = Pt(space_after)
    paragraph.paragraph_format.space_before = Pt(space_before)
    paragraph.paragraph_format.keep_with_next = keep_with_next
    paragraph.paragraph_format.alignment = halignment
    paragraph.style = style

    run = paragraph.add_run(text)
    run.italic = italic
    run.bold = bold

    if font:
        run.font.name = font

    if heading_style == 'Normal' and font != 11:
        run.font.size = Pt(font_size)

    return paragraph

`


Unable to parse a mpd

Checklist

[X] I'm reporting a bug unrelated to a specific site
[X] I've verified that I'm running yt-dlp version 2022.07.18 (update instructions) or later (specify commit)
[X] I've checked that all provided URLs are playable in a browser with the same IP and same login details
[X] I've checked that all URLs and arguments with special characters are properly quoted or escaped
[X] I've searched the bugtracker for similar issues including closed ones. DO NOT post duplicates
[X] I've read the guidelines for opening an issue

Provide a description that is worded well enough to be understood
i am unable to parse a mpd.
mpd link - https://m-9857s3.ll.dash.row.aiv-cdn.net/iad_2/da5b/d190/94c6/458a-a2d3-a52b993d431f/f011db32-ef5d-44ab-a34d-5b3e591e4a4e_corrected.mpd
Provide verbose output that clearly demonstrates the problem

[X] Run your yt-dlp command with -vU flag added (yt-dlp -vU <your command line>)
[X] Copy the WHOLE output (starting with [debug] Command-line config) and insert it below

Complete Verbose Output
[debug] Command-line config: ['-vU', '--allow-unplayable-formats', '--no-warnings', '-F', 'https://m-9857s3.ll.dash.row.aiv-cdn.net/iad_2/da5b/d190/94c6/458a-a2d3-a52b993d431f/f011db32-ef5d-44ab-a34d-5b3e591e4a4e_corrected.mpd']
[debug] Encodings: locale cp1252, fs utf-8, pref cp1252, out utf-8, error utf-8, screen utf-8
[debug] yt-dlp version 2022.07.18 [135f05e]
[debug] Python 3.10.5 (CPython 64bit) - Windows-10-10.0.19044-SP0
[debug] Checking exe version: ffmpeg -bsfs
[debug] Checking exe version: avconv -bsfs
[debug] Checking exe version: ffprobe -bsfs
[debug] Checking exe version: avprobe -bsfs
[debug] exe versions: none
[debug] Optional libraries: Cryptodome-3.15.0, brotli-1.0.9, certifi-2022.06.15, mutagen-1.45.1, sqlite3-2.6.0, websockets-10.3
[debug] Proxy map: {}
[debug] Fetching release info: https://api.github.com/repos/yt-dlp/yt-dlp/releases/latest
Latest version: 2022.07.18, Current version: 2022.07.18
yt-dlp is up to date (2022.07.18)
[debug] [generic] Extracting URL: https://m-9857s3.ll.dash.row.aiv-cdn.net/iad_2/da5b/d190/94c6/458a-a2d3-a52b993d431f/f011db32-ef5d-44ab-a34d-5b3e591e4a4e_corrected.mpd
[generic] f011db32-ef5d-44ab-a34d-5b3e591e4a4e_corrected: Downloading webpage
[generic] f011db32-ef5d-44ab-a34d-5b3e591e4a4e_corrected: Extracting information
ERROR: An extractor error has occurred. (caused by KeyError('sourceURL')); please report this issue on  https://github.com/yt-dlp/yt-dlp/issues?q= , filling out the appropriate issue template. Confirm you are on the latest version using  yt-dlp -U
  File "C:\Users\Dell\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 644, in extract
    ie_result = self._real_extract(url)
  File "C:\Users\Dell\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\generic.py", line 2915, in _real_extract
    info_dict['formats'], info_dict['subtitles'] = self._parse_mpd_formats_and_subtitles(
  File "C:\Users\Dell\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 2882, in _parse_mpd_formats_and_subtitles
    representation_ms_info = extract_multisegment_info(representation, adaption_set_ms_info)
  File "C:\Users\Dell\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 2766, in extract_multisegment_info
    extract_Initialization(segment_list)
  File "C:\Users\Dell\AppData\Local\Programs\Python\Python310\lib\site-packages\yt_dlp\extractor\common.py", line 2761, in extract_Initialization
    ms_info['initialization_url'] = initialization.attrib['sourceURL']
KeyError: 'sourceURL'

Provide non-DRM example


✨ Add ChecksumType field in image

What this PR does / why we need it:
This PR adds support for the new BMH field ChecksumType in the image, and updates all the dependencies.

/cc @kashifest
/cc @Jaakko-Os
/cc @fmuyassarov

/test-v1a4-integration

/test-v1a4-integration

@maelk, is md5 a default ChecksumType in CAPM3? are sha256, sha512 can be used as well?

yes, metal3-dev-env uses md5sum to compute the checksum. Other are also supported.

Looks good to me. But someone who has more expertise in repo shall review.


Treemap feedback

I've been playing with the new treemap chart and noticed few things:

I'm not able to show labels on sub items (the drill down example)
There is 1px border when using childBorderWidth:0, should be 0. Setting anything different that 0 works as expected.
I can't get rid of the text at the left bottom corner (the breadcrumb?), shows the name property in the series, I need to set name:''. I've tried breadcrumb:{show:false} but that doesn't help

Anyway, really glad to see all the new charts in 2.2.3!

hi thank you for using the treemap chart.
please give me your options that can not work.

Hi, here is the code:
{
"series": [
{
"name": "test",
"type": "treemap",
"itemStyle": {
"normal": {
"label": {
"show": true,
"formatter": "{b}"
},
"childBorderWidth": 0,
"childBorderColor": "#ffffff",
"borderWidth": 0,
"borderColor": "#ffffff"
},
"emphasis": {
"label": {
"show": false
}
}
},
"data": [
{
"name": "Monday",
"children": [
{
"name": "test1",
"value": 10
},
{
"name": "test2",
"value": 20
},
{
"name": "test3",
"children": [
{
"name": "test a",
"value": 10
},
{
"name": "test b",
"value": 20
},
{
"name": "test c",
"value": 30
}
],
"value": 30
}
],
"value": 100
},
{
"name": "Tuesday",
"value": 20
},
{
"name": "Wednesday",
"value": 40
},
{
"name": "Thursday",
"value": 60
},
{
"name": "Friday",
"value": 60
},
{
"name": "Saturday",
"value": 75
},
{
"name": "Sunday",
"value": 70
}
]
}
]
}
and here is the output:

childBorderWidth:0 should render child items without border
not sure how to display labels test1, test2, test3
seems like there are only two levels of data supported, test a, test b, test c is not considered
sorry, the breadcrumb:{show:false} works fine

Anyway, this is an awesome chart!

I will research it.  Please wait for days.

I have fixed the bug of childBorderWidth:0. You will see it in the next version.
not support to display sub labels and sub sub data, because we cant find a good way to show them. You have to make use of drilling down
My English is not so good. If you cant understand, you can commit again :)

Thanks for that!
Regarding sub data, what if you do something that d3.js does, for example:

the structure of this tree map is:
root
- button
- tabset
- tab
- tab
- button
- button
Which means parents with children don't show label - they are just containers. This allows better control of how things group.
Thanks again!

sorry, the structure got flatten after commenting, here is a snip:

we dont support it now.  sorry.

no problem - thanks a lot for the border fix in 2.2.5!


Move rsocket modules to spring-cloud-incubator

Rather than labeling modules in this project as a tech preview, the team decided to move them to the incubator until such time as rsocket-java is up to date with the routing/forwarding spec extension

See https://github.com/spring-cloud-incubator/spring-cloud-rsocket


Input control are missing support for onTouched when used with formControlName

I'm using DxTextBox on an angular form.  When a users tabs off the DxTextBox, it should call the onTouched method registered in DxTextBoxValueDirective to mark to form control as touched.  This allow the form to choose to display error messages after a user tabs off an unchanged text box.  It looks like the value accessor missed binding the onFocusOut event of the text box to the call to onTouched.
This should be similar to how the valueChanged event of the text box triggers a call to onChange.
A quick scan of code appears to show this linkage may be missing from several of the input controls.
I'm using   "devextreme-angular": "16.2.3-rc.3"

Hi,

Currently, DevExtreme widgets support built-in validation. Here is an example: https://js.devexpress.com/Demos/WidgetsGallery/Demo/Validation/Overview/Angular/Light/

We will consider enhancing support for ControlValueAccessor in future releases.

We have implemented this feature, and you can use it using version 17.1.9-beta.2 and 17.2.4 beta.1


Map spawning a copy of a monster just caught

Must be a bug but I had two instances yesterday where I went and captured a rare, came back, and the same rare came up again in the same location, lIke 10 mintestinal after. I then went out and nothing was there. Literally just happened to me too. Now I know not to go out if the 'coincidence' happens again but just thought I'd bring it up.

Effin smart text...'like 10 minutes after'

This is a bug with pokemon go, not this program. I've seen it happen a few times. A pokemon will appear, It'll expire, and, a few minutes later it will reappear. It is the same pokemon, so if you've already caught it you won't be able to catch it again.

I consider it a second chance bug. Had a few instances where I didn't get there in time and a few minutes later got my chance. But as RedHerring27 posted - once you caught it (or if it escaped) you won't see it again.


Code Base: Persist User Metric Classifications

As discussed during your milestone 3 presentation, it would be really useful if the app saved or exported user selections.  This would make using the page more convenient since you would not have to re-click the buttons every time you reload the page.  It would also be useful for future neural network or data mining projects.

Moving forward, we would like to implement a database system that would solve this issue. Thanks for the suggestion!


Is everything OK with this piece of code?

https://github.com/openssl/openssl/blob/859521e57970027c2ec763928753a1e5f843cf69/providers/implementations/asymciphers/rsa_enc.c#L164-L169
I think it should be the same as
https://github.com/openssl/openssl/blob/859521e57970027c2ec763928753a1e5f843cf69/providers/implementations/asymciphers/rsa_enc.c#L249-L257

Yeah, the second case seems to be correct. Will you submit a PR?

Yes


Can't get my strip to light up :(

Hey guys,
I finally got everything setup and was so excited to see my light strip light up, but nothing happened. :(
I successfully installed the software on the board and connected all the hardware:

This is the lightstrip I am using:
https://www.amazon.com/gp/product/B079ZZ9BYH/ref=ppx_yo_dt_b_asin_title_o00_s01?ie=UTF8&psc=1
I can successfully connect to the Web Interface as well:

Here is my definitions.h
//#define USE_WS2812FX_DMA       // LED_PIN is ignored & set to RX/GPIO3  Uses WS2812FX, see: https://github.com/kitesurfer1404/WS2812FX
//#define USE_WS2812FX_UART1     // LED_PIN is ignored & set to D4/GPIO2  Uses WS2812FX, see: https://github.com/kitesurfer1404/WS2812FX
//#define USE_WS2812FX_UART2     // LED_PIN is ignored & set to TX/GPIO1  Uses WS2812FX, see: https://github.com/kitesurfer1404/WS2812FX

// Neopixel
#define LED_PIN 2       // LED_PIN (14 / D5) where neopixel / WS2811 strip is attached
#define NUMLEDS 24       // Number of leds in the strip
//#define LED_TYPE_WS2811  // Uncomment if LED type uses 400 KHz (classic 'v1' (not v2) FLORA pixels, WS2811 drivers)
#define BUILTIN_LED 2    // ESP-12F has the built in LED on GPIO2, see https://github.com/esp8266/Arduino/issues/2192
#define BUTTON 4         // Input pin (4 / D2) for switching the LED strip on / off, connect this PIN to ground to trigger button.

#define HOSTNAME "McLighting01"   // Friedly hostname

#define HTTP_OTA             // If defined, enable ESP8266HTTPUpdateServer OTA code.
//#define ENABLE_OTA         // If defined, enable Arduino OTA code.
#define ENABLE_AMQTT         // If defined, enable Async MQTT code, see: https://github.com/marvinroger/async-mqtt-client
//#define ENABLE_MQTT        // If defined, enable MQTT client code, see: https://github.com/toblum/McLighting/wiki/MQTT-API
#define ENABLE_HOMEASSISTANT // If defined, enable Homeassistant integration, ENABLE_MQTT or ENABLE_AMQTT must be active
#define ENABLE_BUTTON        // If defined, enable button handling code, see: https://github.com/toblum/McLighting/wiki/Button-control
//#define MQTT_HOME_ASSISTANT_SUPPORT // If defined, use AMQTT and select Tools -> IwIP Variant -> Higher Bandwidth
#define ENABLE_LEGACY_ANIMATIONS // Dont disbale this for now
#define ENABLE_E131              // E1.31 implementation
#define USE_HTML_MIN_GZ             //comment for using index.htm & edit.htm from SPIFFs instead of PROGMEM
//#define CUSTOM_WS2812FX_ANIMATIONS  //uncomment and put animations in "custom_ws2812fx_animations.h"

#ifdef ENABLE_E131
  #define START_UNIVERSE 1                    // First DMX Universe to listen for
  #define END_UNIVERSE 2                      // Last Universe to listen for, starting at UNIVERSE
                                              // MUST: END_UNIVERSE >= START_UNIVERSE
#endif

//#define WIFIMGR_PORTAL_TIMEOUT 180
//#define WIFIMGR_SET_MANUAL_IP

#ifdef WIFIMGR_SET_MANUAL_IP
  uint8_t _ip[4] = {192,168,0,128};
  uint8_t _gw[4] = {192,168,0,1};
  uint8_t _sn[4] = {255,255,255,0};
#endif

#ifdef MQTT_HOME_ASSISTANT_SUPPORT
  #define MQTT_HOME_ASSISTANT_0_87_SUPPORT // Comment if using HA version < 0.87
#endif

#if defined(USE_WS2812FX_DMA) and defined(USE_WS2812FX_UART)
#error "Cant have both DMA and UART method."
#endif
#if defined(ENABLE_MQTT) and defined(ENABLE_AMQTT)
#error "Cant have both PubSubClient and AsyncMQTT enabled. Choose either one."
#endif
#if ( (defined(ENABLE_HOMEASSISTANT) and !defined(ENABLE_MQTT)) and (defined(ENABLE_HOMEASSISTANT) and !defined(ENABLE_AMQTT)) )
#error "To use HA, you have to either enable PubCubClient or AsyncMQTT"
#endif
#if ( !defined(ENABLE_HOMEASSISTANT) and defined(MQTT_HOME_ASSISTANT_SUPPORT) )
#error "To use HA support, you have to either enable Homeassistant component"
#endif

// parameters for automatically cycling favorite patterns
uint32_t autoParams[][4] = {  // color, speed, mode, duration (milliseconds)
  {0xff0000, 200,  1,  5000}, // blink red for 5 seconds
  {0x00ff00, 200,  3, 10000}, // wipe green for 10 seconds
  {0x0000ff, 200, 14,  5000}, // dual scan blue for 5 seconds
  {0x0000ff, 200, 45, 15000}  // fireworks for 15 seconds
};

#if defined(ENABLE_MQTT) or defined(ENABLE_AMQTT)

  const char mqtt_will_topic[] = HOSTNAME "/status";
  const char mqtt_will_payload[] = "OFFLINE";
  const char mqtt_intopic[] = HOSTNAME "/in";
  const char mqtt_outtopic[] = HOSTNAME "/out";
  bool mqtt_lwt_boot_flag = true;

  #ifdef ENABLE_MQTT
    #define MQTT_MAX_PACKET_SIZE 512
    #define MQTT_MAX_RECONNECT_TRIES 4

    int mqtt_reconnect_retries = 0;
    uint8_t qossub = 0; // PubSubClient can sub qos 0 or 1
  #endif

  #ifdef ENABLE_AMQTT
    uint8_t qossub = 0; // AMQTT can sub qos 0 or 1 or 2
    uint8_t qospub = 0; // AMQTT can pub qos 0 or 1 or 2
  #endif

  #ifdef ENABLE_HOMEASSISTANT
    const char mqtt_ha_state_in[] = "home/" HOSTNAME "_ha/state/in";
    const char mqtt_ha_state_out[] = "home/" HOSTNAME "_ha/state/out";

    const char* on_cmd = "ON";
    const char* off_cmd = "OFF";
    bool stateOn = false;
    bool animation_on = false;
    bool new_ha_mqtt_msg = false;
    uint16_t color_temp = 327; // min is 154 and max is 500
  #endif

  //#define ENABLE_MQTT_HOSTNAME_CHIPID          // Uncomment/comment to add ESPChipID to end of MQTT hostname
  #ifdef ENABLE_MQTT_HOSTNAME_CHIPID
    char mqtt_clientid[64];
  #else
    const char mqtt_clientid[] = HOSTNAME;
  #endif

  char mqtt_host[64] = "";
  char mqtt_port[6] = "";
  char mqtt_user[32] = "";
  char mqtt_pass[32] = "";
#endif

// ***************************************************************************
// Global variables / definitions
// ***************************************************************************
#define DBG_OUTPUT_PORT Serial  // Set debug output port

// List of all color modes
#ifdef ENABLE_LEGACY_ANIMATIONS
  #ifdef ENABLE_E131
    enum MODE { SET_MODE, HOLD, OFF, SETCOLOR, SETSPEED, BRIGHTNESS, WIPE, RAINBOW, RAINBOWCYCLE, THEATERCHASE, TWINKLERANDOM, THEATERCHASERAINBOW, TV, CUSTOM, E131 };
  #else
    enum MODE { SET_MODE, HOLD, OFF, SETCOLOR, SETSPEED, BRIGHTNESS, WIPE, RAINBOW, RAINBOWCYCLE, THEATERCHASE, TWINKLERANDOM, THEATERCHASERAINBOW, TV, CUSTOM };
  #endif
  MODE mode = RAINBOW;         // Standard mode that is active when software starts
  bool exit_func = false;      // Global helper variable to get out of the color modes when mode changes
#else
  enum MODE { SET_MODE, HOLD, OFF, SETCOLOR, SETSPEED, BRIGHTNESS, CUSTOM };
  MODE mode = SET_MODE;        // Standard mode that is active when software starts
#endif

MODE prevmode = mode;

int ws2812fx_speed = 196;   // Global variable for storing the delay between color changes --> smaller == faster
int brightness = 196;       // Global variable for storing the brightness (255 == 100%)

int ws2812fx_mode = 0;      // Helper variable to set WS2812FX modes

bool shouldSaveConfig = false;  // For WiFiManger custom config

struct ledstate             // Data structure to store a state of a single led
{
  uint8_t red;
  uint8_t green;
  uint8_t blue;
};

typedef struct ledstate LEDState;     // Define the datatype LEDState
LEDState ledstates[NUMLEDS];          // Get an array of led states to store the state of the whole strip
LEDState main_color = { 255, 0, 0 };  // Store the "main color" of the strip used in single color modes

#define ENABLE_STATE_SAVE_SPIFFS        // If defined, saves state on SPIFFS
//#define ENABLE_STATE_SAVE_EEPROM        // If defined, save state on reboot
#ifdef ENABLE_STATE_SAVE_EEPROM
  char current_state[32];               // Keeps the current state representation
  char last_state[32];                  // Save the last state as string representation
  unsigned long time_statechange = 0;   // Time when the state last changed
  int timeout_statechange_save = 5000;  // Timeout in ms to wait before state is saved
  bool state_save_requested = false;    // State has to be saved after timeout
#endif
#ifdef ENABLE_STATE_SAVE_SPIFFS
  bool updateStateFS = false;
#endif

// Button handling
#ifdef ENABLE_BUTTON
  #define BTN_MODE_SHORT "STA| 1|  0|245|196|255|255|255"   // Static white
  #define BTN_MODE_MEDIUM "STA| 1| 48|245|196|255|102|  0"    // Fire flicker
  #define BTN_MODE_LONG "STA| 1| 46|253|196|255|102|  0"  // Fireworks random

  unsigned long keyPrevMillis = 0;
  const unsigned long keySampleIntervalMs = 25;
  byte longKeyPressCountMax = 80;       // 80 * 25 = 2000 ms
  byte mediumKeyPressCountMin = 20;     // 20 * 25 = 500 ms
  byte KeyPressCount = 0;
  byte prevKeyState = HIGH;             // button is active low
  boolean buttonState = false;
#endif

I would be so grateful if someone could take a look at this and see what I am doing wrong. When I connect power only the blue board light comes on. The strips don't do anything at all. I try to select different colors in the web interface and nothing happens either. So discouraged right now. Hope someone has any insight!

I also just measured Voltage on various parts:
Bottom part of the board: 5.16V
Top part of the board: 3.29V
End of the LED strip (other side of where I am connecting my power and data): 5.05V
So it appears as if everything gets power?

I uncommented this line:
#define LED_TYPE_WS2811  // Uncomment if LED type uses 400 KHz (classic 'v1' (not v2) FLORA pixels, WS2811 drivers)
but it still doesn't seem to work.

Alright, now I connected my data to PIN 1 on the board (as was shown on the hardware page):

And changed the LED pin line back to14:
#define LED_PIN 14       // LED_PIN (14 / D5) where neopixel / WS2811 strip is attached
Now two random LED's on the strip actually light up:

But just those two and any settings on the web interface don't make any changes on the strip.
Wits end.

Hi @VeniceNerd,
this should usually work as far as I can tell from the photos.
The PIN and number of LEDs can now be configure via HTTP call: https://github.com/toblum/McLighting/wiki/HTTP-REST-API#get-pixelconf-http10
Please try http:///pixelconf?pin=14 and http:///pixelconf?ct=
If that diesn't work, please try out a simple strandtest script (e.g. https://github.com/adafruit/Adafruit_NeoPixel/blob/master/examples/strandtest/strandtest.ino) to make sure the strip itself works and to check where the problem originates.
Regards
Tobias

@VeniceNerd so, I'm going to give this a shot (I've literally been in your shoes).
A few things just looking over your setup...

I'm not clear what you have connected on your data line. If is a fuse you're probably ok. If it's a resistor, check the resistance as I used a too high value and it caused some issues.
You mention that you're using LED_PIN 14, but your strip is connected to what appears to be D1. Is that correct?

#define LED_PIN 14 // LED_PIN (14 / D5) where neopixel / WS2811 strip is attached

If you're getting DC voltage across the board, then the most likely culprit is something on the data line (either noise or something else).
When I built mine, I had failed in several ways before finally getting it right:

Wrong resistor value inline on the data line
Backwards connected data line (doh)
Unusual pins lighting up (actually it was quite similar to what you describe) when I was connected to the wrong data pin.

I suggest re-flashing your ESP with the full code you have just to make sure you have your changes propagated. I made this mistake as well. I guess it's the side effect of working on this stuff late at night after a long day . :)
I also suggest using a short strip with 4-5 LEDs on it for testing. I've been using that for validating my board wiring before doing an install somewhere in my house.

The PIN and number of LEDs can now be configure via HTTP call: https://github.com/toblum/McLighting/wiki/HTTP-REST-API#get-pixelconf-http10
Please try http:///pixelconf?pin=14 and http:///pixelconf?ct=

I'm so sorry, but I don't really know how to run those commands. Where do I need to go to run them?

If that diesn't work, please try out a simple strandtest script (e.g. https://github.com/adafruit/Adafruit_NeoPixel/blob/master/examples/strandtest/strandtest.ino) to make sure the strip itself works and to check where the problem originates.

Sorry again, but what should I do with that script? Do I upload it somewhere?

Thank you!!!!

I'm not clear what you have connected on your data line. If is a fuse you're probably ok. If it's a resistor, check the resistance as I used a too high value and it caused some issues.

The data line just goes from the Arduino to the strip. It's a very short cable so I currently do not have anything connected on the data line.

You mention that you're using LED_PIN 14, but your strip is connected to what appears to be D1. Is that correct?

That is correct. My strip is connected to D1. Should I be changing it to D1 in the code as well? I noticed that the tutorial on this site had D1 on the board but D14 in the software.

Backwards connected data line (doh)

What does that mean?

I suggest re-flashing your ESP with the full code you have just to make sure you have your changes propagated. I made this mistake as well. I guess it's the side effect of working on this stuff late at night after a long day . :)

How do you re-flash the ESP with the full code? I have just been making changes in the definitions.h and then compile and re-upload.
Also: should I uncomment this line or leave it commented?

//#define LED_TYPE_WS2811  // Uncomment if LED type uses 400 KHz (classic 'v1' (not v2) FLORA pixels, WS2811 drivers)

Two more additions:
The strip definitely works. I connected it to one of those prebuilt WiFi controllers and all LED’s light up. So something is wrong in my setup.
Maybe this video will show what I am doing wrong:
https://www.dropbox.com/s/i9zzdt664eaxkmx/Video Jul 02%2C 17 47 56.mov?dl=0

Hi @VeniceNerd,
sorry for my late answer. Unfortunately there were a few instructions on the setup missing for the latest version. I added these to the wiki now: https://github.com/toblum/McLighting/wiki/Software-installation#configure-pin--rgb-mode--numleds-only-for-v221-or-later
So I think you just need to call:
http://<IP_ADDRESS>/pixelconf?ct=300&rgbo=grb&pin=5
to tell McL that you're using 300 LEDs in GRB mode and that the daa pin is connected to pin 5 (D1).
Sorry for the confusion.
Best regards
Tobias

So I think you just need to call:
http://<IP_ADDRESS>/pixelconf?ct=300&rgbo=grb&pin=5
to tell McL that you're using 300 LEDs in GRB mode and that the daa pin is connected to pin 5 (D1).

YES! That finally lit my my LED strip!!! Out of curiosity, why is pin 5 on D1? Is there any rhyme or reason to it?
Also, I don't think the RGB mode is set correctly for my strip yet. Even when I set it to "static red" my strip looks like this:

My strip is actually an RGBW RGB+Cool White SK6812 LED strip.
Do I need to set a different mode for this strip? Also, how do you actually set the pure white channel in McLighting? I did install the developer branch which I believe had white mode support?
Thanks for all your help!

Hi @VeniceNerd,
I'm not sure why they chose this pinout scheme, but you can use this as a reference: https://randomnerdtutorials.com/esp8266-pinout-reference-gpios/
I think the RGB mode is OK as a wrong mode would only swap the colors (red would be green and so on). Does it react to any changes you do via UI? Seeing the serial output when you change the mode would be interesting.
I'm not sure with the white mode. I'm waiting for my new SK6812 strip to arrive from china. It might be that there is still some code missing to support RGBW as there is still a big PR https://github.com/toblum/McLighting/pull/364 pending. I'll have to figure that out too.
Regards
Tobias

Hey Tobias,
I’d love to send you one of my SK6812 strips. Would be more than happy to contribute to this project. Especially since I’d really like support for these strips. ;)
Going by you name I assume you are in Germany? (I’m German too). If you can PM me your address I’ll send you a strip right over!

@VeniceNerd Yes, I think that's the best option now if you want to use RGBW. Maybe we'll have a combined version someday, but for the short term, the mantioned version is great.

I think the RGB mode is OK as a wrong mode would only swap the colors (red would be green and so on). Does it react to any changes you do via UI? Seeing the serial output when you change the mode would be interesting.
I'm not sure with the white mode. I'm waiting for my new SK6812 strip to arrive from china. It might be that there is still some code missing to support RGBW as there is still a big PR #364 pending. I'll have to figure that out too.

McLighting currently supports RGB strips only. This means it sends out 24 bits per pixel, which will be used as 8 bits for each of the three colours. SK6812 strips however have a fourth channel and thus use 32 bits per pixel. This is the reason why a configuration for solid red ends up in a coloured strip.
To make McLighting compatible with 32bits/pixel is actually documented here:
https://github.com/toblum/McLighting/pull/364

@toblum

Maybe we'll have a combined version someday, ...
It is already "combined"
See:

@bpohvoodoo whaaaaat??? How do I get to that page???

If you use my version just click on the gear icon

@VeniceNerd should probably close the issue, if you're up and running.

I haven’t had time to try anything new yet. Will happen this weekend.

@bpohvoodoo looks like the installation guide on your fork isn’t working. Links to a random Wordpress blog. Do I just follow the installation guide here but use your fork instead of the toblum version?

@bpohvoodoo
That looks great. I have tried it out recently and it looks great. Unfortunately I don't find the time at the moment to do something similar in my McLighting version. I have to pause the project a bit as I really don't find the time.
For the meanwhile I will post a note on the project readme and will also mention your fork for all people looking for more functionality. I hope this is OK for you.
Regards
Tobias

I just tried McLighting for the first time today.
I came across this post as my LEDs were not lighting.
My D1 mini clone had previously been running another well known firmware without issue.
The above solution of http://<IP_ADDRESS>/pixelconf?ct=300&rgbo=grb&pin=5 also got me up and running.
I am much enjoying the improved effects.
Question is why did it not take the correct pin from the sketch? Is it because it had already been programmed? How do you do a full istall and overwrite everything?

Hi @grumpazoid,
the pin setting in code is mainly for legacy reasons. For newer versions you can do the config via REST call as described here: https://github.com/toblum/McLighting/wiki/Software-installation#configure-pin--rgb-mode--numleds-only-for-v221-or-later
Greetings

Question is why did it not take the correct pin from the sketch? Is it because it had already been programmed? How do you do a full istall and overwrite everything?

The ESP has several memory regions. There is at least one for firmware code and some for user data - this is where configuration is stored. It is not McLighting specific. If you are unaware about this you may not know the difference between flashing and flashing after a full reset. So there is a chance McLighting just used settings that were left from other firmware and got to wrong conclusions.
How exactly the flash/full reset get triggered your side depends on the tools you are using.

@toblum @HiranChaudhuri Thank you - all makes sense now.


fix(loader): update commit if more than a minute elapsed

Reduces timestamp update frequency

Closes #21


Evaluate needed changes to make stats dynamic in admin UI

As an admin user, I want to see stats that reflect the current data in production, not static data.
We need to evaluate what do we need to do in order to achieve that behaviour.

closing as a new ticket has been created.


[WFLY-19376] Upgrade WildScribe 3.0.0.Final

Generate WildFly management model from its experimental stability level
Display Ⓟ, Ⓒ, Ⓔ  characters next to resp. preview, community, and  experimental in the generated HTML pages

This fixes https://issues.redhat.com/browse/WFLY-19376

This requires a big jump in the minimum maven version:
[ERROR] Failed to execute goal org.jboss.wildscribe:wildscribe-maven-plugin:3.0.0.Final:generate-site (generate-site) on project docs: The plugin org.jboss.wildscribe:wildscribe-maven-plugin:3.0.0.Final requires Maven version 3.9.4

Currently it's
        <!--
            Build configuration.  Override JBoss Parent settings as necessary.
            For example: <version.surefire.plugin>2.11</version.surefire.plugin>
          -->
        <maven.min.version>3.6.0</maven.min.version>

We may want to do that (I know it's come up a few times) but we should discuss.
Whatever our min is we should probably find a way to enforce that it works in our various plugins so we don't end up with a plugin requiring an update that we're not willing to accept, or perhaps not detecting that our min has increased. (I got lucky detecting this by building the docs on a machine with an old maven version.)

Actually, it wasn't a very old maven version; it's 3.9.1.

Good catch. It shouldn't actually require that new of a version, there might just be a dependency marked incorrectly or some property set wrong. I've seen this before. I'll have a look.

This should fix it https://github.com/wildscribe/wildscribe/pull/68. The question becomes, do we want to do a 3.0.1, or go ahead and include some extra changes and do a 3.1.0?

@jamezp @jmesnil Looking at the open issues and PRs in https://github.com/wildscribe/wildscribe I think getting 3.0.1 out and integrated makes sense. Other than the parent pom bump the other items don't sound like things that are certain to get done quickly.

Superseded by #17991


STREAMS-651 processor module for api of http://pipl.com/

resolves https://issues.apache.org/jira/browse/STREAMS-651

LGTM 👍


IMU Unit System

Hi, I'm currently doing a research using beamNG.tech and one of the sensors I'm using is the IMU provided.
I need for my research the linear acceleration values of the vehicle.
I'm having a little bit of trouble understanding the IMU values, as they are not very well documented.
These are my questions:
a) Which unit system is the IMU reporting? (rad/s for rotation? Newtons for linear force?)
b) The force reported uses the mass of the vehicle or it just represents the acceleration? (If so, is it reporting m/s^2?)
Thank you,

Hey, the IMUs do report rotation in rad/s and acceleration in m/s^2. This does not include gravity. The current value of gravity can be obtained in Lua using core_environment.getGravity(), but I can expose that to BeamNGpy with relative ease (I actually exposed setGravity() but I guess never did the same for the getter 🙃.)
What your second comment points out does seem like a bug though; I'll change the IMUs to rotate according to the orientation of the vehicle they are mounted to.

Was this resolved? Does the IMU now relate to the vehicle's orientation?


Upgrade google-api-client version from 1.27.0 to 1.28.0

Thanks for stopping by to let us know something could be better!
Please include as much information as possible:
Environment details

OS: Linux
Java version: Java 8
google-cloud-java version(s): 1.63.0

Steps to reproduce

?
?

Stacktrace
$ mvn compile

[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR] /usr/local/google/home/suztomo/cloud-opensource-java/example-problems/class-removed-from-google-http-client/src/main/java/com/google/cloud/tools/examples/HelloTransport.java:[31,50] cannot access com.google.api.client.http.apache.ApacheHttpTransport
  class file for com.google.api.client.http.apache.ApacheHttpTransport not found

Code snippet
  public static void main(String[] arguments) throws GeneralSecurityException, IOException {
    GoogleApacheHttpTransport.newTrustedTransport();
    System.out.println("Hello Transport");
  }

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
  <modelVersion>4.0.0</modelVersion>

  <groupId>com.google.cloud.tools.opensource</groupId>
  <artifactId>class-removed-from-google-http-client</artifactId>
  <version>1.0-SNAPSHOT</version>

  <name>class-removed-from-google-http-client</name>

  <properties>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <maven.compiler.source>1.8</maven.compiler.source>
    <maven.compiler.target>1.8</maven.compiler.target>
  </properties>

  <dependencies>
    <dependency>
      <groupId>com.google.cloud</groupId>
      <artifactId>google-cloud-bigtable</artifactId>
      <version>0.81.0-alpha</version>
    </dependency>
    <dependency>
      <groupId>com.google.api-client</groupId>
      <artifactId>google-api-client</artifactId>
      <version>1.27.0</version>
    </dependency>

  </dependencies>
</project>

External references such as API reference guides used

?

Any additional information below
Com.google.cloud:google-cloud-core-http:1.63.0 and the following Maven artifacts depend on com.google.api-client:google-api-client:1.27.0, which contains com.google.api.client.googleapis.apache.GoogleApacheHttpTransport. This class (version 1.27.0) does not work with com.google.http-client:google-http-client:1.28.0 used by com.google.cloud:google-cloud-bigtable:0.81.0-alpha and other artifacts.

com.google.cloud:google-cloud-core-http:1.63.0
com.google.apis:google-api-services-bigquery:v2-rev20181104-1.27.0
com.google.apis:google-api-services-compute:v1-rev20181022-1.27.0
com.google.apis:google-api-services-dns:v1-rev20180813-1.27.0

By upgrading the google-api-client version to 1.28.0, the libraries will become more compatible in users' environments. I see api-client.version is set 1.27.0 in pom.xml

Not sure, but I think google-cloud-java now depends on 1.28.0 and that's the cause of the issue. We either need to upgrade everything else or downgrade google-cloud-java to <IP_ADDRESS>.

I would suggest an explicit import on com.google.http-client:google-http-client-parent if you're using a class from that artifact.  We reduced the dependencies on google-cloud-java to only depend on artifacts that we use explicitly.
Also, I would suggest that you upgrade your dependency on com.google.api-client:google-api-client

@elharo and I are meeting on this.  It looks like

I see the version has been updated to 1.29.2. Thank you. Closing this issue.
<api-client.version>1.29.2</api-client.version>

https://github.com/googleapis/google-cloud-java/blob/7ac29d0/google-cloud-clients/pom.xml#L157


Admin boundary way stops rendering when edited

Was editing some admin boundaries and while dragging a node the way it was part of just stopped rendering. No errors in the console window.
At first I thought I'd never reproduce this, but after a bit of debugging I've determined it's related to the "Boundaries" and "Other Features" map feature layers. The easiest way to reproduce:

Disable all map features except for boundaries
Find a boundary line which isn't tagged and only exists as a relation member
Drag a node belonging to the way, then deselect the way
At this point the line disappears, but will render again if you enable "Other Features"

I know about the described behavior and I've always considered it a feature, although I concur there's a minor bug in there.
There is general inconsistence about untagged lines that are members of relations (either multipolygons or boundaries). Sometimes iD recognizes their relation membership and filters them according to it, and sometimes not, bundling them with "other features". I haven't recognized a pattern, sometimes different lines of the same type behave differently on the same screen.
The "right" answer is probably to treat those lines according to the relation they make up.

#6277 is about lines that are members of boundary relations not hiding when they should be hiding, so I think this is a duplicate. It seems to have something to do with the order features are downloaded and added to the graph, or some kind of race condition.


Change getDeviceStates region tag to "iot_get_device_states".

I don't believe the build failures are due to this change?

Need to change the end tag, too.

Done, thanks.


Fix parsing scripts where the source_p is NULL.

To give more background: parsing an empty string is valid. Since the input buffer has a size of zero, any source_p should be accepted. However, we assign some special meaning to NULL in some cases in the code, and passing a NULL source_p results a segfault. This patch fixes this by setting a non NULL value to source_p.


Emit an io.Reader for streaming blobs

This updates blobs with the streaming trait to io.Reader instead of []byte. They'll now look like:
import (
	"io"
)

type MyStruct struct {
	StreamingBlob io.Reader
}

Or if they have the mediatype trait too:
import (
	"io"
)

type CityImageData io.Reader

func (m CityImageData) MediaType() string {
	return "image/jpeg"
}

Note that this PR is dependent on this Smithy change. This PR is still merge-able. It'll run, but the code path is effectively dead until the Smithy PR is merged and this generator updates to that version.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of your choice.

A note, since io.Reader is an interface we don't need to generate it as a *io.Reader instead it can just be io.Reader

To extend on @skmcgrail  comment, generated interface types should never be used as pointers. they should always be value as a general rule. Since interfaces are already a pointer, similar to maps and slices.

Does smithy allow a non-toplevel input/output shape to contain a Stream member?

Nope, wouldn't really make any sense.

Will this account for the behavior differences between an stream used as input vs a stream used as output?

No, you can't guarantee a given shape isn't used for only output or only input. I know we discussed making custom types for operation inputs / outputs, so that might be a customization we can do then.

type CityImageData io.Reader

Smithy won't be able to create a type alias of an interface. Go does not allow type alias of interfaces.
The MediaType shapes probably will need to structures that contain a underlying reader. What populatees that media type's reader will be based on how it is used. e.g. streaming vs nested.
type CityImageData struct {
     io.Reader
}

type AsCityImageData(r io.Reader) *CityImageData { return &CityImageData{Reader: r} }

Downside of this approach is that if the underlying io.Reader provided by the user has any additional optimizations like ReadFrom etc, those will not be used by utilities like io.Copy because Go type assertions will not query embedded interface implementations. Potentially could generate these helpers onto the generated MediaType to compensate, as a fallback.

We'll also want to be aware that the the existing SDK's designs treat input and output streams differently. Input stream being be an io.Reader, and output io.ReadCloser.
The smithy generation could consider requiring an io.ReadCoser for input as well, but this is uncommon outside of the http.Client's Request since most ownership of Closing is expected to be the caller, not the callee.
Utilities like ioutil.NopCloser are available if a user doesn't want Close to be called or dont' want to use a type that implements io.Closer.


Sam/nix and conventional ami

This PR will need minor follow up prior to approval/merge for github actions that are dedicated to specifically merging to develop. It supersedes pr #953
Documentation of changes in https://github.com/supabase/postgres/pull/1012
Conventional AMI approach
The existing/conventional AMI build approach installs postgres from the postgresql-common ubuntu/debian package at the time of the AMI build. In addition, it builds extensions, and wrappers from source at the point of AMI build, and installs them as ‘.deb’ packages.

Nix packaged postgresql bundle approach
In the nix approach, we use the postgresql provided by nixpkgs (currently pinned at version 15.6 vi a76c4553d7e741e17f289224eda135423de0491d commit of nixpkgs-unstable branch locked via https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/flake.lock#L114 )
Nixpkgs sources from https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/generic.nix#L52 ← this URL
the nixpkgs package applies the following patches for aarch64-linux pg 15.6
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/disable-resolve_symlinks.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/less-is-more.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/hardcode-pgxs-path.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/specify_pkglibdir_at_runtime.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/findstring.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/locale-binary-path.patch
https://github.com/NixOS/nixpkgs/blob/a76c4553d7e741e17f289224eda135423de0491d/pkgs/servers/sql/postgresql/patches/socketdir-in-run-13.patch
When a PR is submitted to the supabase/postgres repo updating any of the nix packages maintained there, a build of the entire bundle is triggered on supported systems (x86_64-linux and aarch64-linux as of this writing). When this the nix ci workflow is initiated, nix is able to source from our binary cache (currently located in a publicly readable aws s3 bucket at https://nix-postgres-artifacts.s3.amazonaws.com ) and will check for any component dependency which has an exact match and has already successfully built. Nix will source that built version from the cache, and only build the items that have changed. If nix cannot build a changed item, the build will fail. If the build succeeds, nix will perform flake “checks” (scripted tests with dependencies managed by nix). An example of the “check” is seen here
Our CI implmentation of nix has only 2 trusted public keys and 2 specified nix caches (ours and the upstream nixpkgs community cache https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/docker/nix/Dockerfile#L5 and on the AMI at https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/scripts/nix-provision.sh#L20 )
https://github.com/supabase/postgres/actions/runs/9468138054/job/26083806922?pr=953#step:6:813  this starts the database, and enables several extensions post-build. If this test fails, the build will also fail. If the nix build and check succeeds, the build will upload the artifacts to the nix cache for re-use prior to stopping this workflow.

In the debian/ubuntu postgresql-common package, the “postgres” user is created, and postgres is installed to locations that are conventional for debian/ubuntu. In the nix approach, we explicitly create the “postgres” linux user, and then we use the nix profile install method to install the nix-built binaries for postgres, into the nix profile for the “postgres” user (located a /home/postgres/.nixprofile on the ami machine). We then alias the installed file locations to the conventional debian/ubuntu locations for postgres installation. nix profile command will give us an imperative way to install, uninstall, and upgrade packages that we build with nix going forward, allowing us to integrate our nix-built packages with debian/ubuntu distributions.
2 Stage AMI approach
The Ansible and Packer code has been forked in parallel in the same repo, so that both the nix-built approach, and the existing ubuntu/debian package approach can be supported in paralell. This will allow continued production rollouts under the old method, while also allowing targeted rollouts with the nix build AMI.
The existing build lives under the same ansible folder, and the companion packer hcl files have been retained. The parallel nix AMI build has parallel packer files with nix inserted into the name, and a new folder ansible-nix  . Both of these builds use the same command line command recipe to initiate them.
Description of 2 stage approach
The previous packer/ansible build used the https://developer.hashicorp.com/packer/integrations/hashicorp/amazon/latest/components/builder/ebssurrogate exclusively. The new nix-based retains the ebssurrogate approach to build and configure everything except for the postgres bundle.
The nature of nix builds is that they are already “sandboxed” and isolated at build time, and the results are store in a read only directory called the “nix store”. Nix has never had the need to support building in chroot as ebssurrogate packer build does, and so running nix in chroot has never been supported for these reasons. Therefore, a second stage of the AMI build was introduced, that securely sources the private  “stage1” AMI built by the stage 1 ebssurrogate approach, and then installs the nix built suapbase postgres/extensions/wrappers bundle from binary cache using the conventional github.com/hashicorp/amazon packer plugin, and limited to installing, configuring and testing postgres from either files uploaded in the first stage, or sourced from nix cache (other than stage 2 ansible playbook and unit test files). The workflow that performs these 2 stages is located here https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/.github/workflows/ami-release-nix.yml As more Supabase projects are packaged in nix, they will be moved into this 2nd stage for installation and configuration. In the 2nd stage we run migration and unit tests, and linux user/group assignment checks with a temporarily installed copy of osquery https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/ansible-nix/tasks/stage2/playbook.yml#L71 and https://github.com/supabase/postgres/blob/sam/2-stage-ami-nix/ansible-nix/files/permission_check.py
The 2nd stage also creates path aliases to the nix-installed binaries so that files and configurations are still where they are expected to be as much as possible. This allows post-AMI-build init scripts like https://github.com/supabase/infrastructure/blob/develop/init-scripts/project/00-init.sh to continue to succeed in running.
We are maintaining documentation on how to work with the nix portion of supabase/postgres at https://github.com/supabase/postgres/tree/sam/2-stage-ami-nix/nix/docs  and will continue to expand that as much as possible.
Current Progress on adoption in https://github.com/supabase/postgres
There is an umbrella draft PR at https://github.com/supabase/postgres/pull/953 which includes the building of an aarch64-linux AMI
Docker image PR https://github.com/supabase/postgres/pull/986
Docker AIO Image PR https://github.com/supabase/postgres/pull/987
The Ansible and Packer code has been forked in parallel in the same repo, so that both the nix-built approach, and the existing ubuntu/debian package approach can be supported in paralell. This will allow continued production rollouts under the old method, while also allowing targeted rollouts with the nix build AMI.

@darora just wanted to follow up that in this new PR testinfra ami tests are now passing for the nix ami build
https://github.com/supabase/postgres/actions/runs/9672036422/job/26683662377?pr=1012#step:10:95
This resolves https://github.com/supabase/postgres/pull/953#discussion_r1650347893
I have moved the docker work to a new PR that should be coming up tomorrow (and will deprecate the old)

In my view we still need some automatic deduplication of config files to be sure they don't have differences and will not have in the future. E.g ansible-nix/files/kong_config/kong.service.j2 and postgres/ansible/files/kong_config/kong.service.j2 look like copies and many other files like that. If we have files derived from other ones, I think it's better to have one "master" one and the other to be derived in an automatic way. Otherwise, we could get easily confused in the changes between them. And also this make PR somewhat bulky and hard to review. Maybe a script that derives config files from main Supabase repo is a suitable solution for this. Or symlinks to existing files.

@pashkinelfe that's reasonable to me, thanks.

The diff itself looks fine, though someone will need to test an actual upgrade to see if there's any other hurdles that are not immediately obvious. Agreed with Pavel if it's doable in a straightforward manner. Otherwise, I guess we'll have to review a diff of the two dirs locally as a one-time exercise.


[reconcile] introduce connection model

When a connection's labels change, there should be a way to reconcile this.

Not necessary with label mappings.


Batch support

I suppose batch support is not yet implemented?
The inference speed is nice but it needs to be batched in order to use 100% GPU

Hey. Batch support is there for normal inference, but not for adaptive depth/ width. However, at least for adaptive width it should be easy to add batch support. I can have a look into it the next days.

Using match_pair function with input shapes [1, 3, 480, 640] where 1 is batch size crashes the code.
Maybe match_pair is not the correct function to use?
Exception has occurred: RuntimeError
Expected 3D (unbatched) or 4D (batched) input to conv2d, but got input of size: [1, 1, 3, 480, 640]

Any updates on this?

Yes, setting detection_threshold=0.0 should yield the same number of keypoints every time, but there might be some corner cases where less than max_num_keypoints have a detection score >0. With the filter_threshold parameter you can then set a confidence threshold on the correspondences. This is also the setup we used for training LightGlue.
Thanks for reporting this issue, very glad to see people using LightGlue :)

Cant quite get it to work. Im getting index errors
My inputs to mather are:
data = {'image0': {**feats0, 'image': image0}, 'image1': {**feats1, 'image': image1}}
1024 is the feature count, 4 is batch size
Where data looks like:
{
  "image0": {
    "keypoints": (4, 1024, 2),
    "keypoint_scores": (4, 1024),
    "descriptors": (4, 1024, 256),
    "image": (4, 3, 512, 512)
  },
"image1": {
  "keypoints": (4, 1024, 2),
  "keypoint_scores": (4, 1024),
  "descriptors": (4, 1024, 256),
  "image": (4, 3, 512, 512)
 }
}

Exception has occurred: IndexError
The shape of the mask [4, 1024] at index 0 does not match the shape of the indexed tensor [1, 1024] at index 0
  File "/home/ma/src/nerfstudio_preprocess/LightGlue/lightglue/lightglue.py", line 400, in _forward
    ind0, ind1 = ind0[mask0][None], ind1[mask1][None]
  File "/home/ma/src/nerfstudio_preprocess/LightGlue/lightglue/lightglue.py", line 343, in forward
    return self._forward(data)
  File "/home/ma/src/nerfstudio_preprocess/lightning_system.py", line 27, in match_pair
    matches01 = self.matcher(data)
  File "/home/ma/src/nerfstudio_preprocess/lightning_system.py", line 70, in predict_step
    r = self.match_pair(frames_t, still_batch_t, feats0, feats1[0])
  File "/home/ma/src/nerfstudio_preprocess/gui/callback.py", line 42, in on_predict_batch_end
    for res in output_iterator:
  File "/home/ma/src/nerfstudio_preprocess/walker.py", line 220, in main
    inferencer.predict(ls, dataloader)
  File "/home/ma/src/nerfstudio_preprocess/walker.py", line 239, in <module>
    main()
IndexError: The shape of the mask [4, 1024] at index 0 does not match the shape of the indexed tensor [1, 1024] at index 0

Here are the shapes ar line 400 in lightglue.py:
ind0.shape
torch.Size([1, 1024])
ind1.shape
torch.Size([1, 1024])
mask0.shape
torch.Size([4, 1024])
mask1.shape
torch.Size([4, 1024])

You can reproduce this by trying:
 data = {
            "image0": {
                "keypoints": torch.rand(4, 1024, 2).cuda(),
                "descriptors": torch.rand(4, 1024, 256).cuda(),
                "image": torch.rand(4, 3, 512, 512).cuda(),
            },
            "image1": {
                "keypoints": torch.rand(4, 1024, 2).cuda(),
                "descriptors": torch.rand(4, 1024, 256).cuda(),
                "image": torch.rand(4, 3, 512, 512).cuda(),
            }
        }
matches01 = matcher(data)

Hi @marisancans,
with the latest merge we updated the defaults to perform early stopping / pruning by default. You have to manually switch off adaptive depth and width, which lack batch-support. Then your code should work again :)

Im not sure what I did, but I managed to do it without disabling anything
You can check if it is correct, because I have no idea what I did there to be honest.
Here are the changes I did:
Added batch support to ind0 and ind1

Also here

And here:

Now whats left is postprocessing

You are right, the results are wierd and not a lot of points found.
For now it looks like I got it working in batch mode, performance increase is huge.
Thanks for support!

For future reference in case someone stumbles upon the same problem:
This is how I am doing batching.
Get features
# Where im0 and im1 are image pairs (B, C, H, W)
feats0 = extractor({ "image":  im0 })
feats0 = extractor({ "image":  im1 })

Use matcher
custom_match_pair(im0,im1, feats0, feats1)
Do filtration of points, note that I havent yet figured out how to do scaling (but i think its possible, currently both image pairs are the same sizes)
def custom_match_pair(self, image0, image1, feats0, feats1):
data = {'image0': {**feats0, 'image': image0}, 'image1': {**feats1, 'image': image1}}
    matches01 = self.matcher(data)
    matches0, mscores0 = matches01['matches0'], matches01['matching_scores0']

    pred = {**{k+'0': v for k, v in feats0.items()},
            **{k+'1': v for k, v in feats1.items()},
            **matches01}
    # Maybe needed if you get funky nvidia errors
    # pred = {k: v.detach() if isinstance(v, torch.Tensor) else v for k, v in matches01.items()}

    # if scales0 is not None:
    #     pred['keypoints0'] = (pred['keypoints0'] + 0.5) / scales0[None] - 0.5
    # if scales1 is not None:
    #     pred['keypoints1'] = (pred['keypoints1'] + 0.5) / scales1[None] - 0.5
    # del feats0, feats1
    # torch.cuda.empty_cache()

    # create match indices
    # matches0, mscores0 = pred['matches0'], pred['matching_scores0'] # matching_scores

    m_kpts0_all = []
    m_kpts1_all = []
    matching_scores_all = []

    valid = matches0 > -1

    for v, m, kp0, kp1, scores in zip(valid, matches0, pred['keypoints0'], pred['keypoints1'], mscores0):
        matches = torch.stack([torch.where(v)[0], m[v]], -1)

        m_kpts0, m_kpts1 = kp0[matches[..., 0]], kp1[matches[..., 1]]

        matching_scores = scores[v]

        m_kpts0_all.append(m_kpts0)
        m_kpts1_all.append(m_kpts1)
        matching_scores_all.append(matching_scores)

    return m_kpts0_all, m_kpts1_all, matching_scores_all

Hi @marisancans,
match_pair does not support batches, but the forward passes of DISK, SuperPoint and LightGlue support batched inputs. However, there are some problems that need to be solved:

If the input images have different shapes, square padding or cropping is required for DISK/SuperPoint. Unfortunately, both DISK and SuperPoint tend to yield plenty of border artifacts, which could be solved by forwarding a padding mask.
To batch keypoints/descriptors, you need to ensure that the exact same number of keypoints are detected in each image. You can try reducing the  s.t. the extractors always find  features, or pad the keypoints/descriptors with random numbers.detection_threshold``max_num_keypoints

Assuming you have two sets of batched images, a simple pipeline could look like this (After PR #22 is merged):
extractor = SuperPoint(max_num_keypoints=1024, detection_threshold=0.0).eval().cuda()
matcher = LightGlue().eval().cuda()

feats0 = extractor({'image': im0})
feats1 = extractor({'image': im1})
matches01 = matcher({'image0': {**feats0, 'image': image0}, 'image1': {**feats1, 'image': image1}})

hi i do pad the keypoints with 0, but i got a low accuracy like 5%. is here a better way to solve it.
because of my dataset_image size is quite small, i set the max_num_keypoints = 128 to avoid the problem


Make default test pattern narrower

Note
Issues in this repo are for tracking bugs, feature requests and questions for the tasks in this repo
For a list:
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
If you have an issue or request for the Azure Pipelines service, use developer community instead:
https://developercommunity.visualstudio.com/spaces/21/index.html )
Required Information
Entering this information will route you directly to the right team and expedite traction.
Question, Bug, or Feature?
Type: Feature
Enter Task Name: VSTestTask2
list here (V# not needed):
https://github.com/Microsoft/azure-pipelines-tasks/tree/master/Tasks
Environment

Server - Azure Pipelines or TFS on-premises? Both, this is not specific to one customer.

If using TFS on-premises, provide the version:

If using Azure Pipelines, provide the account name, team project name, build definition name/build number:

Agent - Hosted or Private:  Both, this is not specific to one customer.

If using Hosted agent, provide agent queue name:

If using private agent, provide the OS of the machine running the agent and the agent version:

Issue Description
VSTest2 task specifies a very wide dll matching pattern which includes many unexpected dlls. This includes all TestPlatform dlls,
test framework dlls, and often dlls from local nuget package cache. This is slowing TestPlatform down, and highlights issue which was previously (in TestPlatform 17.2) silently ignored.
Instead a narrow pattern that matches dlls from bin folder only should be suggested as the default, as it fits most test projects we see in the wild and matches the common test dll naming pattern such as:
Product.Test.dll
Product.Tests.dll
ProductUnitTests.dll
Product.Unit.Tests.dll

It also does not match *TestPlatform*.dll or MSTest.TestFramework.dll, so no additional exclude patterns are necessary.
This leads to more correct test runs, and faster test runs as less dlls are included.
Related https://github.com/microsoft/vstest/issues/3939
I've implemented the change here https://github.com/microsoft/azure-pipelines-tasks/pull/16772 but there was never any feedback on it.
I am from the team that owns test platform.
Task logs
There are no logs to be shared, this is about configuration.
Troubleshooting
Error logs

Not stale.

Not stale, this is still causing us issues.


Replaced WebL10n with i18next in Stopwatch activity

Fixes #1378 for Stopwatch activity.

Nice. Thanks

@VishnuVardhanBR Hello I want to know that axios.min.js and i18nextmin.js and even l10.js will remain same in every case
am I correct please if I am wrong please correct me

@soham2312 Yes, axios.min.js and i18next.min.js will remain the same, l10n.js may require some changes in some activities but is mostly the same.
You can check the guide here if you haven't already.


Keyvault 7.5 dropped sas_definition API bindings

Package Name: azure-keyvault-keys
Package Version: 4.9
Operating System: Ubuntu
Python Version: 3.10

Describe the bug
A clear and concise description of what the bug is.
In this PR (https://github.com/Azure/azure-sdk-for-python/pull/31525), generated bindings for all API versions were dropped in favor of a single set of generated bindings for a single API. This is great. Except some of the models and operations did not get generated by the new pattern. Here are the ones that are important to me:
SasDefinitionBundle
SasDefinitionUpdateParameters
SasDefinitionCreateParameters
SasDefinitionAttributes
get_sas_definition
set_sas_definition
update_sas_definition
These are now missing from the package. Please add them back!
To Reproduce
Steps to reproduce the behavior:

Search for the above symbols in the latest keyvault packages and see that they do not appear.

Expected behavior
A clear and concise description of what you expected to happen.
I expect these API bindings to exist.
Screenshots
If applicable, add screenshots to help explain your problem.
N/A
Additional context
Add any other context about the problem here.
N/A

Thanks for the feedback, we’ll investigate asap.

Hi @NCarter3, thank you for opening an issue. As new API versions have been released, we've updated our code generation approach to reduce the amount of excess code we produce for each library. For example, at one point we separated the specifications for certificate-, key-, and secret-related types since we only need to generate types that are relevant for each library.
The SAS-related models and methods that you mention here are Storage-specific items that weren't considered relevant for azure-keyvault-* libraries, and were previously only generated for older API versions that didn't have the domain-specific separation that I mention above.
Could I ask how you were accessing these types, and what your use scenario is? Were you importing these items from the _generated directory, or through a public API?

You are accurately observing that I was getting to these via the _KeyVaultClient that is internal to KeyVaultClientBase. So tsk tsk to me for doing that ;)
I had assumed it was an oversight that these methods were not exposed on KeyClient.
Am I missing that these methods are in a different, storage-related package?

@NCarter3 My understanding is that managing Storage keys with Key Vault is a legacy feature (see Key Vault's feature documentation), and as such is wasn't carried over when the legacy azure-keyvault library -- which is now just a metapackage -- was deprecated in favor of separate azure-keyvault-* libraries. azure-keyvault was the last Python library to include public API support for this feature.
I can discuss this further with the SDK and service teams to confirm our support plan for this, but my understanding is that the deprecated support was intentional as the service feature was marked as a legacy one.

Hi again @NCarter3, thank you for your patience. I've confirmed that this is a legacy feature, which is why it was intentionally unsupported in our newer Python libraries like azure-keyvault-keys -- similarly, Storage discontinued support after service version 2018-03-28 and the Azure CLI discontinued support in version 2.54.
As the feature documentation states, it's recommended to update to Microsoft Entra ID authentication. Microsoft Entra managed identity is a particularly useful and secure solution that removes the need for credentials like passwords or SAS tokens.


[Usage]: How do we add model hooks?

Your current environment
PyTorch version: 2.1.0+cu118
Is debug build: False
CUDA used to build PyTorch: 11.8
ROCM used to build PyTorch: N/A

OS: Ubuntu 22.04.3 LTS (x86_64)
GCC version: (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Clang version: Could not collect
CMake version: Could not collect
Libc version: glibc-2.35

Python version: 3.10.12 (main, Jun 11 2023, 05:26:28) [GCC 11.4.0] (64-bit runtime)
Python platform: Linux-5.4.0-172-generic-x86_64-with-glibc2.35
Is CUDA available: True
CUDA runtime version: 11.8.89
CUDA_MODULE_LOADING set to: LAZY
GPU models and configuration: GPU 0: NVIDIA A100 80GB PCIe
Nvidia driver version: 535.154.05
cuDNN version: Could not collect
HIP runtime version: N/A
MIOpen runtime version: N/A
Is XNNPACK available: True

CPU:
Architecture:                       x86_64
CPU op-mode(s):                     32-bit, 64-bit
Address sizes:                      48 bits physical, 48 bits virtual
Byte Order:                         Little Endian
CPU(s):                             192
On-line CPU(s) list:                0-191
Vendor ID:                          AuthenticAMD
Model name:                         AMD EPYC 7643 48-Core Processor
CPU family:                         25
Model:                              1
Thread(s) per core:                 2
Core(s) per socket:                 48
Socket(s):                          2
Stepping:                           1
Frequency boost:                    enabled
CPU max MHz:                        2300.0000
CPU min MHz:                        1500.0000
BogoMIPS:                           4600.19
Flags:                              fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb cat_l3 cdp_l3 invpcid_single hw_pstate ssbd mba ibrs ibpb stibp vmmcall fsgsbase bmi1 avx2 smep bmi2 invpcid cqm rdt_a rdseed adx smap clflushopt clwb sha_ni xsaveopt xsavec xgetbv1 xsaves cqm_llc cqm_occup_llc cqm_mbm_total cqm_mbm_local clzero irperf xsaveerptr wbnoinvd arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold v_vmsave_vmload vgif umip pku ospke vaes vpclmulqdq rdpid overflow_recov succor smca
Virtualization:                     AMD-V
L1d cache:                          3 MiB (96 instances)
L1i cache:                          3 MiB (96 instances)
L2 cache:                           48 MiB (96 instances)
L3 cache:                           512 MiB (16 instances)
NUMA node(s):                       2
NUMA node0 CPU(s):                  0-47,96-143
NUMA node1 CPU(s):                  48-95,144-191
Vulnerability Gather data sampling: Not affected
Vulnerability Itlb multihit:        Not affected
Vulnerability L1tf:                 Not affected
Vulnerability Mds:                  Not affected
Vulnerability Meltdown:             Not affected
Vulnerability Mmio stale data:      Not affected
Vulnerability Retbleed:             Not affected
Vulnerability Spec store bypass:    Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Vulnerability Spectre v1:           Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Vulnerability Spectre v2:           Mitigation; Retpolines, IBPB conditional, IBRS_FW, STIBP always-on, RSB filling, PBRSB-eIBRS Not affected
Vulnerability Srbds:                Not affected
Vulnerability Tsx async abort:      Not affected

Versions of relevant libraries:
[pip3] numpy==1.24.1
[pip3] torch==2.1.0+cu118
[pip3] torchaudio==2.1.0+cu118
[pip3] torchvision==0.16.0+cu118
[pip3] triton==2.1.0
[conda] Could not collectROCM Version: Could not collect
Neuron SDK Version: N/A
vLLM Version: N/A
vLLM Build Flags:
CUDA Archs: Not Set; ROCm: Disabled; Neuron: Disabled
GPU Topology:
GPU0    CPU Affinity    NUMA Affinity   GPU NUMA ID
GPU0     X      0-47,96-143     0               N/A

Legend:

  X    = Self
  SYS  = Connection traversing PCIe as well as the SMP interconnect between NUMA nodes (e.g., QPI/UPI)
  NODE = Connection traversing PCIe as well as the interconnect between PCIe Host Bridges within a NUMA node
  PHB  = Connection traversing PCIe as well as a PCIe Host Bridge (typically the CPU)
  PXB  = Connection traversing multiple PCIe bridges (without traversing the PCIe Host Bridge)
  PIX  = Connection traversing at most a single PCIe bridge
  NV#  = Connection traversing a bonded set of # NVLinks

How would you like to use vllm
I want to add model hooks to a vllm model.
I found a way to the model with:
    def add_hooks_to_vllm_model(self):
        for name, param in self.model_instance.engine.model_executor.driver_worker.model_runner.model.named_modules():
            if isinstance(param, MixtralDecoderLayer): # we need to create a list of all the layers we want to catch
                hook = CaptureHook()
                param.register_forward_hook(hook)
                self.hooks.append(hook)

But these seems to only capture information for the first token prediction rather than all of them.
Is there a better way of using model hooks within the library?

I have the same question. Have you solved this?

If you set enforce_eager to be True, then it will add call the hooks everytime.

But I couldn't find a cleaner way of accessing model hooks.

Any updates?

Did you find an efficient implementation of this for all tokens?


Add accessor for range

This is my first try at really using xtensor... so please excuse me if I missed something.
In my use case, I have to create a block matrix, so I want to create a list of contiguous ranges, each new range starting at the stop value of the previous one. Right now I cannot use the xt::range (actually  xrange_adaptor) facility as it is impossible to access to the range components.
So the question is would it be acceptable to give const access to the range components (start, stop, and step) so that range  can be used in algorithms.  Otherwise, I need to create my own range class which will be redundant....
It's just a matter of adding the const accessors to the private fields (I can submit a patch if the idea is accepted). This would also be helpful for debugging purposes as the user can then print the computed ranges...
By the way, I do not understand why the methods normalize and get_stepped_range are not static and private. Static potentially slightly improves the efficiency (not having to pass this and the compiler may use the fact that it does not depend on the object members).

Hi,
No problem for adding accessors to the fields. Regarding the normalize and get_stepped_range, this sounds as an acceptable change (although I'm not sure that this can be a bottleneck).

Thank's for the answer.
See PR #1399.

Fixed in #1401


[Sample] math tutor

First beta is released. https://repo1.maven.org/maven2/com/azure/azure-ai-openai-assistants/


refactor(behavior_velocity_planner): new framework for creating virtual wall markers(#3851)

Description
Hotfix to beta/v0.8.0
https://github.com/autowarefoundation/autoware.universe/pull/3851 PR by @maxime-clem

Tests performed
confirmed scenario_simulator's result including CI ondemand view.

Not applicable.
Effects on system behavior

Not applicable.
Pre-review checklist for the PR author
The PR author must check the checkboxes below when creating the PR.

[x] I've confirmed the contribution guidelines.
[x] The PR follows the pull request guidelines.

In-review checklist for the PR reviewers
The PR reviewers must check the checkboxes below before approval.

[ ] The PR follows the pull request guidelines.

Post-review checklist for the PR author
The PR author must check the checkboxes below before merging.

[ ] There are no open discussions or they are tracked via tickets.

After all checkboxes are checked, anyone who has write access can merge the PR.

Sorry I made PR by mistake


[ha-agent] Consume RC HA_AGENT

What does this PR do?
[ha-agent] Consume RC HA_AGENT
Motivation
Need for HA Agent feature.
Describe how to test/QA your changes
1/ Run Agent with HA Agent feature enabled:
ha_agent:
  enabled: true
  group: my-group01

2/ Write messages from backend to HA_AGENT RC product
3/ Check in logs that the RC listener for HA_AGENT is receiving the msg
Possible Drawbacks / Trade-offs
Additional Notes

/merge